M2.1 — Tokenization and Sequence Contracts
How text becomes model input: learning and applying tokenizers, preserving IDs and offsets, formatting conversations, batching sequences, and checking compatibility.
How text becomes model input: learning and applying tokenizers, preserving IDs and offsets, formatting conversations, batching sequences, and checking compatibility.