Transformer System

Our ability to learn about the world and act on that knowledge has transformed the conditions in which we live. Mathematics, science, and engineering extend this ability beyond any individual’s experience. We inherit explanations and methods that took generations to develop, then use them to investigate questions that earlier generations could not yet formulate. Understanding intelligence means, in part, understanding the capacity that makes this accumulation and extension of knowledge possible.

Mathematics gives this accumulation a particularly precise form. A result can be expressed through definitions and assumptions, examined independently of its author, and used in problems far removed from the one that motivated it. Its development has drawn on practical work as well as abstract investigation. The probability, linear algebra, and optimization used in deep learning belong to this shared history; they did not originate as a self-contained technology for building neural networks.

One aim of artificial intelligence is to construct systems that can learn from experience and use what they learn beyond the examples provided. A computational model makes particular hypotheses about this process testable. We can change its representations or learning rule and examine the consequences. Building useful machines and investigating the mechanisms of intelligence become connected tasks.

Learning to build these systems is an ascent through one part of a much larger mountain of knowledge. There are routes through mathematics, physics, biology, and computation, and a path opened in one discipline can make progress possible in another. The work is collective, but understanding still requires following the argument for oneself. Knowing that a result exists does not give us the knowledge needed to derive it, question its assumptions, or build on it. We have to work through the steps that connect it to what we already understand.

An advanced paper often takes several of those steps in a single line. An attention formula may assume that the reader can already move between probability distributions, matrix operations, and differentiation; a fast implementation adds assumptions about memory and parallel execution. Taking these assumptions apart is part of serious study, including for an experienced reader entering a neighboring area. An omitted derivation can make the next idea appear inaccessible. Reconstructing it gives us the means to proceed, one argument at a time, without giving up the depth of the subject.

Foundation models give this study an immediate practical importance. They are trained on broad data and adapted to many tasks, so their capabilities and limitations can affect many of the systems built around them. A model used for scientific analysis, software development, or information retrieval is consequently more than an isolated experimental result. Its behavior and the conditions under which it can be operated become questions for the people who build and rely on those systems. (Bommasani et al., 2021)

Autonomous agents extend this development from the production of an output to the organization of a process. A model can propose a calculation or a program; an agent can arrange for it to be executed, examine the result, and use that observation to decide what to do next. The longer-term prospect is that such systems become participants in research, engineering, and the operation of organizations. In that setting, their capabilities affect which work can be undertaken, their costs affect who can undertake it, and their failure modes affect what can responsibly be delegated.

Understanding the model is consequential at precisely these points. Suppose an agent fails to use a fact encountered earlier in a task. Improving its training, retrieving the fact again, changing how the context is assembled, and changing the retention of attention state are different interventions. They address different possible causes. Or suppose a system produces better answers after being given more inference time. The improvement may come from a better sequence of model decisions, from generating and selecting among more candidates, or from additional external evidence. An aggregate success rate alone does not distinguish these explanations.

Researchers and engineers who can trace such effects through the model and its execution have more precise ways to investigate, compare, and improve the system. They can ask which part of a computation should change and what evidence would establish the benefit. This remains valuable when the model, the hardware, or the surrounding framework changes.

Numinens develops this understanding through Transformer System: the mathematics, architectures, training procedures, implementations, and deployment conditions of the learned model. A second program, Autonomous Agents, will build on that foundation to examine persistent, environment-coupled operation around a trained model. The two subjects connect the acquisition of capabilities to their use in sustained work. The first task is to understand the model deeply enough to reason about that connection.

Learning, Representation, and Intelligence

The study of these systems is also part of a broader scientific problem: how can experience be organized into knowledge that remains useful beyond the observations from which it was acquired? Learning requires some regularity that can be carried from one situation to another. It also requires a way to represent that regularity without discarding distinctions on which a future decision may depend. Memory, representation, and generalization are closely connected in this problem.

Language provides an unusually rich setting in which to investigate it. Text contains explanations, arguments, descriptions of experiments, and programs. Much of it records the results of reasoning performed by other people. A language model encounters those results through a statistical learning objective. Understanding the relationship between that objective and the capabilities acquired during training is a more demanding question than whether the model can reproduce familiar forms of expression.

Consider next-token prediction on mathematical derivations and source code. The training objective rewards assigning probability to the observed continuation. The eventual task may instead be to construct a valid proof or a program that satisfies a specification. To transfer successfully, the model must learn relationships that help on the new problem, rather than only reproduce the surface form of an argument or a program. The distinction becomes experimentally important when familiar examples succeed but small changes to their assumptions cause failure.

This creates several connected questions. Which structures can the architecture represent? Which of them are favored by the data and optimization? How can inference use the resulting representations to construct and check a solution? The answers involve different evidence: an analysis of the available computation, controlled training comparisons, and evaluations that distinguish transfer from repetition. Post-training and inference-time procedures can change the outcome, so an account of the pretrained forward computation alone is incomplete.

The intellectual importance of model engineering lies partly in making these questions experimentally accessible. A model gives us a specified computation, parameters that can be changed, and outputs that can be tested. We can vary a representation, intervene on an information path, or alter the learning objective and examine the consequences. This offers a concrete way to investigate mechanisms associated with intelligence without assuming that a successful model explains intelligence in its entirety.

The mathematics behind this work is cumulative. Probability, optimization, linear algebra, and numerical analysis supply methods developed for many other problems; computing systems make their application at scale possible. The scientific knowledge present in the training data has an equally collective history. Studying a learning system brings these two accumulations together: the formal methods used to construct it and the recorded knowledge from which it learns. Understanding the interaction is a substantial intellectual undertaking, independently of any particular application.

Why Transformers?

The Transformer provides a productive starting point because its architecture exposes a persistent tension in sequence modeling. A prediction may require information from distant positions, but the means of making that information available affects both what can be learned and how the model can be computed. Recurrent models carry information through successive state updates. Convolutional models connect positions through local operations and depth. Attention allows a prediction to read from available representations according to learned, input-dependent relationships.

The 2017 Transformer organized its principal sequence computations around attention. Known input positions could be processed together within a layer, while the attention pattern connected positions without requiring a recurrent update at each intervening step. This changed the opportunities for training parallelism. It also created substantial pairwise computation and memory traffic as sequences grew. The architecture’s importance is inseparable from the research devoted to making that computation trainable and efficient. (Vaswani et al., 2017)

Pretraining then provided ways to reuse the resulting representations. BERT’s 2018 work demonstrated adaptation through task-specific fine-tuning of a pretrained Transformer. GPT-3’s 2020 study investigated tasks specified through instructions and examples in the input, without parameter updates for those tasks. These are different mechanisms of reuse. Both made the relationship between a broadly trained model and its downstream uses a substantial object of study. (Devlin et al., 2018; Brown et al., 2020)

Reuse makes two problems especially consequential. One is how training develops capabilities that remain useful across tasks. The other is how those capabilities can be made affordable to train and use repeatedly. The DeepSeek reports illustrate research on both. DeepSeek-V3 combines sparse expert computation and reduced attention-cache state with a coordinated training system. DeepSeek-R1 examines reasoning behavior under reinforcement learning and different post-training arrangements. Studying such systems requires us to distinguish architectural capacity, learning procedure, and execution cost, then examine how their choices interact. (DeepSeek-V3; DeepSeek-R1, January 2025 version)

The emphasis on language also makes the connection to agents direct: instructions, documentation, code, and many tool results can be exchanged through a model’s text interface. Perception introduces additional representations. The multimodal articles examine how images and other structured inputs are encoded and fused, preserving their geometry and execution requirements rather than treating them as ordinary text.

Transformers are the reference architecture here, not a presumed endpoint. Recurrent and state-space architectures offer other ways to retain and access information. Sparse attention changes connectivity within a Transformer, while hybrid architectures combine different kinds of sequence computation. A detailed account of the Transformer makes these choices easier to evaluate: which limitation motivates the change, what computation replaces it, and under which conditions is the replacement preferable?

From a Mathematical Model to an Executing System

Attention illustrates why the study cannot end with the layer equation. Computing scores, normalizing them, and combining value vectors specifies a mathematical operation. A straightforward implementation stores the pairwise scores before completing the normalization and weighted sum. As sequence length increases, this intermediate storage grows quadratically. The mathematical definition does not require that particular schedule.

FlashAttention exploits that freedom. It processes attention in tiles, carrying a normalization state and a partial output from one tile to the next. When the normalization changes, the previously accumulated quantities are rescaled so that they remain compatible with the enlarged set of scores. The full matrix need not be written to high-bandwidth memory. The benefit comes from connecting the algebra to the memory hierarchy, while retaining the dense attention operation rather than substituting a sparse pattern. Floating-point evaluation order still affects numerical agreement. (Dao et al., 2022)

Each part of this explanation creates a further question. What state is sufficient to continue the calculation? How can two partial results be combined? Which quantities are needed during backward propagation, and which can be reconstructed? Once those questions are answered, the tensor layout and execution schedule become intelligible. Work partitioning, synchronization, and accelerator resources then determine how effectively the algorithm runs on a device.

Generation gives the same computation a different workload. During prompt processing, many input positions are known. During one-token decoding, each active request contributes a new position while previously computed keys and values are retained. The available parallelism and the relative costs of arithmetic, memory access, and state management change. Serving multiple requests adds allocation and scheduling decisions. Consequently, a gain measured during prompt processing cannot simply be carried over to decoding or to the throughput of a complete service.

Following this chain changes what an optimization claim means. It becomes possible to identify whether an improvement comes from a different computation, a different numerical representation, less memory movement, or a better match between the schedule and the workload. It also becomes possible to recognize when a proposed improvement addresses a cost that was not limiting the application.

Resource allocation also extends beyond an individual training run. Suppose two candidate models meet the same evaluation requirement, but one costs more to train and less to serve under the intended workload. Over enough requests, that initial expense may be recovered. The choice depends on the volume and composition of future use, as well as on what the training experiments establish about quality. Scaling relations enter this decision as empirical estimates whose extrapolation needs justification; measured inference costs connect them to the deployment being considered. Optimizing training expenditure alone would answer a different question.

Transformer System follows these connections through adaptation, conditional computation, model transfer, compression, and multi-token generation as well as attention. The roadmap gives the detailed routes through the mechanisms and their empirical evaluation.

From Transformer System to Autonomous Agents

A trained model can propose an action without having performed it. To use that proposal, a system needs an execution environment, a record of the result, and a procedure for deciding how the result affects the task. ReAct studied one concrete arrangement in which model-generated reasoning, actions, and observations are interleaved. The interaction makes new information available during a task and allows subsequent decisions to depend on what actually happened. (Yao et al., 2022)

The Autonomous Agents series will develop the surrounding infrastructure incrementally, beginning with a fixed trained model accessed through an API. The initial setting is digital: files, programs, services, and other interfaces through which actions and observations can be inspected. Longer tasks introduce specific requirements. State must survive interruptions; relevant evidence must remain available; actions must respect permissions; and failed operations must be distinguishable from completed work. Memory, planning, tool execution, recovery, and evaluation will be introduced through these requirements. The agent-engineering roadmap (the future Digital Lifeform program) describes that construction.

Holding the weights fixed permits comparisons of changes to the surrounding system without simultaneously changing the model. Comparable tasks, budgets, and evaluation conditions are still needed to attribute an improvement. Retrieval may supply evidence that was absent from the context; a different task representation may preserve information that was previously lost; a recovery procedure may allow a run to continue after an interruption. These mechanisms can improve an agent without updating the underlying model’s weights.

Continual learning introduces a further problem: how experience from use should alter the model while preserving earlier capabilities. Self-iteration then requires methods for proposing changes and establishing that they are improvements. Embodied interaction adds physical sensing, control, and the consequences of acting in the world. This is the longer research progression envisaged here. It is not a universal chronology: embodied-model research, including systems such as OpenVLA, is already active. The fixed-model construction provides a setting in which the contribution of the agent infrastructure can first be investigated on its own terms. (Kim et al., 2024)

A Cumulative Technical Study

This series assumes familiarity with probability, linear algebra, differentiation, and programming. The work is to reconstruct the arguments that connect those foundations to contemporary models and their execution. A simple example is used to expose a mechanism before additional dimensions, training requirements, or runtime conditions are introduced.

The articles retain the intermediate derivations and implementation traces needed to examine a method critically. A reader should be able to follow a formula into tensor shapes, storage, masks, and execution order; compare alternative constructions; and identify which assumptions a reported result depends on. Training and deployment are included wherever they change the interpretation of the method. The resulting account is intended to support research and engineering decisions, including the decision to reject an approach that is unsuitable for the problem.

Advanced material often compresses several earlier results into a single step. Reconstructing that step is not a retreat from depth. For example, the stability of softmax, the composition of online normalization state, and the scheduling of tiled attention form a dependency chain. Understanding each link makes the next result open to analysis rather than acceptance on authority.

There are many routes worth pursuing through mathematics, physics, biology, and computation. The route developed here joins the scientific investigation of learning to the construction of systems that may participate in further intellectual work. Its practical importance and its intellectual depth are reasons to study it together, from the representation learned by a model to the system in which that representation is used.

We begin with sequence modeling. The first article reconstructs the information-access problem that motivates recurrence, convolution, and attention, then examines how the Transformer combines its components into a trainable and executable model.

Continue with M1.1 - Why Transformers, Which Architecture, and What Executes.