M8.8 - Native Binary and Ternary Transformers

M8.8 - Native Binary and Ternary Transformers

0x00 From Quantized Networks To Low-Bit Architectures

Most of M8 begins with a trained floating-point Transformer and asks how to approximate it. Binary and ternary networks begin with a more radical proposal: make the restricted arithmetic part of the model that optimization is allowed to learn.

This idea predates large language models. BinaryConnect used binary weights during forward and backward propagation while retaining precise stored weights for gradient accumulation. XNOR-Net extended binary approximations to weights and convolution inputs, tying the representation to specialized binary operations. These systems established two principles that remain central in BitNet:

1. the forward operator can use a discrete representation while training maintains continuous latent state;
2. an arithmetic saving exists only when hardware and kernels exploit the restricted alphabet.

Transformers make the old problem harder. Their linear layers hold most parameters, but attention, residual streams, normalization, embedding/output boundaries, and autoregressive state remain numerically coupled. A binary matrix is therefore not a drop-in storage trick. Scale placement, activation quantization, normalization order, optimizer behavior, and the set of layers replaced by the low-bit operator become architectural choices.

0x00A Native Training And Conversion Solve Different Initial-Value Problems

There are two broad routes to an extreme low-bit Transformer:

BitNet and BitNet b1.58 follow the first route. Their scaling evidence asks whether a model trained under the constraint can approach a comparable high-precision model as model and data scale increase. OneBit follows the second route: decompose a teacher matrix into a binary symbol matrix plus value vectors, initialize the low-bit student, and transfer behavior through distillation.

These routes should not be compared only by final bit count. Native pretraining spends the full training budget but lets every layer adapt to the numerical architecture. Conversion preserves access to an existing model but must solve a difficult initialization and recovery problem.

0x00B Binary And Ternary Are Representations, Not Complete Operators

A binary alphabet $\{-1,+1\}$ can replace multiplication by sign-controlled addition in the inner loop. A ternary alphabet $\{-1,0,+1\}$ adds a skip case. But a complete linear layer can still require:

• floating or integer activation scaling;
• high-precision normalization and residual operations;
• wider accumulation across the reduction dimension;
• per-row, per-group, or global weight scales;
• packing and unpacking into machine-addressable words;
• value vectors or other high-precision correction state;
• a target-specific kernel that turns symbols into useful throughput.

“Multiplication-free” is therefore a statement about one part of the matrix inner loop, not the entire layer or model. Even the ternary name “1.58 bit” refers first to $\log_2 3$, the information needed to distinguish three equiprobable symbols. Physical fixed-width storage commonly needs two bits per symbol before packing and compression effects are considered.

0x00C Low-Bit Architecture Changes The Training Dynamics

In a floating model, a small update changes a weight value immediately. In a binary or ternary forward path, many latent updates leave the emitted code unchanged until a threshold is crossed. The useful state is therefore not only latent weight motion but code occupancy and code transitions.

Training must manage:

continuous latent parameters
    -> centering or normalization
    -> binary/ternary projection
    -> activation quantization
    -> integer or sign-structured accumulation
    -> rescaling and residual integration
    -> surrogate backward field

Learning rate, normalization placement, scale ownership, initialization, and optimizer state can all change the frequency and stability of code transitions. This is why native low-bit design belongs after M8.7’s QAT foundations but cannot be reduced to another fake-quant recipe.

0x00D The Arithmetic Is Part Of The Model

M8.7 started from an existing floating-point architecture and asked how optimization could adapt it to a chosen low-bit deployment grid. Native binary and ternary Transformers move the constraint earlier. The code alphabet, scale placement, normalization, latent parameterization, training schedule, and target operator are part of the model design.

The owner question is:

What changes when binary or ternary arithmetic is an architectural and pretraining constraint rather than a post-training conversion target?

The answer is not simply “use fewer bits.” A native low-bit model carries at least five coupled contracts:

1. a training parameter contract for the high-precision latent state updated by the optimizer;
2. a forward numerical contract that maps latent weights and live activations to binary or ternary codes;
3. a stability contract for normalization, scaling, residual paths, and code transitions;
4. an artifact contract for packed symbols, scales, high-precision exceptions, and non-BitLinear tensors;
5. an operator contract for unpacking, accumulation, rescaling, tiling, and target hardware.

BitNet and BitNet b1.58 train those contracts from scratch. OneBit starts from a trained high-precision model, changes each linear representation, initializes the new factors from the teacher weights, and performs quantization-aware knowledge distillation. They belong in one article because they expose the same architectural question, but they are not the same training regime.

This article owns:

The local A36 article provides the BitNet, b1.58, and OneBit spine. This article corrects three dangerous shortcuts in that source: BitNet training keeps latent weights at high precision rather than storing trainable weights as INT8; “1.58 bits” names a ternary alphabet information bound rather than guaranteeing a 1.58-bit physical runtime format; and OneBit is a teacher-initialized conversion/distillation method rather than native-from-scratch pretraining in the BitNet sense.

0x01 Four Meanings Of “One Bit”

0x01A Alphabet Cardinality

For a binary weight code,

Two symbols require one bit per symbol under a fixed-length code:

For a ternary code,

and the ideal fixed-alphabet information count is

That derivation explains the name b1.58. It does not describe a conventional addressable machine type. A simple fixed-width ternary code usually consumes two physical bits per weight because one bit cannot encode three states.

0x01B Entropy Is Distribution Dependent

If the observed probabilities of negative, zero, and positive codes are $p_-$, $p_0$, and $p_+$, the symbol entropy is

$H(Q)=\log_2 3$ only when all three symbols are equally likely. A zero-heavy matrix can have lower entropy, but exploiting it requires entropy coding or a sparse representation. Those representations add block boundaries, indexes, decoding state, and irregular access. They may be smaller on disk while being worse for a dense matrix kernel.

0x01C Logical, Physical, And Executable Bits

The following quantities must not share one bits column:

The headline bit width is only the first row. M8.9 will own the concrete file and runtime ABI, but M8.8 must already make its required fields explicit.

0x01D Training State Is Not The Artifact

Let a matrix contain $P=NK$ logical weights. A native low-bit training step can still hold:

The illustrative mixed-precision Adam path is still about $16P$ bytes before activations, temporary codes, communication buffers, and allocator overhead. Native low-bit design primarily changes the forward arithmetic and eventual inference object; it does not automatically make pretraining a one-bit-memory process.

0x02 Original BitNet: Deriving BitLinear

0x02A Tensor Contract

Flatten batch and sequence into $T=BS$. For one linear projection:

BitNet replaces the dense floating-point projection with a BitLinear path. The original paper keeps a high-precision latent $W$ for optimization but derives a binary matrix for each forward pass.

0x02B Centralize, Then Take The Sign

The paper defines the tensor mean

then binarizes around that mean:

Mean subtraction changes the decision threshold from zero to $\alpha$. It tends to prevent a shifted latent distribution from collapsing most codes to the same sign. It also couples all entries: changing one latent weight changes $\alpha$, which can move the threshold seen by every weight in the tensor.

BitNet uses

as a post-matmul weight scale. Why is mean absolute value a natural binary scale? For a fixed binary matrix $B$, solve

Differentiate:

Therefore,

If $B=\operatorname{Sign}(W)$, then $\langle W,B\rangle=\sum|W_{ij}|$ and $\lVert B\rVert_F^2=NK$, yielding mean absolute weight. The original BitNet formula combines a centralized sign matrix with $\lVert W\rVert_1/(NK)$, so the scale should be treated as the paper’s method definition, not presented as the exact least-squares optimum for every centered-sign case.

0x02C Quantize The Normalized Activation

For signed $b$-bit activation codes, define $Q_b=2^{b-1}$ and

A deployment-friendly per-token integer code is

The corresponding reconstruction is approximately

The original BitNet paper reports per-tensor activation quantization during training and per-token quantization during inference. That is a deliberate stability/efficiency choice, but it means training and serving do not use identical scale granularity. A reproduction must record that difference rather than calling the graphs numerically identical.

0x02D Integer Accumulation And Rescaling

After normalization and activation coding, the core operation is

For per-token activation scales and one weight scale,

This equation exposes all work hidden by the phrase “1-bit matmul”: normalize $X$, reduce absmax, quantize $X$, decode or consume binary $B$, accumulate into a wider integer type, and apply $\beta\gamma_t/Q_b$. The weight multiply simplifies to selecting a sign, but reductions, accumulation, rescaling, residual addition, and nonlinear layers remain.

0x02E Tensor-Level Implementation Trace

This is an illustrative signed integer realization, not a literal transcription of every original BitNet activation path. BitNet v1, section 2.1 specifies symbolic clipping endpoints and also a shifted nonnegative path before nonlinearities. Here round means nearest, ties to even; the integer range is [-128,127], the encoder uses a_scale + eps, and the decoder uses a_scale / 128. Reductions and division must use a dtype that represents the positive epsilon; a zero row produces zero codes/output. The centered binary tie remains -1, as in the source. Training gradients require a separately declared STE: this hard-code sketch is only forward semantics.

def bitlinear_forward(x, latent_w, eps=1e-5, training=True):
    # x: [B, S, K], latent_w: [N, K]
    x_norm = sub_layer_norm(x)                       # [B, S, K]

    w_mean = latent_w.mean()                         # scalar, or group-local
    w_code = where(latent_w - w_mean > 0, +1, -1)  # [N, K]
    w_scale = latent_w.abs().mean()                  # scalar

    if training:
        # Original paper: per-tensor during training.
        a_scale = x_norm.abs().amax()                # scalar
    else:
        # Deployment path: one scale for each token row.
        a_scale = x_norm.abs().amax(dim=-1, keepdim=True)  # [B, S, 1]

    qmax = 128
    a_code = round(x_norm * qmax / (a_scale + eps))
    a_code = clamp(a_code, -128, 127).to(int8)       # [B, S, K]

    acc = binary_int8_matmul(a_code, w_code.T)       # [B, S, N], wide acc
    y = acc * (w_scale * a_scale / qmax)             # broadcast scales
    return y

This is reference semantics, not a promise that an eager implementation is fast. Materializing w_code, launching separate reductions, and converting layouts each step can erase the intended benefit. Training may fuse fake binarization with a floating GEMM; inference needs a packed-weight kernel that consumes the code format directly.

0x03 Why BitNet Adds Normalization

0x03A The Variance Problem

Consider one output coordinate

Under the paper’s simplifying assumptions that terms are independent, centered, and identically distributed,

If weight and activation terms are independent,

For scaled binary weights $\widetilde w=\beta B$ with $B^2=1$,

Standard initialization aims for $K\beta^2\approx1$. Then

Normalizing the activation before quantization makes the second moment predictable and keeps the projection output near the scale expected by the residual stack. The original BitNet uses a SubLN-style implementation for this role.

The derivation is an approximation, not a proof of stability. Transformer activations and learned weights are not independent, quantization changes their distributions, and residual paths correlate layers. Its engineering value is that it identifies the quantity normalization is controlling.

0x03B Normalization Placement Changes The Architecture

Moving normalization is not a metadata change:

A checkpoint is therefore tied to its normalization topology. Replacing SubLN with a familiar Pre-LN block after training is not a runtime optimization; it changes the model.

0x04 Training A Discrete Forward With Continuous State

0x04A Latent Weights Accumulate Sub-Threshold Updates

The optimizer updates a high-precision latent matrix $W$. The forward uses $B(W)$ or $Q(W)$. Without latent state, an update smaller than one discrete code step would disappear. With latent state, many small updates can accumulate until a threshold is crossed.

For original binary BitNet, define the centered margin

The code is $+1$ when $m_{ij}>0$ and $-1$ otherwise. A code transition occurs when an optimizer step changes the sign of $m_{ij}$. Because $\alpha$ is recomputed from the entire tensor, the transition condition is coupled:

This explains why one entry’s raw update is not enough to predict its code transition.

0x04B STE Selects A Surrogate Field

The true derivative of Sign is zero almost everywhere and undefined at zero. BitNet uses a straight-through estimator so gradients reach the latent weights. A conceptual implementation is:

def binary_code_ste(w):
    alpha = stop_gradient(w.mean())
    hard = where(w - alpha > 0, 1.0, -1.0)
    return (hard - w).detach() + w

The forward value is hard; the backward derivative through the final expression is approximately one. Other implementations may propagate derivatives through the scale or mean, clip the latent-weight gradient, or use a different surrogate. Those choices can produce the same forward codes and different optimization.

0x04C Code Motion Is A Better Diagnostic Than Latent Motion Alone

Track at least:

A tiny flip rate can mean the optimizer is making progress inside latent bins or that learning is stalled. A very large flip rate can mean useful exploration or unstable threshold thrashing. Read it with loss, gradient norm, latent margin histograms, code occupancy, and scale trajectories.

The original BitNet paper reports that larger learning rates improved convergence in its setup because small latent updates often did not change binary codes. That is source-reported method evidence, not a universal instruction to increase the learning rate for every binary model.

0x04D Training Loop And State Trace

for tokens in loader:
    optimizer.zero_grad()

    # Each BitLinear derives hard codes from latent FP weights.
    logits = model(tokens, quantized_forward=True)
    loss = next_token_cross_entropy(logits[:, :-1], tokens[:, 1:])
    loss.backward()                 # STE supplies latent-weight gradients

    log(code_occupancy(model))
    log(code_flip_rate(model))
    log(threshold_margin_histogram(model))
    log(weight_and_activation_scales(model))

    optimizer.step()               # updates latent FP weights/moments

Checkpointing must save the latent state and optimizer state during training. Saving only hard codes destroys sub-threshold progress and usually prevents faithful resume. Export does the opposite: it freezes the final code/scale contract and drops training-only latent and optimizer state.

0x05 BitNet b1.58: Deriving Ternary Weights

0x05A Absmean Defines Both Scale And Thresholds

For $W\in\mathbb{R}^{N\times K}$, BitNet b1.58 defines

Keeping epsilon but leaving exact ties to the declared rounding policy, rounding yields

The zero state is therefore not an independent pruning decision. It is induced by a scale-dependent dead zone. If $\gamma_W$ grows, both thresholds move away from zero and more weights may become zero. If it shrinks, the dead zone narrows. Every latent entry influences the shared absmean, so code transitions remain coupled.

0x05B Worked Code Trace

Take one toy row:

Its absmean is

Neglecting epsilon in this hand trace, the decision thresholds are $\pm0.25$, so

If an implementation treats the absmean as a reconstruction scale, the corresponding dequantized approximation is

This scaled reconstruction makes two errors visible: large values saturate to one shared magnitude, while small values become exactly zero. Original b1.58 v1, equations 1–3 display a unit-valued ternary tensor. That does not specify every later implementation’s full decoder: a4.8 v1, equation 2 and BitNet v2 v1, equation 1 explicitly multiply the codes by absmean. Therefore an artifact must select unit_ternary ($\widehat W=Q_W$, no weight scale) or scaled_ternary ($\widehat W=s_WQ_W$, declared scale dtype/axis). Neither the model-family name nor the codes alone select the decoder. Training can reorganize latent weights and surrounding layers around its chosen forward; PTQ cannot assume that a pretrained dense matrix already has useful ternary geometry.

0x05C Signed Per-Token Activations

BitNet b1.58 keeps the W1.58A8 regime and uses signed per-token activation quantization without a zero-point. The original paper establishes that granularity and signed range symbolically; the precise 127-based equation below is a named later-paper realization, explicit in a4.8/v2, not an endpoint/tie rule inferred from the original b1.58 paper. For one token row,

For the later-paper realization, the activation reconstruction scale is $s_{x,t}=\gamma_{x,t}/127$, not the epsilon-stabilized encoder denominator. The teaching reference below chooses nearest-even ties, float64 reductions/division, finite nonempty arrays and positive finite epsilon. Its max-normalized absmean avoids an overflowing sum or prematurely underflowing each summand, but is not a guarantee of an exactly rounded real-arithmetic mean. It rejects a nonzero weight population whose mean rounds to zero, an overflowing encoder denominator, or a nonzero activation row whose reconstruction scale rounds to zero—even when epsilon makes that row’s codes all zero. It also rejects an activation scale whose reconstructed $\pm127$ endpoints overflow. An all-zero row has zero codes and zero scale. These explicit numerical conventions are not claims about an unspecified runtime’s rounding or dtype.

import numpy as np

def ternary_codes_reference(w, eps=1e-5):
    w = np.asarray(w, dtype=np.float64)
    if w.size == 0 or not np.isfinite(w).all() or not np.isfinite(eps) or eps <= 0:
        raise ValueError("finite nonempty weights and positive epsilon required")
    peak = np.max(np.abs(w))
    gamma = 0.0 if peak == 0 else np.mean(np.abs(w) / peak) * peak
    if peak > 0 and gamma == 0:
        raise ValueError("nonzero weight mean not representable")
    with np.errstate(over="ignore"):
        denominator = gamma + eps
    if not np.isfinite(denominator):
        raise ValueError("weight denominator not representable")
    return np.clip(np.rint(w / denominator), -1, 1).astype(np.int8), gamma

def activation_a8_reference(x, eps=1e-5):
    x = np.asarray(x, dtype=np.float64)
    if x.ndim != 2 or min(x.shape) == 0 or not np.isfinite(x).all() or not np.isfinite(eps) or eps <= 0:
        raise ValueError("finite nonempty [T,K] input and positive epsilon required")
    gamma = np.max(np.abs(x), axis=1, keepdims=True)
    with np.errstate(over="ignore"):
        denominator = gamma + eps
    if not np.isfinite(denominator).all():
        raise ValueError("activation denominator not representable")
    sx = gamma / 127
    if np.any((gamma > 0) & (sx == 0)):
        raise ValueError("nonzero activation scale not representable")
    with np.errstate(over="ignore"):
        endpoint = sx * 127
    if not np.isfinite(endpoint).all():
        raise ValueError("activation reconstruction endpoint not representable")
    q = np.clip(np.rint((x / denominator) * 127), -128, 127).astype(np.int8)
    return q, sx

The original b1.58 paper combines its quantization with LLaMA-like RMSNorm, SwiGLU, RoPE, and bias-free linear layers. These choices matter because b1.58 is a model recipe, not a codebook that can be inserted into an arbitrary checkpoint while preserving the function.

0x05D What The Ternary Kernel Actually Computes

For one output row $i$ and activation vector $a$:

Partition indices by ternary sign:

Then

Zero-coded positions contribute nothing. This is the mathematical basis for the add/subtract description. But a fast implementation still has to decode packed symbols, load activation vectors, schedule vector additions, accumulate without overflow, and apply scales. If zeros are not skipped structurally, they may save multiplication semantics without reducing memory traffic or instruction count proportionally.

def reference_ternary_dot(a_int8, packed_q):
    q = unpack_ternary(packed_q)  # {-1, 0, +1}, reference only
    acc = 0
    for j in range(len(a_int8)):
        if q[j] == 1:
            acc += int(a_int8[j])
        elif q[j] == -1:
            acc -= int(a_int8[j])
    return acc

0x05E Storage And Compute Must Be Measured Separately

The original b1.58 paper’s reported latency and memory results used a 2-bit kernel. That is consistent with three symbols occupying fixed two-bit slots, not evidence that the physical kernel moved exactly 1.58 bits per weight. Its arithmetic-energy result is based on an operation energy model and its end-to-end measurements use the paper’s hardware/software setup. Treat all such numbers as source-reported.

An inference ledger for a native ternary model is at least

Serving adds

Native ternary weights do not make the K/V cache ternary. The M5/M8.6 cache contract still depends on attention architecture, context length, cache dtype, pages, qparams, and temporary residual state.

0x06 OneBit: A Different Route To A Binary Model

0x06A OneBit Is Not Native-From-Scratch BitNet

M6.10 owns generic teacher/student objectives and transfer evidence. M8.8 specializes that intake to OneBit’s changed binary representation: teacher/checkpoint identity, synthetic-data lineage, sign/value-vector initialization, normalization graph and QAT-to-pack parity remain required low-bit artifact fields. This does not move generic distillation ownership into M8.

OneBit targets an existing trained model. It replaces each selected FP linear matrix with:

1. one binary sign matrix;
2. an FP16 input-dimension value vector $g$;
3. an FP16 output-dimension value vector $h$;
4. a Post-LayerNorm after the binary projection;
5. teacher-guided QAT to recover capability.

For $X\in\mathbb{R}^{T\times K}$, $S\in\{-1,+1\}^{N\times K}$, $g\in\mathbb{R}^{K}$, and $h\in\mathbb{R}^{N}$:

The input vector rescales columns before the binary matrix. The output vector rescales rows after it. They restore two axes of magnitude information without materializing an FP16 $N\times K$ matrix.

0x06B Deriving SVID

Decompose the pretrained weight by sign and magnitude:

Approximate the nonnegative magnitude matrix with a rank-one outer product:

Then

Elementwise,

For one output coordinate:

Stacking every row gives

Therefore initialize

This algebra explains the execution order. Pre-scaling $X$ and post-scaling the result avoids reconstructing $S\odot(ab^\top)$ as a dense FP matrix. OneBit calls this Sign-Value-Independent Decomposition (SVID). The paper considers SVD and NMF for the rank-one magnitude approximation and reports faster convergence with NMF in its setup.

0x06C The 1.0073-Bit Ledger

For an $N\times K$ binary matrix plus two FP16 vectors, payload bits are

For $N=K=d=4096$:

There are two legitimate denominators:

1. per original dense matrix slot,

2. the OneBit paper’s average over the sign entries plus vector parameters,

The difference is only accounting, but the denominator must be named. Neither number includes packing headers, alignment, layer norms, embedding/output tensors, or runtime workspace.

0x06D Initialization Is Followed By Distillation

SVID is only an initialization. A rank-one magnitude matrix cannot exactly preserve arbitrary $|W|$. OneBit trains the student with teacher probabilities and normalized hidden-state alignment. In the paper’s notation:

The reported method did not add ordinary language-modeling loss. It generated 132K synthetic sequences from the teacher, used deterministic top-1 selection for the first few tokens followed by stochastic generation, trained the student for 50 epochs, and kept W1A16. These are paper-specific details, not defaults for all binary conversion.

The paper also reports that adding attention-score alignment to logit and hidden-state losses failed to converge in its ablation. That is valuable negative evidence: more teacher constraints are not automatically better when the student architecture has changed.

def initialize_onebit(fp_weight):
    magnitude = fp_weight.abs()
    a, b = nonnegative_rank1_factorization(magnitude)
    return LatentSignWeight(fp_weight), b, a

def onebit_linear(x, latent_sign_weight, g, h):
    sign = binary_sign_ste(latent_sign_weight)  # forward: +1 at either signed zero
    y = binary_matmul(x * g, sign.T)
    return layer_norm(y * h)

for tokens in synthetic_teacher_corpus:
    optimizer.zero_grad()  # one optimizer step per batch; no implicit accumulation
    with no_grad():
        teacher_logits, teacher_hidden = teacher(tokens)
    student_logits, student_hidden = student(tokens)
    loss = soft_target_ce(student_logits, teacher_logits)
    loss += hidden_weight * normalized_hidden_mse(
        student_hidden, teacher_hidden
    )
    loss.backward()
    optimizer.step()

0x06E Why The Value Vectors Change Stability

The preceding teaching forward uses the total sign rule +1 if w >= 0 else -1; binary_sign_ste must combine that exact forward with a declared surrogate derivative. Apply the same rule at export. This is a deliberate convention, not an assertion that the released OneBit implementation already enforces it. At official commit 42d6d7b, training uses torch.sign, allowing zero. The converter at that commit disables its binary-alphabet assertion and maps a zero sign to the bit decoded as +1. Retaining that source forward requires a declared zero-resolution step and measured conversion parity; it is not code-exact by assumption.

This small NumPy reference defines a binary export alphabet independently of any optimized packer:

import numpy as np

def binary_sign_reference(w):
    w = np.asarray(w, dtype=np.float64)
    if w.size == 0 or not np.isfinite(w).all():
        raise ValueError("finite nonempty latent weights required")
    return np.where(w >= 0, 1, -1).astype(np.int8)

def binary_pack_reference(sign):
    sign = np.asarray(sign)
    if sign.ndim != 2 or min(sign.shape) == 0 or not np.isin(sign, [-1, 1]).all():
        raise ValueError("nonempty [N,K] binary codes required; zero is not binary")
    bits = ((1 - sign.astype(np.int8)) // 2).astype(np.uint8)
    # Little-bit-first; unused tail bits are zero, outside the logical extent.
    return np.packbits(bits, axis=1, bitorder="little"), sign.shape[1]

def binary_unpack_reference(packed, logical_k):
    packed = np.asarray(packed)
    if (packed.dtype != np.uint8 or packed.ndim != 2 or packed.shape[0] == 0
            or type(logical_k) is not int or logical_k <= 0
            or packed.shape[1] != (logical_k + 7) // 8):
        raise ValueError("invalid binary storage extent")
    bits = np.unpackbits(packed, axis=1, bitorder="little")
    if np.any(bits[:, logical_k:]):
        raise ValueError("noncanonical padding")
    return 1 - 2 * bits[:, :logical_k].astype(np.int8)

Pure signs have magnitude one, while pretrained FP weights are usually much smaller. The vectors let the binary projection recover input- and output-axis magnitudes:

They also modulate gradients. Ignoring normalization for one step,

The same vectors that shape the forward magnitude shape the surrogate gradient reaching the latent sign parameter. They are not merely storage metadata. This helps explain why OneBit’s architecture and initialization can transfer a teacher more stably than dropping a pretrained matrix directly behind a sign function.

0x07 Paper Variants Change Different Bottlenecks

0x07A Variant Matrix

The methods should not be ranked by headline bits alone. They spend precision in different places and start from different assets.

0x07B BitNet a4.8: Hybrid Activation Precision

BitNet a4.8 observes that inputs to attention and FFN linear layers are easier to quantize than intermediate activations before attention output and FFN down projections. It uses 4-bit quantization for the easier inputs, then sparsifies and quantizes the difficult intermediate states to 8-bit. The paper reports that only 55% of parameters are activated under its scheme and explores low-bit K/V.

This is not a uniform W1.58A4 graph. Its advantage depends on a hybrid dense/sparse runtime, the selected activation threshold, the cost of sparse metadata, and kernels that can exploit the pattern. The paper is marked work in progress; its benchmark claims should remain source-reported.

0x07C BitNet v2: Rotate Difficult Activations

BitNet v2 replaces the sparse intermediate path with H-BitLinear at attention output and FFN down projections. It applies an online Hadamard transform before activation quantization:

The fast transform costs $O(K\log K)$ rather than materializing a dense $K\times K$ matrix. It spreads concentrated channel outliers before INT4 quantization. BitNet v2 trains W1.58A8 from scratch and then continues with A4 using a smaller amount of data in the paper’s recipe.

The transform is not free, and it changes the operator graph. It must be placed and fused consistently with the ternary projection. This connects directly to M8.3: orthogonal transforms can preserve a floating function when compensated, but native training may instead learn the architecture with the transform present.

The paper also explores post-RoPE low-bit Q/K/V and keeps the BOS-token K/V at higher precision. That is a cache-schema choice, not an automatic property of native ternary weights; M8.6 owns the full persistence contract.

0x07D 2B4T Changes The Evidence Surface

The BitNet b1.58 2B4T report provides an open native model at roughly two billion parameters trained on four trillion tokens, together with CPU and GPU inference implementations. This is stronger deployment evidence than projecting only from small models or simulated arithmetic.

It still does not prove that every architecture, scale, dataset, hardware target, or quality metric will preserve the same tradeoff. Treat the model as one reproducible point in a larger design space. The official bitnet.cpp project currently exposes multiple platform-specific packed types and optimized kernels. That reinforces the central lesson: a ternary mathematical matrix is not one universal physical representation.

0x08 Model Parallelism Makes Scale Ownership Architectural

0x08A Global Reductions Break Shard Independence

Suppose $W$ is sharded across $G$ tensor-parallel ranks. A global mean, absmean, activation absmax, or LayerNorm statistic requires collective communication. The scalar payload may be small, but adding synchronization to every BitLinear layer can dominate latency.

Original BitNet proposes group-local quantization and normalization. For weight group $g$:

For activation group $g$:

Each rank can derive local codes and scales without an additional all-reduce. But group-local quantization is not algebraically identical to global quantization. It changes thresholds, scales, and normalization domains. The checkpoint must be trained for that group topology or validated after a topology change.

0x08B Row And Column Parallelism Need Different Broadcasts

For a row-parallel linear layer, each rank owns a slice of the input dimension and produces a partial output. Input value vectors, activation scales, and ternary groups must align with that slice before the output reduction. For a column-parallel layer, each rank owns output rows, so weight scales and OneBit’s $h$ vector are naturally output-local, while $g$ and input activation state may be replicated.

This table is a design map, not a framework API. The exact ownership depends on the partition and fused kernel.

0x08C Distributed Training Trace

def tp_bitlinear(x_local, w_local, partition):
    if partition == "row":
        # Quantizer domains are deliberately local to avoid extra collectives.
        qx, sx = quantize_activation_local(x_local)
        qw, sw = ternarize_weight_local(w_local)
        partial = lowbit_matmul(qx, qw.T) * sx * sw
        return all_reduce_sum(partial)

    if partition == "column":
        # All ranks consume the same input rows and produce distinct outputs.
        qx, sx = quantize_activation_shared_contract(x_local)
        qw, sw = ternarize_weight_local(w_local)
        return lowbit_matmul(qx, qw.T) * sx * sw

Tests must compare the chosen sharded contract to its own unsharded reference. Comparing group-local training to a globally quantized oracle and calling every difference an implementation bug would be incorrect; the methods are different.

0x09 Kernel Reality: Addition-Only Is A Partial Description

0x09A Operation Ledger

For one $T\times K$ by $N\times K$ projection, a conventional dense GEMM performs approximately $TNK$ multiply-accumulate pairs. A binary/ternary path can remove general weight multiplication inside the inner product, but still performs:

weight-code loads and unpacking
activation loads
integer additions/subtractions
wide accumulation
activation absmax/absmean reductions
activation quantization and casts
weight and activation scale application
normalization and residual operations
layout transforms, tiling, and synchronization

If an implementation unpacks weights into bytes and calls a generic GEMM, it may save artifact memory but not execute the intended operator. If a specialized kernel spends more time decoding ternary blocks than it saves in arithmetic, the nominal bit width will not predict latency.

0x09B Decode Versus Prefill

Decode often has a small token dimension and repeatedly streams large weight matrices, making compressed weight bandwidth attractive. Prefill has larger matrix dimensions and can use highly optimized dense tensor-core GEMMs, so custom low-bit packing, activation quantization, and accumulator conversion must compete with a strong baseline. A kernel can win decode and lose prefill.

Batch size changes the balance again. BitNet v2 explicitly targets lower activation precision for batched inference, where activation compute and workspace become more important. M8.10 must benchmark both phases and several concurrency points.

0x09C Reference Decoder Before Optimized Kernel

Every packed implementation needs a slow oracle. This executable NumPy reference receives exact unpack and activation-quantizer adapters; it is not a packed kernel. The unpack adapter owns bit order, reserved-pattern rejection and logical tails. The oracle validates the decoded ternary alphabet and rescaling contract. sx is scalar or [T,1]; a scaled weight decoder uses scalar or [1,N] scales. Input-group scales require separately rescaled partial dot products and are intentionally rejected here.

import numpy as np

def ternary_linear_oracle(x_fp, code, act_quantizer, unpack,
                          weight_decode, w_scale=None):
    x = np.asarray(x_fp, dtype=np.float64)
    if x.ndim != 2 or min(x.shape) == 0 or not np.isfinite(x).all():
        raise ValueError("finite nonempty [T,K] input required")
    qx, sx = act_quantizer(x)
    qx, sx, q = np.asarray(qx), np.asarray(sx, dtype=np.float64), np.asarray(unpack(code))
    if (qx.shape != x.shape or not np.issubdtype(qx.dtype, np.integer)
            or np.any(qx < -128) or np.any(qx > 127)):
        raise ValueError("activation codes must be signed INT8 values")
    if q.ndim != 2 or q.shape[0] == 0 or q.shape[1] != x.shape[1] or not np.isin(q, [-1, 0, 1]).all():
        raise ValueError("invalid decoded ternary matrix")
    if sx.shape not in [(), (x.shape[0], 1)] or not np.isfinite(sx).all() or np.any(sx < 0):
        raise ValueError("activation scale must be scalar or [T,1]")
    if np.any((sx == 0) & np.any(qx != 0, axis=1, keepdims=True)):
        raise ValueError("zero scale requires zero activation codes")
    if weight_decode == "unit_ternary":
        if w_scale is not None:
            raise ValueError("unit ternary has no weight scale")
        sw = np.array(1.0)
    elif weight_decode == "scaled_ternary":
        if w_scale is None:
            raise ValueError("scaled ternary requires a weight scale")
        sw = np.asarray(w_scale, dtype=np.float64)
        if sw.shape not in [(), (1, q.shape[0])] or not np.isfinite(sw).all() or np.any(sw < 0):
            raise ValueError("weight scale must be scalar or [1,N]")
        if np.any((sw == 0) & np.any(q != 0, axis=1)[None, :]):
            raise ValueError("zero weight scale requires zero codes")
    else:
        raise ValueError("unknown ternary decoder")
    if x.shape[1] > np.iinfo(np.int32).max // 128:
        raise ValueError("worst-case dot product exceeds INT32")
    acc = qx.astype(np.int64) @ q.astype(np.int64).T
    with np.errstate(over="ignore", invalid="ignore"):
        y = acc * sx * sw
    if not np.isfinite(y).all():
        raise ValueError("rescaled output is not finite")
    return y

Validate in this order:

1. latent-to-code values and scale axes;
2. pack then unpack bit exactness;
3. integer accumulator equality;
4. rescaled layer output tolerance;
5. block output and full-model logits;
6. prefill/decode state evolution;
7. optimized-kernel dispatch and fallback absence.

M8.9 will turn this into a concrete artifact/loader contract.

0x0A Choosing A Native Low-Bit Route

The architecture decision should be made before pretraining because it determines checkpoint compatibility. A native b1.58 checkpoint cannot be reconstructed by ternarizing an arbitrary FP model after the fact. Conversely, choosing OneBit means accepting a teacher, synthetic-data, distillation, and W1A16 conversion pipeline.

0x0B Evaluation And Instrumentation

0x0BA Training Telemetry

For every BitLinear family layer, log:

0x0BB Numerical Unit Tests

Required tests include:

binary mean-threshold tie convention
ternary half-scale threshold tie convention
zero-occupancy histogram on known toy weights
per-token activation scale broadcasting
wide-accumulator overflow bound
latent checkpoint resume reproduces next code transition
group-local quantizer matches group-local oracle
OneBit SVID shape and algebra on a small matrix
OneBit bit-count denominator is named
pack/unpack preserves valid symbols; reserved patterns reject unless explicitly padding
unit/scaled ternary decoder identity and scale-axis rejection
OneBit exact-zero and signed-zero forward/export/pack parity
optimized kernel matches reference decoder

0x0BC Quality And Runtime Evidence

Paper results should be preserved with their conditions:

• The original b1.58 paper compared models trained on RedPajama for 100B tokens and reported parity beginning around its 3B setup. That is not a theorem that all 3B ternary models match FP16.
• Its larger-model latency projections and energy claims depend on its 2-bit kernel and operation-energy model.
• OneBit reports at least 81% of non-quantized performance on its tested LLaMA models and a substantial remaining gap. It is evidence that extreme conversion can work, not evidence of lossless conversion.
• BitNet a4.8 and v2 report their own activation/KV tradeoffs and are marked work in progress on arXiv.
• BitNet b1.58 2B4T provides open model/runtime evidence at a larger token budget, but still represents one recipe and model scale.

M8.10 must rerun evaluation on the actual candidate artifact, workload, hardware, prompt lengths, batch distribution, and quality suite.

0x0C Failure Lab

0x0D Boundaries And Handoffs

M8.8 produces:

binary and ternary logical code definitions
weight centralization, absmean, and activation-scale equations
normalization and variance-control topology
latent-weight and STE training contract
code occupancy, transition, and threshold telemetry
BitLinear and ternary reference forward functions
OneBit sign matrix, value vectors, SVID initialization, and KD lineage
model-parallel scale/group ownership
variant-specific activation and transform graph
artifact field requirements and reference decoder
paper-claim provenance and runtime caveats

M8.9 receives the final code tensors, scale axes, normalization graph, high-precision tensors, pack/unpack oracle, target operator shapes, training provenance, and parity tests. It must choose a concrete format, serialize it, load it without silent reinterpretation, and prove that a supported kernel consumes it.

M8.10 receives the runnable artifacts and must decide whether any of them should ship. It owns workload-weighted quality, memory, prefill, decode, throughput, energy, operational compatibility, promotion, monitoring, and rollback.

0x0E Claim Status

0x0F Publish Gate

• Binary, ternary, information, physical packing, and runtime arithmetic bits are distinct.
• Training latent state and inference artifact state are counted separately.
• Original BitNet centralization, scaling, activation quantization, and rescaling are derived.
• The mean-absolute least-squares derivation is scoped correctly.
• The BitNet variance argument and SubLN role are shown rather than asserted.
• Tensor shapes and a reference BitLinear forward are present.
• STE, latent updates, moving thresholds, code flips, and larger-LR evidence are separated.
• b1.58 absmean thresholds and zero region are derived with a numeric example.
• Ternary add/subtract algebra is separated from physical kernel performance.
• The original paper’s 2-bit runtime measurement boundary is explicit.
• OneBit is classified as teacher-initialized conversion/QAT rather than native pretraining.
• SVID algebra, value-vector execution order, and bit accounting are derived.
• OneBit training data, loss components, initialization alternatives, and negative ablation are represented.
• BitNet, b1.58, OneBit, a4.8, v2, and 2B4T are compared by mechanism and provenance.
• Distributed quantizer/normalization domains and TP ownership are explicit.
• Training telemetry, numerical oracles, artifact requirements, and failure checks are concrete.
• M8.9 retains serialization/runtime integration and M8.10 retains release evaluation.

0xFF References

Local Series And Transformer System Sources

• M6.10 - Teacher/Student Distillation and Model Transfer - generic distillation owner; M8.8 specializes the intake and export contract for OneBit. [TRANSFORMER SYSTEM]
• M8.1 - Quantized Numbers and the Kernel Contract - code, scale, zero-point, accumulation, packing, and kernel algebra. [TRANSFORMER SYSTEM]
• M8.3 - Outlier Geometry and Equivalent Transforms - Hadamard and learned rotations, online transforms, and fusion constraints. [TRANSFORMER SYSTEM]
• M8.6 - Live-Data Quantization - activation and K/V scale lifetime, cache state, and serving ABI. [TRANSFORMER SYSTEM]
• M8.7 - Quantization-Aware Training - fake quantization, STE, learned qparams, distillation, memory, and export baseline. [TRANSFORMER SYSTEM]
• Transformer Series (34) - Quantitative Fundamentals - local quantization arithmetic and training taxonomy. [SOURCE-LOCAL]
• Transformer Series (35) - Fundamentals of Large Model Quantization - local low-bit motivation, outliers, and PTQ/QAT context. [SOURCE-LOCAL]
• Transformer Series (36) - Large Model Quantization Scheme - local BitNet, b1.58, OneBit, and SVID source spine. [SOURCE-LOCAL]

Primary Papers And Official Code

• BinaryConnect: Training Deep Neural Networks With Binary Weights During Propagations - binary forward/backward weights with high-precision stored weights for gradient accumulation; foundational latent-state distinction. [PAPER]
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks - binary weight and activation approximations, scaling factors, and hardware/operator motivation. [PAPER]
• BitNet: Scaling 1-bit Transformers for Large Language Models - BitLinear, centered binary weights, absmax activations, SubLN, STE, mixed-precision latent training, group quantization/normalization, and source-reported scaling/energy results. [PAPER]
• The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits - ternary absmean weights, signed per-token A8, LLaMA-like architecture, source-reported model comparisons, and 2-bit runtime-kernel boundary. [PAPER]
• OneBit: Towards Extremely Low-bit Large Language Models - binary linear architecture, FP16 value vectors, SVID, NMF/SVD initialization, knowledge transfer, bit accounting, and limitations. [PEER-REVIEWED PAPER]
• Official OneBit Repository - released training/evaluation code, data, checkpoints, and reproduction context. [OFFICIAL CODE]
• BitNet a4.8: 4-bit Activations for 1-bit LLMs - hybrid INT4 input and sparse INT8 intermediate activation design, source-reported active-parameter and K/V results. [PAPER, WORK IN PROGRESS]
• BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMs - H-BitLinear, online Hadamard activation smoothing, A8-to-A4 training, and low-bit Q/K/V experiments. [PAPER, WORK IN PROGRESS]
• BitNet b1.58 2B4T Technical Report - open native b1.58 model trained at a four-trillion-token scale and associated CPU/GPU inference evidence. [TECHNICAL REPORT, WORK IN PROGRESS]
• bitnet.cpp - official platform-specific packed formats, CPU/GPU kernels, model conversion, and revision-specific support boundary. [OFFICIAL CODE, VERSION-SPECIFIC]