2 posts · Transformer System

Tag: #straight-through-estimator

← All tags Home

M8.7 - Quantization-Aware Training and Low-Bit Optimization

🗓 2026-07-14 • Transformer System • ⏱ 49 min read

A derivation- and implementation-oriented treatment of quantization-aware training: fake-quant arithmetic, straight-through gradients, learned step sizes and clipping, observer schedules, LLM-QAT data-free distillation, weight/activation/KV insertion sites, efficient QAT variants, distributed training state, and export parity.

#numinens#transformer#quantization#qat#fake-quantization#straight-through-estimator

M8.8 - Native Binary and Ternary Transformers

🗓 2026-07-14 • Transformer System • ⏱ 40 min read

A derivation- and implementation-oriented treatment of native low-bit Transformer design: BitLinear, binary and ternary weight codes, latent-weight optimization, variance control, BitNet b1.58, OneBit SVID and distillation, activation-lowering variants, distributed scale ownership, and the boundary between a code alphabet and an executable runtime.

#numinens#transformer#quantization#bitnet#bitnet-b1-58#onebit