#context-geometry • Tags • Transformer System

2 posts · Transformer System

Tag: #context-geometry

M3.4 — Rotary Position Embedding

🗓 2026-06-29 • Transformer System • ⏱ 55 min read

A complete derivation and implementation trace for rotary position embedding: relative-offset identity, pair layouts, frequency bases, partial rotary, numerical precision, cache integration, and paper/runtime variants.

#numinens #transformer #rope #rotary-position-embedding #llama #qkv

Read →

M3.3 — Positional Geometry and Attention Bias

🗓 2026-06-27 • Transformer System • ⏱ 63 min read

A derivation- and implementation-oriented reconstruction of absolute coordinates, sinusoidal bases, learned positions, relative representations, attention bias, cache offsets, and positional design choices.

#numinens #transformer #position-encoding #context-geometry #sinusoidal #relative-position

Read →

Su	Mo	Tu	We	Th	Fr	Sa
27	28	29	30	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

| #context-geometry

Tag: #context-geometry

M3.4 — Rotary Position Embedding

M3.3 — Positional Geometry and Attention Bias