x = position · y = layer (L0 bottom → output top) · z = rank (1 in front) · drag to orbit, scroll to zoom, click a voxel to select
how to read the disks
One disk per head, strongest first. You are looking down the query: the centre is its tip.
Distance from the centre = attention the key gets, on a hyperbolic scale: the centre would take all of it, every e-fold less is an equal step outward, and the rim is none.
Rings: 50 / 10 / 1 / 0.1%. Dashed ring (softmax heads) = zero match; keys outside it are anti-matched.
Angle = which way a key deviates from the query (its two main directions of variation), so keys that miss in the same way sit together. It doesn't affect attention.
Colour = position in the prompt · click a disk for that head's number line.
Memory (DeltaNet) heads: distance uses each source's share of the head's effective weight (after decay and overwriting), not the raw match; hollow red = net negative.
how to read this
The arrow is the query of the selected token: what this head is looking for.
Left → right along the arrow = how well each earlier token's key matches that query (the only axis to scale).
Ticks mark the attention weight a key would get at that point (softmax heads).
The line at 0 is zero match: keys near it are ignored by this query.
Up/down (and depth in 3D) only spreads tokens by how similar their keys are; it doesn't affect attention.
Size = weight actually used · colour = position in the prompt · labels give what the strongest sources carried.
Memory (DeltaNet) heads: the weight used also depends on decay and overwriting since the key was written, so a key can match and still be small; red ring = net negative.
–
whyrather thanor
how to read this
Everything the model writes into this position's residual stream adds up, exactly, to its final logits (the final norm is a single scale, held at its actual value).
Big plot: the race between the chosen token and two rivals. x = how far it leads the first rival, y = the second, in logits. Each layer adds two arrows, tip to tail: its attention/memory write (solid) and its MLP write (dashed); colour = layer. Shading = who is ahead.
Right: the current layer's arrows split into their largest pieces: a head reading one earlier position (orange; ties into the KV tab), or one MLP neuron (blue) with the words its write points to. Grey = everything else.
Strip: both margins over the layers (click to jump).
Direct attribution: each write's own path to the output. A write that later layers read and act on is credited only for its direct part; mid-layer pieces are often small here even when they matter downstream.