transformer IB-attention-hopfield

Softmax self-attention (the Transformer's core operation) is exactly ONE update step of a continuous modern Hopfield network with an exponential energy — i.e. associative-memory attractor retrieval — with pattern-storage capacity scaling ~2^(d/2) and the softmax inverse-temperature beta as the retrieval-sharpness knob.

Observed quantities

attention = 1 modern-Hopfield update; pattern-storage capacity ~2^(d/2); knob = softmax inverse-temperature beta

Primary source

Ramsauer et al. 2020, ICLR — 'Hopfield Networks is All You Need'

Boundary of this block

FORMAL result (Petrovich flag): a proven mathematical equivalence + capacity bound about an ENGINEERED system, not a measurement of a natural system. Included as the transformer domain's evidence anchor; the broad 'attention IS memory retrieval everywhere' generalization is NOT asserted here as a natural law.

Connections

Outgoing

relates to Iterated serial reproduction behaves as a Markov chain whose STATIONARY distribution equals the transmitters' shared prior: chains started from 2.63, 4.19 and 5.76 cm all converged to the same value within about 20 transmissions. the two domains where the attractor is mathematically exact (Hopfield fixed point ~ Markov stationary / shared prior)

Incoming

relates from Iterated serial reproduction behaves as a Markov chain whose STATIONARY distribution equals the transmitters' shared prior: chains started from 2.63, 4.19 and 5.76 cm all converged to the same value within about 20 transmissions. the two domains where 'attractor' is mathematically EXACT: Markov stationary here, Hopfield fixed point there