Neural Long-Term Memory (Test-Time Memorization)

Linear recurrent models and State Space Models (such as Mamba and RWKV) compress history into a fixed-size state vector, struggling when long-range tasks require recalling dense factual details across millions of tokens. Building on Test-Time Training (TTT) layers (Sun et al., 2024), Google Research introduced Titans on 31 December 2024. Titans pairs a short-window attention branch (short-term memory) with a deep multi-layer neural memory module whose weights are updated online during inference via momentum-smoothed gradients of an associative memory loss ("surprise") plus adaptive weight decay.

Sources: s1, s2, s3

Neural long-term memory reframes inference-time context tracking as an online optimization problem with momentum and forgetting. By allocating high capacity to surprising tokens and decaying routine transitions, it bridges the gap between quadratic full attention and fixed-state recurrent models on needle-in-a-haystack and long-horizon reasoning benchmarks.

Sources: s1, s2

In a Memory-as-Context (MAC) Titans block processing a two-million-token genomic or log stream, incoming segments are chunked; the model queries its deep neural memory weights for relevant historical context, runs local softmax attention over the current segment plus retrieved memory tokens, and then updates the neural memory weights using the associative gradient norm of the current segment.

Sources: s1, s3

Updating neural parameters along the sequence dimension at inference time requires specialized chunk-wise parallel scan kernels to utilize GPU tensor cores efficiently, and compressed weight memory still cannot guarantee lossless verbatim quotation of arbitrary spans the way an external RAG index or exact KV cache can.

Sources: s2, s3