Semantic Compression

A technique that competes with long context: instead of keeping millions of tokens in the KV Cache, the model compresses data into dense semantic vectors that preserve logical relationships, which radically lowers GPU memory usage and makes it possible to handle large documents. A direction explored by, among others, Meta AI.