Attention-to-SSM distillation research

This project explores methods at the structural intersection of attention-based Transformers and linear State Space Models - specifically, linearizing attention layers into SSM (Mamba-2 / Gated DeltaNet) layers via distillation, to study how much of multi-head attention's associative recall can be preserved under linear context-length scaling, at small (sub-2B) scale and for domain-specialized tasks.