ReplaySSM, built by NVIDIA and the Dao AI Lab at Princeton, is available on arXiv, NVIDIA's technical blog said.[1] According to the blog, hybrid models interleave state-space layers with a few attention layers.[1] At serving batch sizes, those layers can be the latency bottleneck up to 100K tokens of context, the blog said.[1] The method keeps a state checkpoint plus a buffer of recent inputs, which the blog says addresses three decoding problems.[1] The blog said it is integrated into TensorRT-LLM, vLLM and FlashInfer.[1]