ReplaySSM, built by NVIDIA and the Dao AI Lab at Princeton, is available on arXiv, NVIDIA's technical blog said.[1] According to the blog, hybrid models interleave state-space layers with a few attention layers.[1] At serving batch sizes, those layers can be the latency bottleneck up to 100K tokens of context, the blog said.[1] The method keeps a state checkpoint plus a buffer of recent inputs, which the blog says addresses three decoding problems.[1] The blog said it is integrated into TensorRT-LLM, vLLM and FlashInfer.[1]
ReplaySSM, built by NVIDIA and Dao AI Lab, is available on arXiv, NVIDIA blog says
The blog says ReplaySSM is integrated into TensorRT-LLM, vLLM and FlashInfer.
1 primary · 0 independent · 0 commentary · 1 vendorSources
- ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding NVIDIA Technical Blog PrimaryVendor