Amazon Web Services published a post describing kernel optimizations it implemented for fused Mixture-of-Experts (MoE) models, an architecture that scales large language model capacity while keeping per-token compute low.[1] The post says MoE operations account for a large share of total compute time, so optimizing them through kernel engineering offers the largest opportunity for performance gains.[1] A fused MoE system merges token routing, data shuffling and expert math into one batched GPU operation, which the post says lowers latency but adds complexity in kernel implementation and debugging.[1] On NVIDIA's Hopper architecture, the WGMMA instruction needs the M dimension to be at least 64 to keep compute lanes busy, and during decoding M often drops to one row, which the post calls a severe bottleneck.[1]
Amazon Web Services implemented kernel optimizations for MoE models
Amazon Web Services says MoE operations account for a large share of total compute time.
1 primary · 0 independent · 0 commentary · 1 vendorSources
- Mixture-of-Experts kernel engineering on Amazon SageMaker AI | Amazon Web Services Amazon Web Services PrimaryVendor