Zyphra paper predicts 35x speedup for sparse attention at 10M tokens
Zyphra's Hierarchical Attention is estimated to run 4.7 to 5.1 times faster than MiniMax's sparse attention, according to a model built on the NVIDIA H100 GPU.
Zyphra published a paper on arXiv introducing Zyphra Hierarchical Attention (ZHA), a sparse attention technique for handling very long contexts, according to aitimes.com. The paper was first submitted on September 30 and a revised version appeared on November 2.
ZHA groups existing context into small summarised units, then builds summaries of those summaries, forming a tree-shaped search structure. When a new query arrives, the method searches from the top level for relevant regions and narrows down to only the parts it needs.
Zyphra's performance prediction model, built around the NVIDIA H100 GPU's compute and memory bandwidth, estimates that ZHA's attention processing is about 32 to 35 times faster than standard attention at a 10-million-token context. The same model puts it at roughly 4.7 to 5.1 times faster than MiniMax's sparse attention (MSA), with the gap estimated to widen as context length grows.
According to the paper, ZHA can be applied to existing pretrained models while keeping their weights, using about 4 billion tokens of distillation training and adding only 81,920 trainable parameters. In experiments with the Qwen3.6-35B-A3B model, it matched standard attention on long-context retrieval, reasoning and agentic coding benchmarks. It scored 63.4% on LongBench v2 against 58% for standard attention while referencing only about 1.1% of the total KV cache, and 67.4% on SWE-bench Verified, the same as standard attention.
The results were not uniform. On NVIDIA Nemotron 3.5 Lightning, ZHA's RULER accuracy fell below standard attention, dropping by up to 25.3 percentage points in deep tree structures. The predicted speedup has not yet been measured in practice, and Zyphra has not released an optimised execution kernel.
Competition in sparse attention
DeepSeek and MiniMax have also been developing sparse attention techniques aimed at lowering the cost of processing long contexts.
Quick answers
What is Zyphra Hierarchical Attention?
It is a sparse attention technique that organises context into summarised units and summaries of those summaries, forming a tree structure that a query searches from the top down.
Has the 35x speedup been measured?
No. The figure comes from Zyphra's performance prediction model built around the NVIDIA H100 GPU, and the company has not released an optimised execution kernel.
How much training does ZHA require?
Zyphra says it can be applied to existing pretrained models with their weights kept, using about 4 billion tokens of distillation training and 81,920 trainable parameters.