DeepSeek open-sources DeepInfer inference framework

DeepSeek published DeepInfer, its inference framework for production environments, as open source on GitHub on October 20 under the Apache 2.0 license, according to aitimes.com. The framework supports running the DeepSeek-V4.1-Flash model and works with Nvidia SM100 GPUs and Huawei Ascend 950 NPUs.

The public version currently handles offline inference only. DeepInfer does not yet include the KV cache systems, separated prefill and decoding processing, or load balancing that live service operations require.

In DeepInfer's published benchmark, prefill was measured with 4 GPUs and decoding with 64 GPUs. With 256 concurrent requests per rank in the decoding test, the framework processed more than 25,000 output tokens per second per GPU. DeepSeek said a single cluster can process 155 trillion tokens per day.

Scaling the batch size from 4,096 to 65,536 raised throughput per GPU from 60,000 to 89,000 tokens per second, while latency rose from 70 ms to 730 ms. DeepInfer combines GPU kernels with system optimizations including FlashMLA, DeepGEMM and DeepEP.

The framework also uses speculative decoding, in which a draft model proposes tokens and the main model verifies them, and reduces prefill computation by reconstructing state from the most recent 128 tokens of sliding window attention. It supports constrained output formats via regular expressions, JSON schema and EBNF grammar.