DeepSeek has open-sourced DeepInfer, the inference engine it says runs its own production AI services. The Chinese lab published it on GitHub on Tuesday, Oct. 20, 2026 under the permissive Apache-2.0 license, alongside a matching tokenizer tool called deepseek-recipe.
An inference engine is the software that serves an already-trained model to users: it splits the model across chips, batches requests and decides how fast answers come back. That is a big part of what an AI company pays per answer.
What is in it
- One model for now: the first release runs DeepSeek-V4.1-Flash, and only in offline mode, Pasquale Pillitteri reports from the README.
- Two chip families: Nvidia SM100 GPUs (CUDA 13) and Huawei Ascend 950 NPUs (CANN 9.2). Both are data-center hardware.
- Techniques: custom kernels, mixture-of-experts parallelism across chips, speculative decoding (guessing several tokens ahead and checking them in one go) and structured output.
- Built on vLLM: per Pillitteri's reading of the README, DeepInfer is built mostly on top of the open-source vLLM engine, with ideas from SGLang and TensorRT-LLM.
DeepSeek's own production figures for V4.1-Flash, as reported by Crypto Briefing: 155 trillion tokens per day per cluster, a 96% input cache hit rate, an average time-to-first-token of around 400 milliseconds, and 230 tokens per second (up to 300 for coding). These are DeepSeek's numbers and have not been independently tested.
Why it matters
Putting Huawei's Ascend 950 next to Nvidia in an official release shows a top Chinese lab treating a domestic chip as a first-class target, as U.S. export rules limit China's access to Nvidia's best GPUs. For Tesla fans following the compute race, from Terafab to Colossus, it's another sign that serving AI cheaply is becoming as competitive as training it.
More: DeepSeek's V4.1-Flash update from Oct. 17 and Nvidia's Vera CPU win at Modal.