DeepSeek open-sources DeepInfer, the engine that runs its AI models

DeepSeek has open-sourced DeepInfer, the software it uses to run its AI models in production. The Chinese AI lab announced the release on X on Tuesday and published the code on GitHub under the Apache 2.0 licence. For now, it supports only DeepSeek-V4.1-Flash.

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!

Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.

🔗 https://t.co/JGAik1kgk0

1/4

— DeepSeek (@deepseek_ai) October 20, 2026

Inference is the work of running a trained model to answer requests. The same model can cost much more or less to run, depending on the software that serves it.

What is in the release

DeepInfer bundles optimised code for DeepSeek’s models with ways to split the work across many chips. It also includes speculative decoding, a technique that drafts several tokens ahead and checks them in one pass. Other features keep outputs consistent from run to run and force answers into set formats, according to the GitHub page.

It runs on two kinds of hardware: Nvidia’s SM100 data-centre GPUs and Huawei’s Ascend 950 chips. DeepSeek said code for future models and further improvements will land in the same place.

The release is not a complete service on its own. It leaves out a cache system, the split between processing prompts and generating answers, and load balancing across machines, the GitHub page says. DeepSeek also released deepseek-recipe, which matches how its official API turns text into tokens and back.

Built on vLLM

DeepInfer was first built mainly on vLLM, a widely used open-source inference engine, DeepSeek says. It also drew on SGLang and Nvidia’s TensorRT-LLM.

The vLLM project thanked DeepSeek on X for crediting the open-source community. It said it would keep improving its own support for V4.1-Flash with ideas from DeepInfer.

DeepSeek measured its published benchmarks on four SM100 GPUs for processing prompts and 64 for generating output. They are the company’s own figures. There is no independent benchmarking of DeepInfer’s gains yet, Bloomberg’s Saritha Rai reported.

Why it matters for cost

Cloud and hosting companies that already run DeepSeek’s models can now use the same methods DeepSeek uses. Companies running the models on their own servers could also cut costs, Bloomberg reported. AI companies usually guard these techniques closely, Bloomberg noted.

The release follows DeepSeek’s September model launch of V4.1-Flash. Earlier this month, it published Huawei chip tools as an alternative to Nvidia’s CUDA. Open-weight models now take 23% of spending on Vercel’s AI Gateway.

It comes a day after the US ordered civilian federal agencies to remove DeepSeek products from their systems. By Tuesday afternoon, the GitHub repository had more than 8,000 stars.