DeepSeek open-sources DeepInfer: what it is and who can run it

DeepSeek open-sourced DeepInfer, the inference engine behind its own AI service, on 20 October. It’s free under Apache 2.0, but it only runs on Nvidia’s B200 and GB200 GPUs and Huawei’s Ascend 950.

That makes this release less of a drop-in tool and more of a look inside how a frontier lab serves a 763-billion-parameter model at scale. Bloomberg reported the release the same day, framing it as a way to cut the cost of running AI.

What DeepSeek released

DeepSeek describes DeepInfer as “the high-performance inference framework that powers DeepSeek’s large-scale production services”. The public version “provides offline inference with the core features needed for serving at scale”, according to the README.

It supports one model family for now, DeepSeek-V4.1-Flash, and loads the DeepSeek-V4.1-Flash-1017 checkpoint by default. That model’s card on Hugging Face lists 763 billion parameters, a 1 million token context window and an MIT licence for the weights.

The repository had passed 10,000 stars by the evening of 20 October, and the Hacker News thread about it had more than 1,600 points within seven hours.

Which GPUs can run DeepInfer

The README lists two targets: Nvidia “SM100” GPUs and Huawei Ascend 950 NPUs. SM100 is Nvidia’s shorthand for compute capability 10.0, and Nvidia’s own table puts only two products there.

So if your team serves models on H100s, or you were hoping to try it on an RTX 5090, DeepInfer isn’t for you yet. You also need Linux, GCC 13 or newer, Python 3.12, and CUDA 13 on Nvidia or CANN 9.2 on Ascend.

What’s inside

Most of the speed comes from parts DeepSeek has open-sourced before. DeepInfer builds on its FlashMLA, DeepGEMM and DeepEP kernels, which first came out during the company’s “Open Source Week” in February 2025, plus newer kernels written in TileLang. At the time, DeepSeek called those releases “humble building blocks of our online service: documented, deployed, and battle-tested in production.”

The features worth knowing about:

• Speculative decoding. A draft-and-verify scheme the README calls DSpark, with confidence-based checking of draft tokens.
• Parallelism for mixture-of-experts models. Expert parallelism with load balancing, and data or context parallelism for attention.
• Batch-invariant inference. The same prompt gives the same output whatever else is in the batch, which helps with testing and evals. It’s Nvidia only, and with a fixed seed it allows fully deterministic sampling.
• Structured outputs. Generation can be held to a regex, a JSON schema or an EBNF grammar.

The README shows performance only as charts, measured on 4 SM100 GPUs for prefill and 64 for decode, with a script to reproduce them. It gives no headline throughput number, and we haven’t been able to check the charts ourselves.

What it doesn’t do yet

DeepSeek is clear that this isn’t a full serving stack. The release has no KV cache system, no split between prefill and decode machines, and no load balancing across replicas. The README also mentions no API server, so you can’t point an OpenAI-style client at it.

If you need to serve DeepSeek-V4.1-Flash today, its model card points to vLLM and SGLang instead.

Most teams that want B200-class hardware will rent it. Our piece on CoreWeave’s bonds looks at the debt behind that rental market. And TSMC’s latest results show how much demand there is for the chips themselves.

What happens next

DeepSeek hasn’t published a roadmap. The missing serving pieces are the obvious next step, and the README says most of the engine work for the prefill and decode split is already done.

Our take

Few teams will run DeepInfer this year, and that’s fine. Its value right now is as a reference: a working, permissively licensed example of how one of the most cost-focused labs serves a huge mixture-of-experts model. The Ascend 950 support stands out too. It’s a production inference engine that treats Huawei’s chips as a first-class target alongside Nvidia’s newest.