TL;DR: DeepSeek DeepInfer is the inference framework DeepSeek uses to serve its own models, and the company open-sourced it on October 20, 2026 under the Apache 2.0 license. The first release supports only DeepSeek-V4.1-Flash and runs on Nvidia’s SM100 (Blackwell) GPUs and Huawei’s Ascend 950 chips. It brings DeepSeek’s kernels, parallelism and speculative-decoding tricks into the open, but it is not a complete serving stack: the KV-cache system, prefill/decode disaggregation and load balancing are left out. All performance figures so far are DeepSeek’s own.
DeepSeek DeepInfer landed one day after the US government ordered federal civilian agencies to remove DeepSeek products. Instead of a new model, the Chinese lab released the software layer that decides how cheaply a model can answer requests. Here’s what is in the repository, what is missing, and who can realistically use it.
What DeepSeek announced
DeepSeek announced DeepInfer on X on Tuesday, October 20, describing it as its “in-house inference framework” and saying it powers DeepSeek-V4.1-Flash “at scale with high throughput and low latency,” according to The Next Web. The code is on GitHub, where the README calls it “the high-performance inference framework that powers DeepSeek’s large-scale production services.”
Inference is the work of running an already-trained model to answer prompts. The same model can be much cheaper or more expensive to serve depending on the software around it, which is why AI companies usually keep this layer private. If you’re new to how models process text, our explainer on what tokens are in AI covers the basics.
DeepInfer at a glance
| Item | Details (per DeepSeek’s GitHub README) |
|---|---|
| License | Apache 2.0 |
| Supported model | DeepSeek-V4.1-Flash (default checkpoint: V4.1-Flash-1017) |
| Hardware | Nvidia SM100 GPUs, Huawei Ascend 950 NPUs |
| Mode in this release | Offline inference |
| Build requirements | Linux, GCC 13 or newer, Python 3.12, CUDA 13 (Nvidia) or CANN 9.2 (Ascend) |
| Not included | KV-cache system, prefill/decode disaggregation, load balancing across replicas |
| Origins | Built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM |
What’s inside the release
The README lists the main pieces:
- Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and custom TileLang kernels. Several of these libraries were already open-sourced by DeepSeek earlier.
- Parallelism: attention data parallelism and balanced context parallelism, plus mixture-of-experts expert parallelism with load balancing (EPLB).
- DSpark speculative decoding: a draft step proposes several tokens ahead and a confidence estimate decides which ones get verified.
- Asynchronous scheduling: overlaps CPU batch scheduling with work on the accelerator. It’s on by default.
- Batch-invariant inference and deterministic sampling: identical inputs give identical outputs regardless of batch makeup. Both are Nvidia-only for now.
- Structured outputs: generation can be constrained to a regex, JSON schema or EBNF grammar.
DeepSeek also released a companion library, deepseek-recipe, which converts requests into token sequences using the same logic as its official API, according to SiliconANGLE.
How DSpark works
SiliconANGLE describes DSpark this way: a draft model proposes five tokens at a time, and a confidence head predicts which of them the main model will accept, so verification compute goes only to the likeliest candidates. DeepSeek says this cuts the number of draft tokens checked per step by about 30% and estimates a 4% to 7% throughput gain at its production load. A second technique replays only a short final slice of each prompt through the decoder layers, which the company says nearly halves total prefill computation.
What’s missing, and why it matters
DeepInfer is not a drop-in replacement for a full serving platform. The README says three things are required for a complete inference service but not included: a KV-cache system, separation of prefill and decode onto different machines, and load balancing across data-parallel replicas. It notes that most of the engine work for disaggregation is already in place.
SiliconANGLE reports that DeepSeek held back the cache and load-balancing code because it is tightly coupled to internal infrastructure, though a Zhihu blog post explains how both work. The smallest deployment DeepSeek describes as efficient uses 64 GPUs for prefill and another 64 for decoding. In other words, this is a toolkit for cloud providers and large AI teams, not something you’ll run on a desktop.
The hardware fine print
- Only Nvidia’s SM100-class Blackwell GPUs are listed. SiliconANGLE notes Hopper and older Nvidia architectures are not supported.
- Huawei’s Ascend 950 is a first-class target, building on DeepSeek’s September 30 release that brought the TileLang kernel language to Huawei chips.
- The determinism features work only on Nvidia hardware, and all published benchmarks were run on Nvidia chips.
DeepSeek’s own numbers
According to SiliconANGLE, a deployment Q&A posted with the code says one DeepSeek cluster serves 155 trillion tokens a day and that revenue at off-peak pricing works out to $38 per GPU per hour. DeepSeek calls serving a massive user base at a reasonable price “the core problem an inference framework has to solve.”
Treat these as company claims. The Next Web points out that no one outside DeepSeek has independently benchmarked DeepInfer’s gains yet, citing Bloomberg’s reporting.
Why DeepSeek is doing this now
This isn’t the first time DeepSeek has considered releasing its engine. In April 2025 it explained that its internal engine was an early, heavily customized fork of vLLM tied to its cluster tools, and that a small research team lacked bandwidth to maintain a big open-source project. At the time it chose to contribute pieces to existing projects instead.
The timing also lines up with policy pressure. DeepInfer arrived the day after DHS ordered agencies to drop DeepSeek products, which we covered in DHS orders DeepSeek removed from federal civilian IT systems. Releasing infrastructure under a permissive license lets hosting companies serve DeepSeek’s open-weight models more efficiently, wherever they’re based. Bloomberg, as cited by The Next Web, noted that companies running the models on their own servers could cut costs.
Early reaction
- The vLLM project thanked DeepSeek on X for crediting the open-source community and said it would keep improving its own V4.1-Flash support with ideas from DeepInfer, per The Next Web.
- The GitHub repository passed 8,000 stars by Tuesday afternoon, the same report said.
- DeepSeek says inference code for future models will land in the same repository.
Who should care
Inference providers and cloud hosts already serving DeepSeek models get production-tested kernels and scheduling ideas. Teams on Huawei Ascend hardware get a serious open framework for a frontier-class model. Most developers won’t run DeepInfer directly, but they may benefit indirectly if hosts pass on lower serving costs. If you’re weighing the model itself, start with our look at DeepSeek V4.1-Flash-1017.
The bottom line
DeepSeek DeepInfer opens up the part of the AI stack that companies usually guard most closely: the software that turns GPUs into answers. It’s narrow for now, with one model, two chip families and key serving pieces held back. But it hands rivals and partners a detailed look at how DeepSeek keeps its prices low, and independent benchmarks will show whether the claimed gains hold up.
The Teck Hustlers Editorial Team covers technology, business, finance and gaming news for readers who want the story behind the headline. Every article is researched, written and edited by our in-house team, checked against company announcements, filings and reputable reporting, and updated when new facts emerge. We focus on what each development means for founders, professionals, investors and everyday users. Have a tip or a correction? Reach us through our contact page.
