Despite the U.S. Ban, DeepSeek Open-Sources Its Inference Engine
Set Trending Topics as a preferred source on Google.
Washington is shutting DeepSeek out, and DeepSeek is responding with openness: The Chinese A.I. lab has just released DeepInfer, the software it uses to run its own models at scale. The code is available on GitHub under the permissive Apache 2.0 license. What stands out most is the hardware DeepInfer runs on: Huawei chips as well as Nvidia’s.
What DeepInfer Is
An inference engine is the software that actually runs a fully trained A.I. model, taking in requests and generating responses. It largely determines how fast and how cheaply a model runs. According to DeepSeek, DeepInfer is “the high-performance inference framework that powers DeepSeek’s large-scale production services.” At launch, it supports DeepSeek-V4.1-Flash, of which DeepSeek released a faster version just last Saturday. Code for future models is set to land there as well.
On the technical side, DeepInfer brings together several components that DeepSeek has already released individually in recent years, such as the FlashMLA, DeepGEMM and DeepEP libraries. It also includes methods for spreading a model across many chips, a technique that speeds up responses by using drafts, and structured outputs, for example in JSON format. DeepSeek’s software was originally based on vLLM, a widely used open-source project, but has since been heavily rebuilt.
Huawei on a Par With Nvidia
At launch, exactly two hardware platforms are supported: Nvidia’s latest Blackwell-generation data center chips and Huawei’s Ascend 950 A.I. accelerators. As The New Stack notes, older Nvidia chips such as Hopper are missing. That fits DeepSeek’s push to become less dependent on Nvidia. According to Bloomberg, the lab plans to deploy at least 160,000 Ascend chips in a data center in Inner Mongolia to run its models.
The timing is notable. Just this week, the U.S. Department of Homeland Security ordered all federal civilian agencies to remove DeepSeek from their systems, including other vendors’ products built on DeepSeek. At the same time, the United States is also moving against shipments of Nvidia chips to Chinese companies outside China, according to Reuters. With DeepInfer, DeepSeek is showing that it can run its models efficiently on Chinese hardware, and it is making it easier for companies around the world to run DeepSeek models on their own servers.
Not Quite Finished Yet
The release is not complete, though. Key parts needed for large-scale production use are missing, such as management of the cache for previously computed requests, splitting computation stages across different machines and load balancing between multiple instances. Some features, such as reproducible results, only work on Nvidia chips. DeepSeek does not give concrete performance figures, and there are no published benchmarks at all for Huawei chips.
DeepInfer thus competes with established open-source options. Besides vLLM, which originated at the University of California, Berkeley, these are mainly SGLang and Nvidia’s own TensorRT-LLM. Whether companies switch to DeepInfer is likely to depend on how quickly DeepSeek delivers the missing pieces.
