HeyHomees Lab Notes · No. 5 · DeepSeek · Published October 20, 2026
By HeyHomees Newsroom ·
In 30 seconds
- Runs offline inference for DeepSeek-V4.1-Flash with custom kernels and zero-overhead batch scheduling.
- Supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, requiring Linux, GCC ≥ 13.0, Python 3.12, CUDA 13 or CANN 9.2.
- Provides batch-invariant and deterministic outputs, structured generation, and draft-token acceleration.
What is it?
DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services and is released as open source for offline inference with core serving features. It currently supports the DeepSeek-V4.1-Flash model and includes optimized kernels, parallelism strategies, and deterministic sampling.
How does it work?
The engine uses custom kernels such as FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and TileLang to accelerate compute. It offers parallelism like Attention DP, balanced CP, MoE EP + EPLB, and features DSpark with confidence-scheduled verification to accelerate decoding with draft tokens. Asynchronous scheduling overlaps CPU batch scheduling with device execution, and batch-invariant inference keeps results identical across batch compositions.
What does it need to run?
DeepInfer runs on Linux with GCC ≥ 13.0, Python 3.12, and either CUDA 13 for NVIDIA SM100 GPUs or CANN 9.2 for Huawei Ascend 950 NPUs. It requires a KV cache system, disaggregated VE/Prefill/Decode engines, and load-balancing across data-parallel replicas, which are not included in the release.
By the numbers
- CUDA version for NVIDIA: 13
- CANN version for Ascend: 9.2
- GPUs used for prefill benchmark: 4 SM100 GPUs
The honest read
The figures show the hardware and software prerequisites DeepInfer needs (GCC ≥ 13.0, Python 3.12, CUDA 13 or CANN 9.2) and the scale of the benchmark runs (4 SM100 GPUs for prefill, 64 GPUs for decode). No performance numbers (latency, throughput) are reported, and the benchmarks are self-generated by DeepSeek without external validation. The release does not include required components such as a KV cache system or load-balancing infrastructure, so additional engineering is needed to build a complete service.
What we don't know yet
- Exact latency or throughput results for the reported benchmarks
- Support for hardware beyond NVIDIA SM100 and Huawei Ascend 950
- Details of the default model checkpoint size or download size
Sources
- README, GitHub (primary)
Researched and drafted with AI from the sources above; every figure is checked against them by code before it is published, and a line that fails is dropped.