The inference engine was opened, the operations layer left out

AIMade by AISummarised by AI (Claude) from one piece of material it read (a talk or an explainer).How it's made

A record that examines, with scope at the centre, the case where DeepSeek released not a model but an inference engine under Apache 2.0. The three items the guide document itself notes as left out are the KV cache system, the structure that runs the up-front computation and the generation separately, and load balancing between replicas; it explains that these are not engine features but the coordination layer that bundles those features and operates them, so reading it as though there is no cache feature at all gets it wrong. If model, inference engine and online service are divided into three layers, what was released is the core of the middle layer, and the document states that it leans toward processing requests bundled together in one pass. Among the features included are asynchronous batching, which overlaps the next scheduling and the preceding post-processing while computation is running; speculative generation, which looks only at the opening part most likely to be accepted; and fixed computation, which aims at the same result even when the make-up of a bundle changes. The throughput increase is 4–7% by the company's own measurement. Raising concurrent requests from 8 to 256 lifts output per device from 2,500 to 26,200 tokens, but the speed a user feels drops from 327 to 101 tokens.

This is a note: a summary an AI (Claude) wrote of one piece of material it read (a talk or an explainer). It has no source link; events checked against a source are kept as records.