PERSONAL SYSTEMS PROJECT / CPU REFERENCE
miniGPT.
Train. Infer. Serve.
A language-model systems project spanning Transformer implementation, recoverable training, inference scheduling, KV memory management, and streaming serving.
Live generation on the project website depends on the demo backend being online.
- Context
- Personal engineering project
- Implementation
- Python · PyTorch · CPU
- Focus
- Training · Inference · Serving
From model code to a serving runtime.
miniGPT is my CPU-first language-model systems project in PyTorch. I built a hand-written GPT and extended it into an end-to-end stack: training, exact checkpoint recovery, cached inference, multi-request scheduling, streaming APIs, and installation-level verification.
The model stays small enough to inspect, while the runtime tackles concrete systems problems: who owns a KV block, how a cancelled request releases resources, and what makes a performance comparison meaningful.
Reproducible state
Model, optimizer, configuration, data identity, and random state.
Scheduled execution
Prefill, decode, continuous batching, and shared KV resources.
Explicit lifecycles
HTTP/SSE, bounded queues, cancellation, and release checks.
A checkpoint is more than model weights.
Implemented GPT components in PyTorch, including causal self-attention, LayerNorm, MLPs, residual blocks, and sampling. The training workflow includes AdamW, learning-rate scheduling, validation, and recorded metrics.
Checkpoint v2 captures the model and optimizer alongside resolved configuration, data fingerprints, and global, batcher, and sampling random states. Resume validation checks these identities before continuing a run.
Training and sampling use separate random-number streams. That separation lets me inspect generated output without silently changing the training trajectory; interrupted/resumed runs are compared with uninterrupted reference runs.
Compute scheduling meets memory ownership.
Cached generation and continuous batching
Implemented prefill/decode KV caching and multi-request execution. Continuous batching, length-aware prefill, and token-budget chunking make scheduling an explicit part of the system rather than a loop around independent generation calls.
Paged KV and automatic prefix caching
Built a fixed-size KV block pool with request ownership, reservation, rollback, and release semantics. Prefix caching shares immutable full blocks, tracks active references, and uses LRU eviction for unreferenced shared blocks.
Pressure, cancellation, and recovery
Added whole-request preemption/recompute and lazy reservation to explore cache pressure. Lifecycle checks cover cancellation, failure cleanup, and reuse so that resource policy can be evaluated separately from model correctness.
Make concurrency boundaries explicit.
Exposed generation through HTTP and Server-Sent Events, with a defined subset of the OpenAI-compatible Completions API. A dedicated EngineRunner thread owns model and engine execution; the HTTP layer submits commands rather than touching model state directly.
Bounded command and stream queues provide backpressure. Client disconnects trigger cancellation, while per-request random state and terminal-state handling keep request behavior inspectable.
The delivery path includes a CLI, runtime manifests, project self-checks, wheel/sdist packaging, and fresh-install verification. The repository also contains Windows/Linux CI and stage-bound test and benchmark records.
Inspect the implementation and its limits.
Fresh-process benchmarks retain environment identity, raw replicates, and comparison policy. Correctness and structural work reduction are reviewed separately from wall-clock speedup.
For example, the prefix-cache record demonstrates avoided prefill work but does not support a universal latency or throughput improvement. The result is kept visible rather than converted into a headline percentage.
Scope: a single-machine CPU reference system for training and serving. The implementation demonstrates language-model infrastructure mechanisms, not production-scale GPU or distributed deployment.
- EngineRunner: execution ownership and bounded queues (opens in a new tab)
- Paged KV: allocation and ownership (opens in a new tab)
- Prefix caching: behavior and benchmark evidence (opens in a new tab)
- System scope and release verification record (opens in a new tab)
Implementation snapshot: bed217c · September 3, 2026. Source links above are pinned to that revision. This page summarizes that snapshot and makes no claim about a live service’s availability.