Inference. Training. One stack.

Six components. Measured on real hardware. Open for individuals.

36 GB

Without NSKVCache · 1M context · Qwen2.5-3B

168.5 MiB

With NSKVCache · 208.7× compression · measured

Full context KV cache. Tested on Laptop (i7-1165G7, 16 GB DDR4). Compression ratio at capacity.

Most of what accumulates in a KV cache during inference never gets retrieved meaningfully. NSRun addresses this at four layers: context compression, sparse attention, sparse MLP activation, and activation-aware quantization.

Components

Four ship inside NSRun. NSTrainer is the training-side counterpart.

Component	What it does	Number	Context
NSRun	GGUF inference engine	11 tok/s	vs 6.53 baseline · Qwen2.5-3B · CPU
NSKVCache	Context compression	208.7×	168.5 MiB at 1M context · measured
NSAttend	Sparse attention	28.7×	vs full dense · seq_len 8192
NSInfer	Sparse MLP activation	3.04×	standalone · +70% prompt eval at 72B
NSQuant	Activation-aware quant	3.8–7.2×	3B: 1.61 GiB · 72B: 19.4 GiB from fp16
NSTrainer	NS-filtered training	86.8% fewer	steps to match 1000-step baseline

NSRun · NSKVCache · NSAttend · NSInfer · NSQuant ship together. NSTrainer ships separately.

What the full stack is built toward

Components measured independently. Combined end-to-end benchmark pending.

Measured

208.7× KV compression · Qwen2.5-3B

3.0× weight compression · Qwen2.5-3B

11 tok/s full stack · CPU

Architecture ceiling

1,600× KV compression

86.8% training compute reduction · via NS-filtered data + analytical init

Source: cross-layer grouping × MLA latent × INT4 × semantic dedup × filler elimination

Requires MLA-trained model. NSTrainer builds it. NSQuant compresses it.

At production scale, KV cache determines how many users a GPU serves. NSKVCache shrinks it 208.7x. NSTrainer cuts training compute by 86.8%. The hardware implications compound.

Run it yourself

A 3B model. 1M token context. Laptop CPU.

Clone and build

git clone https://github.com/OrdersOfMagnitudeLLC/NSRun
cd NSRun && cmake -B build -DGGML_NATIVE=ON && \
cmake --build build --config Release -j$(nproc)

Download a model

huggingface-cli download \
  Qwen/Qwen2.5-3B-Instruct-GGUF \
  --include "*.Q4_K_M.gguf" --local-dir ./models/

Run with 1M context

./build/bin/llama-cli \
  -m ./models/qwen2.5-3b-instruct-q4_k_m.gguf \
  --kv-box -c 1048576 -b 1024

The model is 2.1 GB. Context window is 1M tokens - roughly 750,000 words - using 168 MB of memory instead of 36 GB. GPU optional. -b 1024 sets batch size. Required for coherent generation at long context.