GitHub 上的新仓库 sybil-solutions/inference-optimisation-skill,创建 1 天,已有 915 星(平均每天 749 星)。
Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization
主题:agent-skill, benchmarking, claude-code, inference, llm, speculative-decoding
语言:Python
README:
inference-optimisation
An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:
- single-user and concurrent decode
- speculative decoding
- prefill
- KV cache size
- startup time and time to first token
It works under a memory cap, and every result has to survive a full-context stability check.
It encodes how we actually got these speedups. It is not a list of flags. The workflow:
- Compute the theoretical ceiling.
- Change one thing at a time.
- Measure against interleaved controls.
- Log every result, including the failures.
- Gate on long-context retrieval.
- Qualify a clean, pinned build before publishing.
It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.
What we've been able to accomplish
Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.
| Hardware |
Model / quant / engine |
Before → after |
| Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) |
Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP |
Decode, 1 user: 43.6 → 53 tok/s (median)Decode, 4 users: 84.7 → 103 tok/sPrefill 32K: 1,037 → 1,660 tok/sPrefill 256K: 297 → 628 tok/s (2.1×)KV cache: 264K tokens → 4 × 256KStartup: 113 → 43 sRegistry Lab gates: decode 28.6 → 41.0, prefill 255 → 667 |
| Intel Arc Pro B70 |
Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels |
Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)Same card with llama.cpp Q4_K_M: 25.0Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911Agent-session replay: 193 → 147 s |
| Intel Arc Pro B70 |
Same model, SGLang + the same kernels |
Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day |
| RTX 3090 |
Qwen3.8-27B EXL3 3 bpw + MTP |
96.2 tok/s prose / 141.1 code, 1 user |
| RTX 3090 |
Qwen3.6-35B-A3B EXL3 3 bpw + MTP |
251.7 / 357.9 tok/s, 1 user |
| RTX 3090 |
Flash-Next EXL3, 256K context |
MTP: 45.5 → 65.6 tok/s |
| 4× H200 NVL |
DeepSeek-V4.1-Flash, vLLM + speculative decoding |
Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass |
Sources:
Reading the B390 result
That run is the one this skill grew out of. Its ceiling calculation said:
- Decode: 100 tok/s for one user was only reachable with two useful speculative drafts. This model's draft head accepts ~14% at depth 2, so the practical target was ~88–97 tok/s. We reached ~55% of it; the rest is GEMV kernel efficiency.
- Prefill at 256K: we reached 628 tok/s against a ~670–684 tok/s ceiling.
The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.
How it works
┌────────────────────────────────────────────────────────────────┐
measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
└────────────────────────────────────────────────────────────────┘
│
baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
▲ │
│ queue.sh ctl NEW ctl (serialized, AC-only,
│ memory watchdog, ≥6 samples, medians)
│ │
PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
│ and regresses nothing by >3%
▼
long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
─▶ acceptance gates ─▶ publish
The skill has three parts:
SKILL.md: the procedure.
- Inventory, ceilings, ground rules, the loop and lever order.
- When to stop, how to qualify and publish, and how to report.
references/: loaded only when needed.
ceilings.md: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.
methodology.md: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.
levers.md: every lever with measured evidence. It is tagged general vs stack-specific.
gotchas.md: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.
publishing.md: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.
scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.
bench.py benchmarks any OpenAI-compatible server. It measures:
原文版权归 GitHub 爆发仓库 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。