AIsaAIsaStudio 中文 Open AIsa Studio

sybil-solutions releases inference-optimisation-skill, an agent skill for hill-climbing local LLM serving speed

Original title: sybil-solutions/inference-optimisation-skill: Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

A new GitHub repo, sybil-solutions/inference-optimisation-skill, provides an agent skill for hill-climbing local LLM serving speed, covering single-user and concurrent decode, speculative decoding, prefill, KV cache, startup time and TTFT on any hardware and quantization. It works with Claude Code and SKILL.md-loading agents, follows a one-change-at-a-time workflow with interleaved controls and a long-context stability gate, and includes measured before/after results such as Intel Arc Pro B70 decode reaching 91.2 tok/s at 1 user and 365 tok/s at 16 users with Qwen3.8-27B.

Why it matters

A reusable agent skill that packages a hill-climbing inference speedup methodology, with measured results on several hardware setups and full optimization logs.

Read the original ↗

The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.

Original article

GitHub 上的新仓库 sybil-solutions/inference-optimisation-skill,创建 1 天,已有 915 星(平均每天 749 星)。 Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization 主题:agent-skill, benchmarking, claude-code, inference, llm, speculative-decoding 语言:Python

README:

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:

  • single-user and concurrent decode
  • speculative decoding
  • prefill
  • KV cache size
  • startup time and time to first token

It works under a memory cap, and every result has to survive a full-context stability check.

It encodes how we actually got these speedups. It is not a list of flags. The workflow:

  1. Compute the theoretical ceiling.
  2. Change one thing at a time.
  3. Measure against interleaved controls.
  4. Log every result, including the failures.
  5. Gate on long-context retrieval.
  6. Qualify a clean, pinned build before publishing.

It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.

What we've been able to accomplish

Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.

Hardware Model / quant / engine Before → after
Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP Decode, 1 user: 43.6 → 53 tok/s (median)Decode, 4 users: 84.7 → 103 tok/sPrefill 32K: 1,037 → 1,660 tok/sPrefill 256K: 297 → 628 tok/s (2.1×)KV cache: 264K tokens → 4 × 256KStartup: 113 → 43 sRegistry Lab gates: decode 28.6 → 41.0, prefill 255 → 667
Intel Arc Pro B70 Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)Same card with llama.cpp Q4_K_M: 25.0Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911Agent-session replay: 193 → 147 s
Intel Arc Pro B70 Same model, SGLang + the same kernels Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day
RTX 3090 Qwen3.8-27B EXL3 3 bpw + MTP 96.2 tok/s prose / 141.1 code, 1 user
RTX 3090 Qwen3.6-35B-A3B EXL3 3 bpw + MTP 251.7 / 357.9 tok/s, 1 user
RTX 3090 Flash-Next EXL3, 256K context MTP: 45.5 → 65.6 tok/s
4× H200 NVL DeepSeek-V4.1-Flash, vLLM + speculative decoding Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass

Sources:

Reading the B390 result

That run is the one this skill grew out of. Its ceiling calculation said:

  • Decode: 100 tok/s for one user was only reachable with two useful speculative drafts. This model's draft head accepts ~14% at depth 2, so the practical target was ~88–97 tok/s. We reached ~55% of it; the rest is GEMV kernel efficiency.
  • Prefill at 256K: we reached 628 tok/s against a ~670–684 tok/s ceiling.

The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.

How it works

                ┌────────────────────────────────────────────────────────────────┐
  measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
                └────────────────────────────────────────────────────────────────┘
                                         │
    baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
                    ▲                                                     │
                    │                                    queue.sh ctl NEW ctl (serialized, AC-only,
                    │                                    memory watchdog, ≥6 samples, medians)
                    │                                                     │
   PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
          │                    and regresses nothing by >3%
          ▼
  long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
                                                                      ─▶ acceptance gates ─▶ publish

The skill has three parts:

SKILL.md: the procedure.

  • Inventory, ceilings, ground rules, the loop and lever order.
  • When to stop, how to qualify and publish, and how to report.

references/: loaded only when needed.

  • ceilings.md: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.
  • methodology.md: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.
  • levers.md: every lever with measured evidence. It is tagged general vs stack-specific.
  • gotchas.md: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.
  • publishing.md: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.

scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.

  • bench.py benchmarks any OpenAI-compatible server. It measures:
    • startup and sh

The original belongs to GitHub 爆发仓库 and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.