AIsaAIsaStudio English 打开 AIsa Studio

sybil-solutions 发布 inference-optimisation-skill:爬山式优化本地 LLM 推理速度的 Agent 技能

原标题:sybil-solutions/inference-optimisation-skill: Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization

GitHub 新仓库 sybil-solutions/inference-optimisation-skill 发布一个用于爬山式优化本地 LLM 服务速度的 Agent 技能,覆盖单用户与并发 decode、投机解码、prefill、KV cache、启动时间和 TTFT,适用于任意硬件和量化方式。技能兼容 Claude Code 及任何加载 SKILL.md 风格技能的 Agent,也提供 PROMPT.md 供任意模型直接执行;工作流包括先算理论天花板、一次只改一项、对照交错控制组测量、通过长上下文稳定性检查等。README 附带来源实测数据,如 Intel Arc Pro B70 上 Qwen3.8-27B decode 1 用户 91.2 tok/s、16 用户 365 tok/s。

为什么值得看

这是一个可复用的 Agent 技能,把爬山式推理提速方法整理成流程,并附各硬件实测数据和完整爬山日志。

阅读原文 ↗

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

原文

GitHub 上的新仓库 sybil-solutions/inference-optimisation-skill,创建 1 天,已有 915 星(平均每天 749 星)。 Agent skill for hill-climbing local LLM serving speed (decode, speculative decoding, prefill, KV, startup, TTFT) on any hardware and quantization 主题:agent-skill, benchmarking, claude-code, inference, llm, speculative-decoding 语言:Python

README:

inference-optimisation

An agent skill for hill-climbing the speed of a local LLM server on any hardware and any quantization. It covers:

  • single-user and concurrent decode
  • speculative decoding
  • prefill
  • KV cache size
  • startup time and time to first token

It works under a memory cap, and every result has to survive a full-context stability check.

It encodes how we actually got these speedups. It is not a list of flags. The workflow:

  1. Compute the theoretical ceiling.
  2. Change one thing at a time.
  3. Measure against interleaved controls.
  4. Log every result, including the failures.
  5. Gate on long-context retrieval.
  6. Qualify a clean, pinned build before publishing.

It works with Claude Code and any agent that loads SKILL.md-style skills. Any model can also follow it from PROMPT.md.

What we've been able to accomplish

Results from the hill-climbs this skill was distilled from. Every number was measured on the hardware named; see the linked sources for methods and raw data.

Hardware Model / quant / engine Before → after
Intel Arc B390 iGPU (laptop, 64 GB unified LPDDR5X, ~115 GB/s) Ornith-1.5-35B-A3B, EXL3 4 bpw, SGLang XPU + MTP Decode, 1 user: 43.6 → 53 tok/s (median)Decode, 4 users: 84.7 → 103 tok/sPrefill 32K: 1,037 → 1,660 tok/sPrefill 256K: 297 → 628 tok/s (2.1×)KV cache: 264K tokens → 4 × 256KStartup: 113 → 43 sRegistry Lab gates: decode 28.6 → 41.0, prefill 255 → 667
Intel Arc Pro B70 Qwen3.8-27B, EXL3 4 bpw, vLLM + custom XPU kernels Decode: 91.2 tok/s at 1 user, 365 at 16 (prose, thinking on)Same card with llama.cpp Q4_K_M: 25.0Prefill 4K/32K/128K/254K: 1,654/1,434/928/650 → 2,421/2,259/1,415/911Agent-session replay: 193 → 147 s
Intel Arc Pro B70 Same model, SGLang + the same kernels Decode, 1 user: 82.6 tok/s, vs vLLM 75.7 on the same card and day
RTX 3090 Qwen3.8-27B EXL3 3 bpw + MTP 96.2 tok/s prose / 141.1 code, 1 user
RTX 3090 Qwen3.6-35B-A3B EXL3 3 bpw + MTP 251.7 / 357.9 tok/s, 1 user
RTX 3090 Flash-Next EXL3, 256K context MTP: 45.5 → 65.6 tok/s
4× H200 NVL DeepSeek-V4.1-Flash, vLLM + speculative decoding Crashed under load → 229.1 tok/s decode, 17,546 tok/s prefill; all six gates pass

Sources:

Reading the B390 result

That run is the one this skill grew out of. Its ceiling calculation said:

  • Decode: 100 tok/s for one user was only reachable with two useful speculative drafts. This model's draft head accepts ~14% at depth 2, so the practical target was ~88–97 tok/s. We reached ~55% of it; the rest is GEMV kernel efficiency.
  • Prefill at 256K: we reached 628 tok/s against a ~670–684 tok/s ceiling.

The skill makes the agent compute this up front, so it can tell you early when a target is above the physics.

How it works

                ┌────────────────────────────────────────────────────────────────┐
  measure HW ─▶ │ THEORY.md: bytes/token ÷ bandwidth, FLOPs/token ÷ 50% peak, KV │
                └────────────────────────────────────────────────────────────────┘
                                         │
    baseline ─▶ profile ─▶ pick lever (ceiling gap × likelihood) ─▶ configs/NEW.json
                    ▲                                                     │
                    │                                    queue.sh ctl NEW ctl (serialized, AC-only,
                    │                                    memory watchdog, ≥6 samples, medians)
                    │                                                     │
   PLAN.md + ledger.jsonl ◀── accept only if it beats BOTH neighbouring controls ──┘
          │                    and regresses nothing by >3%
          ▼
  long-context needle gate ─▶ concurrent full-KV soak ─▶ plateau? ─▶ pinned clean build
                                                                      ─▶ acceptance gates ─▶ publish

The skill has three parts:

SKILL.md: the procedure.

  • Inventory, ceilings, ground rules, the loop and lever order.
  • When to stop, how to qualify and publish, and how to report.

references/: loaded only when needed.

  • ceilings.md: how to compute decode, prefill, TTFT, startup and KV maxima, with a worked example.
  • methodology.md: measurement traps. Chunk-counted tokens, mis-sized prompts, acceptance counters, drift, unified-memory accounting.
  • levers.md: every lever with measured evidence. It is tagged general vs stack-specific.
  • gotchas.md: known traps in vLLM, SGLang, llama.cpp and ExLlama, and on Intel, NVIDIA, Apple and AMD hardware.
  • publishing.md: pinning, clean builds, the six acceptance gates, proving the build matches the best, results cards.

scripts/: a small, engine-agnostic harness. It is standard-library Python plus bash.

  • bench.py benchmarks any OpenAI-compatible server. It measures:
    • startup and sh

原文版权归 GitHub 爆发仓库 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。