StudioTuning a single Tesla P40 for Ornith 1.5 35B-A3B: full 262k context at ~60 t/s short, ~30 t/s at 100k depth
Original title: Two days tuning a single Tesla P40: 35B MoE, full 262k context, ~60 t/s short and ~30 t/s at 100k deep. No sub-Q4 quants.
The author tuned Ornith 1.5 35B-A3B (qwen3.5 MoE arch) on a single Tesla P40 24GB (PCIe x1) with a custom llama.cpp fork, holding Q4 weights and q8_0 KV rather than shrinking quants, at full 262k context. Working changes included disabling ECC (+1.5GB VRAM, ~9% decode), int8 dp4a for the flash-attention KQ kernel (prefill +26% at 32k, +41% at 100k), fixing the untrained MTP head (draft acceptance ~45% to 60-85%), and limiting MTP to the last 8k of the prompt (121k prompt: 327 to 373 t/s prefill). Without MTP, decode drops from 60.8 t/s at depth 0 to 29.1 t/s at 100k; with MTP in a real server, 65-74 t/s on short prompts and a 101k-token prompt reads at ~415 t/s and generates at 35-39 t/s. Patches are available on request.
The author shares concrete tuning steps and measured benchmarks for running a 35B MoE with long context on an old Pascal card, with methods transferable to other P40s.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.