ServingStudio Timing Prediction

A ServingStudio timing prediction estimates how long one deployment takes to
run batches you specify. A deployment is one member of a public preset: a
checkpoint, an arch type, a GPU, and one value of each parameter the preset
sweeps, such as tp_size or the MoE workload. You give it cases, each a batch: which
requests prefill how many tokens and which requests decode at what context
length. The service composes measured kernel times along the deployment's cost
tree and returns each case's time, split by section and by tree node. Use this
skill to answer questions such as "how long is one decode step of 64 requests
at 8k context for Qwen3-235B-A22B-FP8 on 8 H200s?"

The API is JSON over HTTPS. Set the base URL once:

API=${SERVINGSTUDIO_API:-https://servingstudio.cs.washington.edu/api/public/v1}
curl -s "$API/health"

If /health does not return {"status": "ok", ...}, ask the user for the base
URL; do not guess another host. $API/openapi.json lists every route.

1. Find a deployment

/models lists every checkpoint with its presets, one per arch type. A preset
has axes (each swept parameter and its values) and members, one per
combination the preset publishes; not every combination of axis values is a
member, so pick from the list. List the members:

curl -s "$API/models" | jq -r '.checkpoints[] | .presets[] | .id as $id | .gpu as $g | .members[]
  | [$id, $g, (.params | tojson), .gpus_per_replica, (.predict | tojson), (.missing | tojson), .error] | @tsv'

Each member gives:

• params: its axis values. A prediction names the member by these, exactly.
• gpus_per_replica: the GPUs one replica of this deployment occupies.
• predict: the case shape it accepts. selector is iter,
  speculative_iter, attn or ffn; groups is how many groups each case
  must have; max_model_len is the longest request a case may hold (every
  member but an FFN side, which holds no context); a speculative member also
  gives query_width.
• missing: profile.db rows its kernels lack, {kind: count}. Only a member
  with an empty missing can be predicted.
• error: why the member does not build, or null.

Read the preset's gpu and axes too. For an MoE preset, the workload
axis's rows show what each workload binds: routing, and the capture it
reads (expert_popularity_file or token_corpus_file, an hf://datasets/...
reference). Prefer a workload that reads a capture; use uniform or random
only when the user asks for it, and name the workload in your answer.

When several members fit the question:

• max_model_len: take the smallest value at or above the longest request in
  your cases. It enters the attention and indexer kernels' configs (logits
  width, blocks per request), so a larger one predicts different kernels, not
  the same ones with headroom.
• mtp_mode (speculative GLM presets): how the MTP proposer layer runs its DSA
  indexer. full_index computes the indexer on every proposer pass;
  index_share computes it on the first pass and reuses that top-k on the
  passes after it.
  Pick the one the user's deployment runs; otherwise report both, from
  members that share every other axis value.
• A preset with no axes has one member; post "params": {}.
• Prefer an iter preset for "how long is one step": it times the whole
  replica. attn and ffn presets time one side of a disaggregated
  deployment (step 5).

curl -s "$API/models" | jq '.checkpoints[].presets[]
  | select(.id == "Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn") | {gpu, axes}'

case_fields in the same response documents every field of a case for each
selector:

curl -s "$API/models" | jq '.case_fields'

2. Read the member's kernels

The tree route takes the preset's id from /models, which has the form
{checkpoint directory}/{arch}, and every axis as a query parameter:

curl -s "$API/models/Llama-3.1-8B/llama3_dense_tp/tree?tp_size=2" > tree.json
jq '{arch, gpu, gpus_per_replica, predict, missing}' tree.json
jq '.kv_cache' tree.json   # the arch's KV layout; bytes over attention_shards GPUs
jq -r '.sections[] | .section as $s | .slots | to_entries[]
  | [$s, .key, .value.name, .value.kernel, .value.config] | @tsv' tree.json

sections holds one entry per section (iter, attn, or the five ffn
sections). Its slots are the section's kernel leaves: slot j gives its
name, its kernel kind, the backends it may use, and its config. The
tree that combines them comes with each prediction (step 4), where every leaf
names its slot.

configs maps each config id to its kernel kind, its scalar identity, the
names of its structured fields, and missing, the rows this member asks of
it and lacks. /kernels/{kernel}/configs/{config} serves the same config,
with the args it sweeps (swept), how many cells each backend measured (measured), and each
cell's metrics (points):

curl -s "$API/kernels/single_gemm/configs/5604a34006bb60d2" | jq '{kind, gpu, identity, swept, measured}'

3. Build the cases

cases is a list of 1 to 64 cases. Each case's form depends on the member's
predict.selector, and its group count must equal predict.groups; read both
from /models or the tree rather than deriving them from parallel sizes.

selector	Case	Each group
iter	{"groups": [...]}	prefill_chunk_pairs [[prefix_len, append_len], ...], plus either decode_kv_lens [kv_len, ...] or decode_count with average_decode_length
speculative_iter	{"groups": [...]}	prefill_chunk_pairs, plus decode_requests [[kv_len, query_width], ...]
attn	{"groups": [...]}	as for iter
ffn	{"tokens_per_group": [n, ...]}	one token count per group

A prefill pair is the tokens already in the KV cache and the tokens this
iteration computes; [0, 2048] is a fresh 2048-token prompt. A decode KV
length is the context the new token attends to. A group may hold only
prefills, only decodes, or both. The examples below use members of the public
presets; each case list goes into the request body of step 4.

Dense, whole iteration (Llama-3.1-8B/llama3_dense_tp, tp_size: 2;
iter, 1 group). Three decodes, then a 2048-token prefill batched with 32
decodes at 4096:

[{"groups": [{"decode_kv_lens": [1024, 2048, 4096]}]},
 {"groups": [{"prefill_chunk_pairs": [[0, 2048]], "decode_count": 32, "average_decode_length": 4096}]}]

MoE with a capture workload (Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn,
attn_tp_size: 4, ep_size: 8, workload: ctx8k_out1k; iter, 2 groups, one
per attention DP group). Each group is one DP group's batch, so a replica
batch of N requests is split across the groups. Here the replica decodes 128
requests, 64 per group; then one group prefills an 8k prompt while the other
decodes one request:

[{"groups": [{"decode_count": 64, "average_decode_length": 8192},
             {"decode_count": 64, "average_decode_length": 8192}]},
 {"groups": [{"prefill_chunk_pairs": [[0, 8192]]}, {"decode_kv_lens": [8192]}]}]

Speculative (GLM-5.3-NVFP4/glm53_vllm_nvfp4_dsa_moe_dflash2,
max_model_len: 8192, workload: diverse_100; speculative_iter, 1 group,
query_width 8). Each decode request is [kv_len, query_width]: its KV length
once the verify step has run, and the member's query_width (draft tokens + 1).
A request with 4096 tokens of context before the step is [4104, 8]:

[{"groups": [{"decode_requests": [[4096, 8], [4096, 8], [6000, 8], [2000, 8]]}]},
 {"groups": [{"prefill_chunk_pairs": [[0, 2048]], "decode_requests": [[4096, 8]]}]}]

Attention side (Qwen3-235B-A22B/qwen3_attn_tp, attn_tp_size: 4;
attn, 1 group):

[{"groups": [{"decode_count": 64, "average_decode_length": 4096}]},
 {"groups": [{"prefill_chunk_pairs": [[0, 4096]]}]}]

FFN side (Qwen3-235B-A22B-FP8/qwen3_fp8_ffn_moe, attn_tp_size: 4, ep_size: 8;
ffn, 2 groups). Tokens each attention group sends to the FFN this pass:

[{"tokens_per_group": [64, 64]}, {"tokens_per_group": [4096, 128]}]

A member rejects a prefill whose prefix_len + append_len exceeds its
predict.max_model_len, and a decode KV length above it. A
speculative kv_len must also be at least query_width.

4. Predict

Post the preset id, the member's params, and the cases. Each params value
must equal the member's value; a number given as a string also matches.

curl -s -X POST "$API/predict" -H 'content-type: application/json' -d '{
  "preset": "Llama-3.1-8B/llama3_dense_tp",
  "params": {"tp_size": 2},
  "cases": [{"groups": [{"decode_kv_lens": [1024, 2048, 4096]}]},
            {"groups": [{"prefill_chunk_pairs": [[0, 2048]], "decode_count": 32, "average_decode_length": 4096}]}]
}' > prediction.json
jq -c '.cases | to_entries[] | .key as $c | .value.sections[] | [$c, .section, .layer, .total_ms]' prediction.json

The answer has sim_commit, cases, one entry per case in request order,
and kernel_time_share, pooled over all the cases: each kernel position's
time on the critical path with its layers summed, largest first, and the sums
by kernel kind. Each case's sections gives, per section:

• total_ms and energy_j, the section's time and modeled energy;
• nodes, the case's cost tree as the Analyzer reads it (analyze gen-iter-breakdown), root first in display order. Each node has its
  node id, kind (sum, max, scale or leaf), depth, label, ms (one call), total_ms (ms times
  every enclosing scale node's n), pct (total_ms over the root's), and
  critical (on the chain of slowest children from the root, the rows
  iter_breakdown.ans marks with ▸). Under a sum every child adds to the
  time, so critical only marks the largest one; rank nodes by total_ms or
  pct to say where the time goes. A leaf names its slot. Runs of identical
  siblings, such as the same kernel on each rank, show one representative node
  with copies (and, under a max, avg_total_ms); its time is one copy's,
  not their sum;
• slot_backend[j], the backend chosen for kernel leaf j; null means the
  leaf did not run for this case (for example the prefill kernel in a
  decode-only batch);
• coverage, the slot indices whose time did not come from inside the
  measured grid, by flag: extrapolated (outside the grid), jit or
  no_coverage.

Where the time goes:

jq -r '.cases[1].sections[0].nodes[]
  | [.ms, .total_ms, (.pct * 10 | round / 10), ("  " * .depth) + .label] | @tsv' prediction.json

Errors carry detail:

Status	Cause	detail
400	params miss an axis, name an unknown one, or match no member	{message, choices}; choices lists every member's params
400	Bad cases: empty, more than 64, wrong group count, unknown field, a length past max_model_len, a wrong query width	the simulator's message, naming the case; for a length past max_model_len, {message, too_long: {max_model_len, pool: null, requests: null, total: null}}
404	Unknown preset id	no public preset '...'
409	The member does not build, lacks profile.db rows, or a case needs a row nobody measured	the build error or the missing rows; the service never profiles, so report it and pick another member or shape

curl -s -X POST "$API/predict" -H 'content-type: application/json' \
  -d '{"preset": "Llama-3.1-8B/llama3_dense_tp", "params": {"tp_size": 3}, "cases": [{"groups": [{"decode_kv_lens": [1024]}]}]}'
# {"detail":{"message":"Llama-3.1-8B/llama3_dense_tp has no member {'tp_size': '3'}","choices":[{"tp_size":1},{"tp_size":2},{"tp_size":4},{"tp_size":8}]}}

Analyze and keep a prediction

Add "analyze": true to the request. The answer gains a prediction_id, and
the Analyzer serves the prediction under $API/analyzer/predictions/{id}/ for
six hours.

ID=$(curl -s -X POST "$API/predict" -H 'content-type: application/json' -d '{
  "preset": "Qwen3-235B-A22B-FP8/qwen3_moe_fp8_dp_attn_ep_ffn",
  "params": {"attn_tp_size": 4, "ep_size": 8, "workload": "ctx8k_out1k"},
  "cases": [{"groups": [{"decode_count": 64, "average_decode_length": 8192},
                        {"decode_count": 64, "average_decode_length": 8192}]}],
  "analyze": true}' | jq -r .prediction_id)
P="$API/analyzer/predictions/$ID"
curl -s "$P/descriptor" | jq '{selector, case_count, gpu, lifecycle}'

Case ids are positions in cases (0, 1, ...). Operation ids are positions
in a case's operations, one per section: 0 for iter, 0 to 4 for the
ffn sections. The useful routes under $P:

Route	Gives
subjects/cases/payload	each case's input, sections (operations) and total
cases/{case}/operations/{op}/subjects/cost-tree/payload	the section's tree with each leaf's kernel config, backend, exact input, time, FLOPs, bytes and achieved TFLOPS or GB/s, each composite's own time (ms) and, for a max or parallel, its critical child (critical); time_share gives each kernel position's share of the section's time on its critical path
cases/{case}/operations/{op}/leaves/{leaf}/subjects/kernel-throughput-analysis/payload	one leaf's measured grid points around its exact input; {leaf} counts the cost-tree payload's nodes in preorder, root 0
cases/{case}/subjects/optimality-waterfall/payload	the case's GPU-seconds (time × GPUs) split into necessary hardware work and the gaps: batching, communication, imbalance, hardware gap
cases/{case}/subjects/optimality-kernel-ladder/payload	the same rungs per kernel position
subjects/kernel-input-distribution/payload	the inputs each kernel position saw across all cases
subjects/scoped-optimality/report?path=iter/root/1	optimality of one subtree; path is the section and child ordinals from the root (or pass label)

Slots outside the measured grid, by name:

jq -r '.cases[0].sections[0] | .coverage.extrapolated as $x
  | .nodes[] | select(.slot != null and (.slot | IN($x[]))) | [.slot, .label] | @tsv' prediction.json

curl -s "$P/subjects/cases/payload" | jq -c '.cases[] | {case_id, total_time_ms}'
curl -s "$P/cases/0/subjects/optimality-waterfall/payload" | jq '{meta, level: (.level | {total, buckets})}'

The waterfall's buckets are hardware_optimal, hardware_gap (best measured
kernel vs spec sheet), power_gap (the rate a GEMM or copy sustains back to back
under the power limit vs its best measured rate), kernel_gap (this kernel vs
that sustained rate), communication, imbalance, batching and idle, in GPU-seconds. Check
meta.necessary_work_available and meta.caveats first: when the Analyzer
cannot label the iteration's necessary work exactly, it estimates it
(meta.necessary_work_mode) and says why in caveats; pass that on with the
numbers.

An unknown or expired id answers 404 prediction_not_found.

5. Interpret and report

What a time is depends on the selector:

• iter: one forward iteration of the whole model on one replica
  (gpus_per_replica GPUs), every group in the case included. Section iter,
  layer -1.
• speculative_iter: one verify iteration of the target model together with
  the proposer pass the tree shows (a root child such as unified.dflash2 or
  MTP layer N). Report the proposer's total_ms beside the case total, so
  the verify and draft costs read separately.
• attn: one layer of attention on one attention shard (layer 0). Every
  layer sees the same batch, so the attention work of an iteration is this
  time times the layer count, which the root label states (94 layers).
• ffn: building blocks of one iteration on the FFN pool: prologue
  (embedding), pre_attn (layer 0's QKV), post_attn for one representative
  middle layer (which stands for each layer but the last), post_attn_last,
  and epilogue (final norm and LM head). One iteration is
  prologue + pre_attn + (layers - 1) × post_attn + post_attn_last + epilogue.

The attn and ffn presets model the two pools of an attention-FFN
disaggregated deployment. The transfer between the pools is not in either
tree, so report the two sides separately. The Analyzer's optimality subjects
sum the sections they are given, so for ffn they cover the five building
blocks once, not a full iteration.

A predicted time is the sum, max and repeat of kernel times along the tree.
It includes in-tree communication (all-reduce, MoE dispatch and combine) but
no CPU scheduling or launch gaps between kernels. It is not a request latency:
there is no queue, no scheduler, no arrival process, and no TTFT or TPOT.
Those need a simulation of the request stream, which /predict does not run.

When you answer, name the preset, the member's params (with the MoE
workload and whether it reads a capture), the GPU and gpus_per_replica, the
exact cases, each case's total_ms, and sim_commit. Mention any slots under
coverage.extrapolated, since their times lie outside the measured grid. Quote
only times the API returns; label any arithmetic on them, such as scaling an
attn time by the layer count, as your own.