State Inspector — debugging Fabric state from the durable store

The thesis you can't derive: the durable store the server already wrote is the
flight recorder. Every commit, every per-scope revision, every read a commit
observed is on disk. cf inspect opens a space SQLite file read-only and
answers who/what/when/why-different questions with no live runtime. Reach for it
whenever you'd otherwise reason about runtime state from the outside — it
replaces a guess with the ground truth the engine itself reads.

Full command reference + flags: packages/state-inspector/README.md. This skill
is the map — what the tool sees, what to trust, and which question each
command answers. Run commands with deno task cf inspect <cmd>; every command
takes --json for machine reading, and a <space> is a DID, a unique
DID-prefix, a space name (resolved the way the runtime derives it), or a path
(local DBs auto-discovered — start with cf inspect spaces).

What it can — and can't — answer

It reads ONE durable store, offline and read-only. That single boundary tells
you when to trust it and when to reach for something else:

It answers authoritatively (the same bytes the engine reads): what is
stored for any entity at any (branch, seq); who/what/when wrote it (per
commit, UTC wall-clock to the second); how a value got there
(history/diff/timeline); whether identities see different values
(overlay); whether the store is internally consistent (anomalous stale
reads); whether the same id agrees across spaces (converge); and the structure
of it all (entities/piece/graph).

It can't — and reaching for it here will mislead you:

• Anything client-side — optimistic writes, cursor lag, what a browser is
  actually rendering. "Converged" / "consistent" describes the durable store,
  not what any client is showing.
• A live or production bug from a local snapshot — a clean local store does
  not explain a prod-only misbehavior; at most it says the local data is
  healthy, so the cause is concurrency / scale / timing or client-side. Staging
  spaces can be fetched with --remote (see below); the dump endpoint is
  deliberately hard-off in production, so a prod space still means copying the
  .sqlite off the box (ssh+scp).
• How long anything took — it has logical order (seq) plus
  second-granularity commit times ("what happened and when"), never latencies or
  durations.
• What was rejected — the engine rejects stale reads before they persist, so
  they are not here. Zero anomalies means consistent, not "no concurrency."
• The live reactive graph — the dependency graph lives in runtime memory only;
  the sole durable scheduler state is the scheduler_basis index (doc-granular
  ids + seqs — no paths, no payloads), empty until the serving loop writes it.
  Entity/commit history always works.
• Change anything — it is read-only; it explains, it never reproduces or
  fixes.

So reach for it whenever you would otherwise guess at durable or multiplayer
state from outside a live runtime; reach for something else for live behavior,
client rendering, performance profiling, or reproducing a bug.

The mental model the output assumes

You will misread the tool without these — they are facts about how memory-v2
stores state, not things a model infers from the data:

• An entity is a tree of top-level paths, not a bare value: value plus
  meta paths argument / result / patternIdentity / internal / schema /
  cfc. The tool classifies an entity (piece / module / stream / schema /
  owned-cell / free-cell) by which paths exist, and resolves lineage from them
  — so "what is this entity" is answerable structurally, and entities /
  piece / graph speak that vocabulary.
• An entity holding no document says which kind of nothing it is. A
  tombstone is its own kind, deleted. unknown is everything else the tool
  cannot make sense of, and its label says which: (undecodable) for a payload
  that does not decode, (no data) for a set that stored none, (absent) for
  an id with no visible row, and {paths} for the one case that DID decode — a
  document whose shape no other kind recognizes. Ask --kind deleted for
  deletions and --kind unknown for trouble; do not read a tombstone as damage,
  and do not read the revision count as evidence either way (it includes the
  delete op). What the entity WAS is not in the listing — it is gone at HEAD,
  and history <id> plus value-at --seq before the delete is what recovers
  it.
• scope_key partitions an entity by identity. The same cell id can hold
  a shared space value AND a per-user:<DID> override AND a
  per-session:<DID>:<sid> override, stored side by side and genuinely
  different. This is where "looks different for me" multiplayer bugs live. The
  runtime resolves what one identity sees by following a link planted at the
  base scope at write time — not by a read-time session→user→space
  fallback (see the honesty contract).
• Multi-USER ≠ multi-SESSION. A cell written by two sessions of one person
  (many tabs/devices) is benign; two distinct principals is real cross-user
  contention. The tool draws this line for you — trust the multiUser flag, not
  the raw session count.
• Two at-rest value formats coexist, both handled: modern data-model
  codec-json (ids of:fid1:…) and legacy plain-JSON sigils (ids of:baedrei…).
  You don't route between them; just know an id's shape tells you the era.
• Reconstruction is engine-faithful, and proven so. State-at-(branch, seq)
  replays through the server's own applyPatch, honors branch inheritance and
  snapshots exactly like read(), and is locked to the real engine by a parity
  test that drives it. So a reconstructed value is what the runtime would read
  — you can build on it without hedging.

The honesty contract — what is ground truth vs. a hint

The tool is deliberate about this, and so must you be when you report findings.
Confusing an approximation for truth is the failure mode that matters here:

• overlay <space> <id> is the ground truth for "who sees what." It shows
  the entity's value in every scope side by side and flags real divergence
  (compared on the raw stored value, depth-complete). For "this cell looks
  different for two users," this is the answer.
• value-at --as <DID> is an APPROXIMATION — the most-specific stored scope
  that holds the id. It cannot, from an id alone, know which declared scope a
  real read targets or follow the base-scope link. Useful as a quick "roughly
  what they see"; never quote it as the runtime read. Prefer overlay.
• conflicts stale-reads are an ANOMALY detector, not lost-update history.
  The engine validates every confirmed read before committing, so a healthy
  store yields zero. A hit means an invariant violation / corruption —
  surface it loudly as such, and read "0 anomalies" as "the store is
  consistent," not "no concurrency happened." (The separate writer-timeline /
  multiUser contention view is normal history.)
• converge / converge-scan are server-view only — durable values
  compared. "Converged" means the stored values agree, not that every client is
  rendering them; client cursor lag and optimistic writes aren't visible here.
• Same id across spaces is usually NOT replica drift. Content-addressed ids
  mean two spaces often hold independent instances of the same pattern, which
  legitimately differ. The scan labels cross-space-linked (real replica →
  drift bug) vs no-cross-space-link (likely independent instance). Don't cry
  wolf on the latter.
• A schema under $link is what the link stores; a $schemaSummary beside
  it describes a schema too large to print. A link's schema is a JSON Schema,
  so true and false are values it can really hold — true selects every
  value, declaring no constraint at all, where a schema with properties in it
  pins a shape at that link. Never read one as the other. The summary is
  { keys, bytes, digest } and is a SIBLING of schema, never a value under
  it, because a link can store a schema of any shape and a summary in the
  schema slot could be a schema some link really holds. The two never both
  appear: a schema key means that is the stored schema, a $schemaSummary key
  means it was too large to print, and neither means the link stores no schema.
  Different digests prove two schemas differ; equal ones make agreement
  overwhelmingly likely without proving it, since the hash is truncated — reach
  for --full-depth when you need certainty or the text itself. An absent
  digest could not be computed, so two summaries that both lack one say
  nothing about whether they agree; an absent keys means the stored schema was
  not an object and so had none.

Which question → which command

Start from the symptom; let the model below generate the path rather than a
fixed recipe. The recurring debugging questions and where they resolve:

• "What's actually in this space / what is this entity?" → spaces →
  summary / entities / piece; graph for how pieces, cells, modules and
  streams connect (--root <id> --depth to drill, --dot for Graphviz).
• "This cell looks different for user A vs B." → overlay <space> <id>
  (truth); scopes / users to see who has per-user/session state;
  identity <DID> for one identity's whole world (its spaces + the scopes it
  owns).
• "A write seems lost / overwritten / a read was stale." → conflicts <space>
  to find contested cells, then conflicts <space> <id> for the writer
  timeline + the anomaly analysis. Remember: a clean result means consistent,
  and a hit is an anomaly worth escalating.
• "How did this entity reach its current value?" → history (every write,
  who), timeline <space> <id> (value after each write),
  diff <space> <id> --from --to (what changed between two seqs),
  value-at … --seq (state at a point).
• "What collaborative operation history is retained, and are its checkpoints
  healthy?" → operations <space> [id] for field epochs, cursors, submissions,
  integrated rows, retained floors, and consistency checks.
• "Is this space writing more than it should be / has it settled?" →
  churn <space> (commits + revisions per time bucket, and the entities driving
  the busiest one). hot ranks by all-time writes and so cannot separate a
  burst from the same writes spread over a week — churn is the shape-in-time
  view: a storm starting, and a settle completing. Use it before and after a
  setsrc on a populated space; --bucket/--since/--until frame the
  window. Pass --until with the moment you stopped watching: the curve then
  covers the window you asked about rather than ending at the last write, which
  is the difference between showing a storm and showing it SETTLE. It reports
  rates and never judges them — what counts as "settled" is yours to decide.
• "Did a migration preserve this space's content?" → cf space (a sibling
  command, not inspect — a clone exists to be written to, so it lives outside
  inspect's read-only contract). `cf space clone --from --to` builds a writable rehearsal copy plus a manifest; `verify` reports what moved against that baseline and `reset` restores it for the next attempt. Two things decide whether a rehearsal means anything, and both are easy to get wrong: `verify` is strict by default (any change exits nonzero, right for an untouched clone, wrong after a migration — pass `--expect-migration`, which gates on entities REMOVED instead), and it cannot see a clobber either way, because overwriting authored content is a change rather than a removal. Stop the server before `reset`: unlinking the database does not reach a process holding it open, so `reset` refuses rather than let a served clone keep serving the discarded attempt — a tripwire for a forgotten stop, not a substitute for it. Full procedure: `docs/development/space-clone-rehearsal.md`. `cf space fingerprint ` runs the content check alone against any store. Compiler-generated internal cells are excluded by default: a pattern update rotates their identities on purpose, so counting them would change the fingerprint on every legitimate migration.
• "Is this entity the same across spaces / did a replica drift?" →
  converge <id> --all / converge-scan --all, trusting the
  replica-vs-instance label.
• "I want to explore everything for a space interactively." →
  html <space> --out file.html (self-contained: tree + graph + detail pane
  with value, schema, CFC labels, lineage, links, source; --app-url adds live
  deep-links).
• "The space I need is on staging/a server, not local." → add --remote [url]
  (defaults to CF_API_URL) to ANY command: it downloads a read-only snapshot
  into ~/.cache/cf-inspect/<host>/ and inspects it offline. spaces --remote
  lists what's dumpable; pull --all --remote caches everything so subsequent
  commands run offline. Signs with CF_IDENTITY/--identity; the server gates
  on a DID allowlist.

Gotchas that will mislead you if unflagged

• scheduler_basis is the only durable scheduler state (besides the
  watermark machinery; serving-loop.md §3b). The retired
  persistentSchedulerState observation tables were deleted in server-execution
  v2 stage C.2, and the tool no longer reads them even from an old snapshot that
  still contains them. The entity-history surface always works; the basis table
  absent (pre-migration store) or empty is normal, not a broken DB.
• A capped result announces itself — and the caps that don't are the ones to
  watch. The space-wide scans (entities, graph, html) print a cap notice
  on stderr in both modes, so silence there means you hold the whole set; a
  --json consumer that discards stderr discards the only warning it gets.
  entities --kind selects during the scan, so --limit counts entities of
  that kind, not entities walked to find them. Scripting a backup or rollback
  payload? Pass --require-complete — a capped scan then exits nonzero with
  nothing on stdout. Check that status where it is produced: a shell pipeline
  reports its LAST command's status, so a scan piped into jq and a redirect
  refuses, and the redirect still writes an empty file and reports success.
  Capture the scan on its own, or set pipefail. It refuses for either kind of
  incompleteness — a cap reached, or an entity that would not reconstruct, which
  a higher --limit never recovers. The other caps stay silent —
  history/hot/contention row limits, and the HTML stale-read pass, which caps
  per bundle and marks un-analyzed cells rather than showing them clean. There
  a count equal to a round cap is still the tell.
• --json is the agent path. Human output elides; for anything you parse or
  chain, pass --json.
• It reads DBs it didn't write. A corrupt/partial row degrades that one
  entity, not the whole command — but if a value looks absent where you expect
  data, check for a decode error before concluding the entity is empty.