Mem Ingest Worker

This skill is designed to run as an isolated worker; when your harness cannot fork, run it inline and keep its output separate from the caller's.

You ingest source files into the wicked-estate knowledge store so later
knowledge.recall / cited-answer calls can return them with source
citations. This ports the brain-era ingest pipeline (FOLD-2, Phase 5-S7)
onto estate: you read, extract, and chunk (DEC-R: the agent is the
reasoner); the engine writes and ranks (knowledge.ingest /
knowledge.write — deterministic, never calls a model). Estate's embedded
method card skill://knowledge-ingest/SKILL.md is the chunking doctrine
this pipeline follows.

Runtime

Script-backed steps use the wicked-garden launcher: wicked-garden run <plugin-root-relative path, e.g. scripts/…> [args]. It is on PATH after npm i -g wicked-garden; otherwise use npx wicked-garden run …; inside a wicked-crew run it is "$WICKED_GARDEN_ROOT/scripts/wicked-garden".
If none of these is available, or Python 3 is missing, skip the script-backed step, say so, and follow the manual alternative where one is given next to it — never invent the script's output.
Relative paths in this skill are relative to the directory that contains this SKILL.md.

Parameters

• source (required): path to a file or directory
• scope (optional): estate scope, default project:<cwd-basename>
• title (optional): document title, default derived from the filename

The write seam

All writes go through the mem backend (stdio shim → estate MCP):

wicked-garden run \
  scripts/mem/estate_memory.py ingest -

with a JSON body on stdin (chunks are long — never inline them in argv):

{"title": "<doc title>", "chunks": ["<chunk 1>", "<chunk 2>"],
 "scope": "<scope>", "source": "<absolute-or-repo-relative source path>"}

Rules that make chunks worth storing:

• Self-contained — one idea per chunk (a section, a slide, a sheet, a
  table row group). Recall returns chunks verbatim; a fragment that needs
  its neighbours is a bad chunk.
• Source provenance always — source is what a cited answer cites. An
  uncited chunk cannot back wicked-garden-search answer. Never omit it.
• For a single standalone fact, use the write action instead
  ({"content": ..., "class": "concept", "source": ...}).

Step 1 — assess the source

• Single file → Step 2 (text) or Step 3 (binary) by extension.
• Directory → Step 4 (batch).
• Health check when in doubt: ... estate_memory.py health '{}' — on
  "ok": false, stop and report the degrade (estate binary missing or
  store unreachable). Do not fake success.

Text extensions: .md .txt .csv .html .htm .json .py .js .jsx .ts .tsx .sh .rs .go .java .yaml .yml .toml. Binary (vision) extensions: .pdf .docx .pptx .xlsx .png .jpg .jpeg .gif .webp.

Step 2 — TEXT files (deterministic chunking)

1. Read the file.
2. Split:
   • Markdown: on H1/H2 headings; sub-split any section over ~800 words
     at paragraph breaks.
   • Code: one chunk per file (small files) or per top-level
     class/function group; prepend a one-line path — what it is header so
     the chunk stands alone.
   • CSV/other text: paragraph groups of ~800 words; for CSV keep the
     header row with every chunk.
3. Send title + chunks + scope + source through the write seam (one ingest
   call per file).

Step 3 — BINARY files (LLM vision)

You receive the document natively via Read — examine it visually and
extract content as markdown. This is the one genuinely agent-only stage of
the pipeline; everything around it stays deterministic.

• PDF: one chunk per logical section (or every 3–5 pages). Read PDFs
  with the pages parameter for long documents.
• PPTX: one chunk per slide or tight slide group; capture speaker-note
  intent, not just visible text.
• DOCX: one chunk per section heading.
• XLSX: one chunk per sheet; render data as markdown tables (downsample
  huge sheets: header + representative rows + totals).
• Images: one chunk describing the visual content, plus any legible
  text verbatim.

Preserve numbers, names, and tables exactly — a cited answer will quote
them. Then write through the same seam as Step 2, source = the original
binary's path.

Step 4 — directories (batch)

Do NOT ingest files one-by-one in conversation. Write a small Python batch
script to the session scratch dir that walks the tree (skip dotdirs,
node_modules, __pycache__, lockfiles), applies the Step-2 chunking to
each text file, and pipes one ingest JSON per file into
wicked-garden run scripts/mem/estate_memory.py ingest -. Run it, then:

• Report files → chunks counts from its output.
• List the binary files it found and vision-ingest the important ones
  yourself (Step 3), or hand the list back for the caller to choose.

Step 5 — verify (seeded recall)

Pick 1–2 distinctive phrases from the ingested content and confirm they
come back cited:

wicked-garden run \
  scripts/mem/estate_memory.py sources '{"query":"<distinctive phrase>"}'

The matching chunk must appear with the right source. If it doesn't,
say so — never report an unverified ingest as done.

Step 6 — report

• {N} files ingested, {M} chunks written, scope {scope}
• doc_ids returned by the backend
• binary files pending vision ingest (if any were deferred)
• verification: which seeded phrases came back, with their source