marin-community

run-research

Run a coordinated multi-session research program only when explicitly requested; maintain a logbook and durable snapshots.

marin-community 4,005 324 Updated 2mo ago
GitHub

Install

npx skillscat add marin-community/marin/run-research

Install via the SkillsCat registry.

SKILL.md

Multi-session research

Long-lived work should leave a durable record: a logbook and enough
commands/config to reproduce results. When the work has a coordinating issue,
keep its summary current. Do not publish secrets or private work the user asks
to hold back.

Use background-research, task-logbook, wandb-reporting, task-snapshot,
and update-docs for their named artifacts. Add a domain skill only when the
research directly requires its workflow.

Core Artifacts

  1. A logbook at .agents/logbooks/<topic>.md.
  2. A living hypothesis queue in the logbook, derived from append-only entries
    and updated as hypotheses are proposed, blocked, falsified, or promoted.
  3. A long-lived branch, for example research/<topic> or
    research/<user>/<issue>-<topic>, with the logbook, research code, configs,
    small artifacts, and test harnesses needed to reproduce results.
  4. One or more commit or tag snapshots for meaningful milestones.
  5. Often a "production" branch that gets PR'd and merged.

Standard Workflow

1. Prologue

  1. Keep an existing user-specified branch; otherwise create or use a long-lived
    research/<topic> or research/<user>/<issue>-<topic> branch.
  2. If a coordinating issue exists, link it bidirectionally with the logbook.
  3. Choose one short experiment ID prefix and use IDs such as MOE-HC-001 in
    entries, runs, and comments; use two to four shared tags.
  4. Record the goal, success and stop criteria, baseline, initial hypothesis
    queue and experiment matrix, relevant code, and references.

2. Research Loop

  1. Forage: gather prior work and local context.
  2. Propose: update the living hypothesis queue and pick the next test.
  3. Run: implement the smallest useful experiment and collect evidence.
  4. Interpret: compare against baseline, decide confidence, and update the
    logbook.
  5. Promote: move decision-relevant claims into the current summary and a
    coordinating issue when one exists.
  6. Seal: snapshot durable results or extract production work.

2.1 Forage: Background Research

Use background-research at the beginning, after a significant change of
direction, or whenever you hit a wall. Assume medium or low effort unless
the decision is expensive.

Use the background-research output to update the logbook's hypothesis queue:
add new candidates, revise weak ones, mark known dead ends, and promote
well-supported ideas into the next experiment matrix. If a coordinating issue
exists, let background-research and task-logbook decide what belongs there.

2.2 Propose / Run / Interpret: Dev Work and Experiments

For research-branch dev work, optimize for learning speed while preserving
operational security and cost controls. Ad-hoc scripts, temporary config knobs,
and copy/paste are acceptable there. Production-facing code keeps the usual
AGENTS.md quality bar.

For each non-trivial experiment:

  1. Do the dev work needed for the experiment.
  2. Run the benchmark or experiment. Use the use-iris skill for long-lived runs.
  3. Append exact commands, config, key outputs, interpretation, and next decision
    to the logbook. Follow task-logbook for issue updates when an issue exists.
  4. Push dense scalar series, plots, or raw artifacts to W&B or another store
    when they are too large for issue comments.

3. Epilogue: Seal

Seal when requested or when the defined goal is reached.

  1. Update the logbook's final TL;DR, conclusion, decision log, and
    negative-results index.
  2. If a coordinating issue exists, update its body and add a final comment
    covering what worked, what did not, confidence, limitations, and ordered
    next steps. Follow task-logbook.
  3. Use update-docs when behavior, operational practice, reusable guidance, or
    durable research findings changed.
  4. Ensure the final logbook entry and snapshot links are present.
  5. Close the coordinating issue when one exists and the research thread is
    complete.

If the research produced useful production changes, extract them into a clean
branch that links the research record without including it. Before finishing,
update the TL;DR and conclusion, list next steps, and link the final snapshot
and production PR when one exists. Add the same links to the coordinating issue
when one exists.

Practical Rules

  • Prefer short-lived code changes unless a persistent harness is clearly useful.
  • Keep benchmark harnesses configurable and minimal.
  • Record exact command lines for every headline number.
  • Treat failures and negative results as first-class data. Record dead ends and
    excessive hyperparameter sensitivity; skip routine bugs or undertuning.