Tune — Live Skill RL

Turn the user's reaction into a targeted SKILL.md overlay edit. Human-in-the-loop reinforcement learning for skills.

Architecture

Base SKILL.md (read-only, shared)
    +
Overlay.md (mutable, per-workspace)
    =
Effective skill at invocation time

Path A (cron):  /api/improve ticks weekly, objective reward harvester
Path B (live):  THIS SKILL — user reaction = reward, edits overlay in-session
                Same overlay.md, same scorecard — both paths compound.

The reward schema (non-negotiable)

Signal	Reward	Source
Explicit positive	+1	"good", "perfect", "keep it", "that nailed it"
Implicit positive	+0.5	user kept output unchanged, shipped it, moved on
Neutral	0	no reaction, switched topic
Implicit negative	−0.5	user edited heavily, reran skill, tweaked params
Explicit negative	−1	"shit", "bad", "nope", "revert", "worse than before"

Do not infer reward beyond these. If you are unsure, classify as neutral and log; do NOT propose an edit on neutral.

The loop (one tick)

1. OBSERVE
   - User just invoked skill <X> at turn T
   - Skill produced output Y
   - User reacted with R

2. CLASSIFY
   - Map R to reward from the schema above
   - If reward in {0}: log and stop. Do not propose.

3. DIAGNOSE
   - If reward < 0: what specifically did the user reject?
     (a phrase? a structural choice? a missing section? tone?)
   - If reward > 0: what specifically did the user approve?
     (same questions, positive framing)
   - Write a one-sentence diagnosis.

4. PROPOSE
   - Read /workspace/atris/skills/<X>/SKILL.md   (BASE, read-only)
   - Read /workspace/atris/skills/<X>/overlay.md (if exists, else empty)
   - Propose ONE targeted edit to overlay.md:
     • Max 5 lines of change
     • Must be additive (append) or a bounded replace of existing overlay lines
     • NEVER edits base SKILL.md

5. SHOW THE DIFF
   - Display a unified diff of the proposed overlay change
   - State the diagnosis and expected outcome in one sentence

6. REQUIRE APPROVAL
   - Wait for an explicit "yes" / "apply" / "go" from the user.
   - On anything else: log as rejected. Do not apply.

7. APPLY + LOG
   - Write the approved change to /workspace/atris/skills/<X>/overlay.md
   - Append one line to /workspace/atris/skills/<X>/scorecards.jsonl:
     {"ts": "<iso>", "skill": "<X>", "reward": <n>, "reaction": "<short>",
      "diagnosis": "<short>", "applied": true, "path": "B-live", "turn": <T>}

Validator rules (hard gates)

1. Base is read-only. If the skill folder has no overlay.md, create it. Never edit SKILL.md directly.
2. Diff gate. A diff must be shown to the user BEFORE any write. No silent edits ever.
3. Approval gate. Explicit "yes/apply/go" required. Silence is not approval.
4. Size cap. A single tick may add at most 5 lines to overlay.md. Larger changes require the user to re-invoke with a bigger scope confirmed.
5. Overlay size cap. If overlay.md exceeds 100 lines, refuse new edits and suggest consolidation (point the user at /api/improve or ask them to manually prune).
6. No ALL CAPS, no em dashes, no "AI" buzzword in overlay content (matches outbound copy rules).
7. Log every tick to scorecards.jsonl — approved, rejected, and neutral. Every tick.

Reward harvester (Path B side)

scorecards.jsonl (append-only):
  {"ts", "skill", "reward", "reaction", "diagnosis", "applied", "path": "B-live", "turn"}

Path A (/api/improve) also writes to the SAME scorecards.jsonl with:
  {"ts", "skill", "reward", "path": "A-cron", "variant_hash", "kept": bool}

Weekly: /api/improve cron tick reads the last 7 days of scorecards.jsonl
        to propose autonomous variants that address patterns Path B caught.

Examples

Example 1 — explicit negative on aeo

User: [runs aeo draft, gets output]
User: "this is too hedgy, it uses 'may' twice"

Tune skill:
1. Reward: −1 (explicit negative, specific complaint)
2. Diagnosis: AEO rule 9 (no hedging) is violated; overlay should strengthen it
3. Propose overlay.md addition:

   ```diff
   --- overlay.md (before)
   +++ overlay.md (after)
   @@ +1,3 @@
   +## Stricter no-hedging (learned 2026-04-17)
   +Before returning, scan the body for: may, might, could, perhaps, possibly, potentially.
   +If found, rewrite the sentence as declarative. No exceptions.

4. Diagnosis: "strengthens rule 9 by making the scan explicit post-generation"
5. Expected: next draft has zero hedging words

"Apply this?"
User: "yes"
[writes overlay.md + logs scorecard, reward=−1, applied=true]

### Example 2 — explicit positive on cold-email

User: "damn that cold email hook is sharp, keep doing that"

Tune skill:

1. Reward: +1

2. Diagnosis: the specific hook pattern worked (e.g., "opened with a stat from their 10-K")

3. Propose overlay.md addition:

   +## Hook that worked (2026-04-17)
   +Prefer opening with a verifiable, recent stat from the prospect's own
   +public material (10-K, earnings call, press release) over generic praise.

4. "Apply this as a preferred pattern?"
   User: "yes"
   [writes overlay.md + logs scorecard, reward=+1, applied=true]

### Example 3 — neutral

User: [gets output, says nothing, moves on]

Tune skill: does NOT proactively run. Only triggers on user-expressed reaction.
If somehow invoked: log neutral, propose nothing.

## Anti-patterns

- **Don't propose on neutral signal.** Silence is not −1 or +1.
- **Don't rewrite the base.** Base SKILL.md is shared across customers. Only overlay diverges.
- **Don't batch multiple changes into one tick.** One tick = one targeted edit. Larger refactors go through `/api/improve` in review mode.
- **Don't claim improvement you can't verify.** "Expected outcome" should be a testable prediction (e.g., "next draft will have zero 'may/might/could'"), not a vibe.
- **Don't run without the user's explicit approval.** Every application is gated. No "this seems good, I'll apply it."

## File layout (what exists where)

atris-cli/atris/skills//SKILL.md ← base, shared (genotype)
/workspace/atris/skills//SKILL.md ← customer copy of base (synced)
/workspace/atris/skills//overlay.md ← mutable, RL-tuned (phenotype)
/workspace/atris/skills//scorecards.jsonl ← append-only reward log

**Invocation-time behavior:** skill callers (backend endpoints like `/aeo/draft`, or Claude Code skill system) read `SKILL.md` AND `overlay.md` and concatenate. If overlay.md is missing, treat as empty.

## Member Spec Overlays

Tune may also propose guarded member-spec changes when the target is `atris/team/<member>/MEMBER.md`.
Keep the same approval gate and write to `atris/team/<member>/overlay.md`; do not silently rewrite the base member spec.
Ground every member overlay proposal in causal evidence from `atris/wiki/.causal.json` or reusable patterns from `atris/wiki/.patterns.json`.
The architect member owns these proposals and must keep them advisory until explicit approval.

## Related

- Base pattern: `atris/skills/autoresearch/SKILL.md` (Karpathy keep/revert)
- Path A tick runner: `/api/improve` (`backend/routers/improve.py`)
- PII sanitizer for cross-customer diffs: `backend/rl/skill_extractor.py`
- Reward compiler: `backend/rl/rewards.py`
- Judge infra: `backend/rl/judges.py`
- Feature: `atris/features/skill-rl/idea.md`

## Rules (non-negotiable summary)

1. Never edit base SKILL.md. Overlay only.
2. Always show the diff before applying.
3. Require explicit user approval.
4. Cap a single tick at 5 lines of change.
5. Log every tick — approved, rejected, neutral.
6. Do not propose on neutral reactions.
7. Strip PII before any cross-customer record (use `skill_extractor`).