Snakemake Workflow Skill

Build production-grade Snakemake 9 workflows on SLURM HPC with reproducible run
organization, container integration, and battle-tested pitfall avoidance.

When to Use This Skill

Use when the user needs:

• A new Snakemake workflow or additional rules for an existing one
• Debugging a Snakemake/SLURM error (check references/debug_patterns.md)
• A SLURM profile for Snakemake (check references/slurm_profiles.md)
• To convert ad-hoc scripts into a reproducible pipeline

Don't use when the task has <3 steps, no per-sample parallelism, and no
dependency DAG. A bash script is simpler and Snakemake overhead isn't free.

Core Architecture

1. One Rule = One Tool

Each rule wraps exactly one tool or operation. Rules compose vertically via input:
dependencies. Optional rule blocks are gated by config booleans (if USE_FEATURE:).
Adding a rule should never break existing rules.

2. Workflow vs Results Separation

The workflow is a tool; each run is an experiment. Never write outputs into
the workflow directory.

Workflow directory (versioned, reusable):

workflows/{workflow_name}/
├── Snakefile
├── config_template.yaml
├── scripts/                    # Pluggable scripts with argparse CLI
├── profiles/slurm/config.yaml  # Workflow-specific SLURM profile
├── test/                       # Test fixtures (<5 min on cpushort)
│   ├── test_config.yaml
│   ├── test_manifest.tsv
│   └── test_regions.bed
└── CHANGELOG.md

Run root (one directory = one experiment):

{output_dir}/
├── config.yaml              # COPY of config (frozen at run start)
├── run_snakemake.sh          # Exact reproduction command
├── manifest.tsv              # COPY of sample sheet
├── run_metadata.yaml         # Auto-generated (date, versions, samples)
├── results/{rule_name}/{sample}/   # ALL rule outputs
├── benchmarks/               # benchmark: directive outputs
├── qc/                       # QC gate sentinel files
└── logs/                     # ALL rule logs

3. Single output_dir Config Key

All subdirectories derived internally — never add separate config keys for logs,
figures, or matrices:

OUTDIR     = config["output_dir"]
RESULTSDIR = os.path.join(OUTDIR, "results")
LOGDIR     = os.path.join(OUTDIR, "logs")
BENCHDIR   = os.path.join(OUTDIR, "benchmarks")
QCDIR      = os.path.join(OUTDIR, "qc")

4. Externalize Complex Logic

Shell one-liners are fine inline (awk, bgzip && tabix). Externalize when the
shell block needs if/for/while, variable manipulation, or multi-step Python.
Rule of thumb: if you can't understand it in 10 seconds, externalize it.

• Scripts go under workflow_dir/scripts/ with argparse CLI
• Reference via os.path.join(workflow.basedir, "scripts", "script.py")
• Never inline complex Python in run: blocks for SLURM-submitted rules —
  run: executes in the coordinator process, not on the compute node

5. Validate Before Every Submission

After any Snakefile edit:

1. snakemake --lint — catches style issues
2. snakemake -n — dry-run validates the full DAG
3. snakemake --dag | dot -Tpdf > dag.pdf — visualize dependencies

Dry-run is the minimum test. A small-data end-to-end test is preferred.

6. Built-in Resource Management

7. QC Gates as Workflow Rules

QC checks are Snakemake rules, not informal post-hoc steps. Pattern: QC rule
produces a .pass sentinel; downstream rules depend on it.

rule qc_alignment:
    """Fail if mapping rate < 80%."""
    input:
        flagstat = os.path.join(RESULTSDIR, "alignment", "{sample}", "{sample}.flagstat"),
    output:
        qc_pass = os.path.join(QCDIR, "alignment_{sample}.pass"),
    run:
        import re
        with open(input.flagstat) as f:
            text = f.read()
        mapped_pct = float(re.search(r"(\d+\.\d+)% mapped", text).group(1))
        if mapped_pct < 80.0:
            raise ValueError(f"QC FAIL: {wildcards.sample} mapping rate {mapped_pct}% < 80%")
        with open(output.qc_pass, "w") as f:
            f.write(f"PASS: mapping_rate={mapped_pct}%\n")

Gate after: alignment, pileup, DMR calling. Don't gate on soft thresholds — log
and report those instead.

8. Config is the Run Manifest

All cohort-specific details (paths, regex patterns, sample ID formats, exclusion
keywords) are config args, never hardcoded. This enables reuse across cohorts
without code changes. Config documents itself with comments.

9. Container Discipline

• Never use :latest tags — pin exact versions (onttools_v3.9.sif)
• Every rule needing container packages must have singularity: IMG
• Load container paths from softwares_containers_config.yaml — never guess
• If a rule has singularity: directive, do NOT add singularity exec in shell:

10. Test Suite

Every workflow ships with test/ containing test_config.yaml, test_manifest.tsv,
and test_regions.bed. Tests must complete in <5 minutes on cpushort using slurmMinimal profile.

Snakemake 9 + SLURM Critical Pitfalls

These are battle-tested fixes. Memorize them — they cause the most debugging time:

For the full pitfall table and debug patterns, read references/debug_patterns.md.

Reference Files

Read these when you need detailed templates or troubleshooting:

Retry Guidance by Rule Type

Incremental Sample Addition

Adding samples to sample_manifest triggers only new per-sample rules (Snakemake
checks output files). Cohort-level rules re-run because the expand list changed.

• Use the same output_dir when adding samples to an existing cohort
• Force cohort re-run only: snakemake --forcerun build_cohort_matrices
• If output_dir changes, everything re-runs (no prior outputs)

Workflow Completion Checklist

Before declaring any workflow complete, run through the checklist in
references/completion_checklist.md. The critical items:

1. Dry-run succeeds (snakemake -n)
2. Profile has mem_mb: 0 and slurm_account in default-resources
3. All container-dependent rules have singularity: IMG
4. Compute-heavy rules have benchmark: directive
5. unset SLURM_MEM_PER_NODE in run script
6. No hardcoded absolute paths in Snakefile
7. Test suite exists and passes

Validated Reference Workflow

The ont_modkit_pileup workflow serves as the reference implementation:

• 5 conditional rules: convert_bed, pileup, aggregate, cohort_matrices, tensor, correlation
• Key patterns: --include-bed for single-pass pileup, chr prefix auto-detection,
  BAM index auto-detection (.bai, .bam.csi, .csi)
• run: for lightweight cohort steps; shell: + singularity: for compute
• Located at: workflows/ont_modkit_pileup/