Nushell for Data Manipulation

Use nu -c '...' instead of python3 scripts or complex jq pipelines
for structured data tasks.
Nushell treats all data as structured tables, records, and lists --
no JSON parsing boilerplate needed.

JSONL Processing

JSONL files (one JSON object per line) are the most common data format
in agent transcripts and logs.

# Parse JSONL and work with it as a table
nu -c 'open file.jsonl | lines | each { from json } | ...'

The open ... | lines | each { from json } pattern
converts JSONL into a nushell table.
From there, use standard table operations.

Count occurrences by field

nu -c 'open data.jsonl | lines | each { from json }
  | where type == "tool_use" | get name
  | uniq --count | sort-by count --reverse'

Filter and select fields

nu -c 'open data.jsonl | lines | each { from json }
  | where type == "tool_use" | select name content'

Aggregate numeric fields

nu -c 'open data.jsonl | lines | each { from json }
  | where type == "tool_use" | get usage.input_tokens | math sum'

Format Conversion

Nushell's open, to, and save commands handle format conversion
without import statements or serialization gymnastics.

# JSON to CSV
nu -c 'open data.json | to csv | save data.csv'

# CSV to JSON
nu -c 'open data.csv | to json | save data.json'

# YAML to TOML
nu -c 'open config.yaml | to toml | save config.toml'

open auto-detects format from file extension.
to csv, to json, to yaml, to toml handle serialization.
save writes to file.

Convert with filtering

nu -c 'open data.json | where calls > 10 | select tool calls | to csv | save filtered.csv'

Filtering and Aggregation

where filters rows. get extracts columns.
Math commands (math avg, math sum, math min, math max) aggregate.

# Filter rows by condition
nu -c 'open data.json | where calls > 10'

# Compute derived columns with insert
nu -c 'open data.json | where calls > 10
  | insert avg_tokens { |r| $r.total_tokens / $r.calls }
  | select tool calls avg_tokens'

# Aggregate a column
nu -c 'open data.json | get total_tokens | math avg'

When NOT to Use Nushell

Use nushell for: multi-record data, format conversion, aggregation.
Do NOT use nushell for these tasks:

Text search across files -- use grep/rg.
Nushell's glob | each { open | lines | where } is slower and far more verbose.

# WRONG
nu -c 'glob **/*.nix | each { |f| open $f | lines | where {|l| $l =~ "pattern"} } | flatten'
# RIGHT
rg "pattern" --glob '*.nix'

Single field extraction from one JSON object -- use jq.
When you have one object and need one field, jq is shorter.

# WRONG
echo '{"a":{"b":"c"}}' | nu -c '$in | from json | get a.b'
# RIGHT
echo '{"a":{"b":"c"}}' | jq -r '.a.b'

Binary data or file downloads -- use curl.
Nushell's HTTP commands cannot save binary files directly.

Tasks requiring external libraries -- use python3.
Statistical analysis, ML, or anything needing pandas/numpy.

Anti-Patterns

Never write python3 boilerplate for JSONL parsing:

# WRONG -- wasteful boilerplate
import json
with open("file.jsonl") as f:
    for line in f:
        obj = json.loads(line)
        if obj.get("type") == "tool_use":
            ...

Never chain jq with sort/uniq for counting:

# WRONG -- unnecessary shell pipeline
jq -r 'select(.type == "tool_use") | .name' file.jsonl | sort | uniq -c | sort -rn

Both are replaced by a single nushell pipeline.

Syntax Gotchas

Nushell is not POSIX sh. These are the most common mistakes:

save requires -f to overwrite existing files.
Without it, nushell errors on existing files.

nu -c '... | save -f output.csv'    # overwrites
nu -c '... | save output.csv'       # errors if file exists

JSONL is not auto-detected by open.
Always use open file.jsonl | lines | each { from json }.
Do not use open file.jsonl alone -- it reads the file as plain text.

No > redirection. Use save instead.

# WRONG
nu -c 'open data.json | to csv' > out.csv
# RIGHT
nu -c 'open data.json | to csv | save -f out.csv'

String interpolation uses $"..." with parenthesized expressions.

nu -c '$"Name: ($row.name), Count: ($row.count)"'

Closure parameters use { |param| ... } syntax.

nu -c '... | each { |row| $row.field }'
nu -c '... | insert new_col { |r| $r.a / $r.b }'

sort-by not sort_by. Nushell uses hyphens, not underscores.
Same for: group-by, str join, math avg, split row.

Extended Patterns

See command-patterns.md
for group-by with aggregation, string building, nested data access,
column operations, piping from external commands, and full format lists.