Records

IOMedia is the record surface every handle answers: one streamed read
(read_arrow_reader) and three explicit write intents (overwrite, append,
merge) in three input shapes (a batch reader, one batch or table, native
rows). The handle's media type - its suffix or declared MediaType - picks
the encoding; one RecordOptions carries every setting, and no call takes a
format argument. A schema is a non-null Struct Field; a read hands back a
stream in which only the current batch is alive.

Hold one model: the options are the query. field declares and casts,
select projects, filter prunes (row groups, partition leaves, Iceberg
manifests) and then filters rows, max_row_size stops pulling, merge_by
keys an upsert, commit_row_size bounds a write. Put them on the call and the
medium does the work before a byte is decoded.

Choose the door

Rules for fast, correct use

1. Keep the reader a reader. Iterate read_arrow_reader, or hand it
   straight to another handle's overwrite_arrow_reader; only the current
   batch is alive. read_all() / intoTable() / collect only when the
   caller asked for a whole table.
2. Push select, filter and limits into the options. Parquet skips
   unprojected column chunks and row groups whose footer statistics rule the
   filter out; IPC skips decoding unprojected columns; a folder skips leaves
   whose column=value path contradicts a filter equality; Iceberg skips
   manifests and files. Filtering rows after the read throws all of that away.
3. Declare the field to cast once. A narrower field is a projection; a
   wider one fills missing nullable columns with nulls; the cast runs in the
   same pass as the decode. A not null column refuses a value, a null or a
   missing column by name - it never stores a default. A nullable declared
   column takes a value it cannot convert as null while safe holds, and
   safe is on by default - so a bad value vanishes silently. Pass
   safe=False / { safe: false } / .with_safe(false) (or declare the
   column not null) when an unconvertible value must be refused.
4. merge_by is required for merge. Keys use Arrow's row format: null
   matches null and the last arrival wins. Merge holds only the stored side in
   memory; merge_by absent is a refusal, never an overwrite.
5. Bound memory with commit_row_size. Unset publishes once at the end;
   N publishes every N rows and the committed prefix survives a later
   failure; 0 is refused before any input is pulled.
6. row_size/column_size/read_arrow_field read metadata only. They
   answer from a footer, a stream header or the manifests; open() caches the
   answer until close().
7. The name picks the encoding and the outer coding. .arrows (IPC
   stream), .arrow/.feather/.ipc (IPC file), .parquet, .avro,
   .txt/.log, a table folder; .gz, .zz, .zst wrap the bytes.
   Parquet compresses internally, so .parquet.gz is refused before a byte is
   written - set compression instead.
8. Parquet and Iceberg are Cargo features in Rust (parquet, iceberg
   implies parquet); the Python and Node packages carry both.
9. Absent is empty; never probe first. An absent resource reads as no
   batches and the first write creates it and its parents. Do not guard with
   exists; an encoding the build lacks (e.g. text/csv) is named by
   record_options().
10. Properties by name are copies. read_arrow_reader(rowheader=...) /
    readArrowReader({ rowheader }) set that property on a copy of the
    handle's (or the given) options; the handle's options are unchanged. Left
    out (... / undefined) keeps the default; None / null clears.
11. JavaScript crosses as copied IPC. One self-contained IPC stream per
    batch; write whole tables or readers, never rows in a loop. Python crosses
    the C Data Interface without copying.
12. One plan per stream. A declared field or plan is compiled once per
    read or write session; build the options once and reuse them across
    calls instead of re-parsing a filter per batch.
13. Pick the input shape that is already in hand. A reader streams; a
    table or batch is wrapped into one; native rows (*_records) cross the
    value contract row by row - a row that omits a required column is refused,
    not defaulted - and cost the most (4,096 rows: about 0.1 ms as
    a batch, 3 ms as records, on the docs' reference machine). Keep rows for
    small or hand-built data, batches for everything else.
14. Plain text has a fixed shape. A text read answers the sixteen event
    columns (currunix first, state last), then body, then one column per
    named rowheader capture - the row header is the only thing that lifts a
    column out of a line. autotype settles each capture's datatype from the
    regex before a byte is read. An object's lines are one chain: a line whose
    own creaunix capture states none takes the earliest currunix the read
    has dated a line of its object by so far; a creaunix capture stands. A write consumes each row's non-empty body.
15. Iceberg commits are snapshots. append and metadata-only commits
    rebase on a concurrent commit; overwrite, merge and compact report a
    conflict instead. A merge keys on the identity partition columns plus
    merge_by, so a row only ever updates its own partition. A scan decodes
    qualifying files side by side (read.parallelism) and hands batches back
    in plan order.
16. Folders read by their layout. A stored column=value layout is
    authoritative; with none on disk, the schema's partition-marked fields
    decide where rows go (with_partition_fields). The first batch to reach a
    leaf performs the write's operation; later ones append.

Pitfalls

• Reading then filtering in the host ([r for r in rows if r["id"] > 3],
  table.filter(...)) - pass filter="id > 3" so the medium prunes.
• merge_* with no merge_by - it raises; pass merge_by=["id"] /
  { mergeBy: ['id'] } / with_merge_by(["id"])?.
• Naming a file trades.parquet.gz - refused ("parquet compresses"); use
  compression="zstd(3)" on a plain .parquet.
• Skipping rows in the host after the read: row_offset (rowOffset,
  with_row_offset) skips leading rows before max_row_size counts, and a
  plan's offset lands there too (options.plan, withPlan, with_plan,
  execute). A batch the skip covers whole is never handed on; a merge with
  a row_offset is refused.
• Calling overwrite_records / read_arrow_reader on a .json, .jsonl,
  .yaml, .toml or .xml handle - refused ("expected a record encoding
  this build implements"); use
  write_arrow / read_arrow (Rust, Python) or the codecs in
  yggdryl-documents.
• Appending rows to a .json/.jsonl/.yaml/.toml/.xml handle with
  write_arrow(..., append) - refused: a structured document is written
  whole, so only overwrite is accepted; to accumulate rows use an
  .arrows, .parquet or .avro handle, or read, extend and overwrite.
• Declaring int32 over a column holding "x" and trusting the read: under
  the default safe a nullable column reads it as null. Pass safe=False /
  { safe: false } / .with_safe(false) to be told.
• JavaScript plain-object rows with strings are inferred as
  dictionary(int32,utf8); every encoding stores them (Avro as the plain
  values), but pass { field } to state the column rather than inherit Arrow
  JS's guess.
• Python: read_arrow_reader(options) positionally is a TypeError; options
  are keyword-only (options=).
• Reading one leaf of a partitioned folder and expecting the partition
  columns: they live in the path; read the folder.
• Reusing an Iceberg Table object after writing through another handle: it
  caches metadata; open the table again.
• Collecting a Parquet read to count rows or learn the schema: row_size
  and read_arrow_field answer from the footer.
• commit_row_size on an Iceberg write: every commit is a snapshot, so a
  small N leaves many snapshots; expire them (expire_snapshots) or commit
  once.
• Building an Arrow JS Int64 vector from numbers: use bigint (1n).
  Native row writes unify number and bigint rows of one column instead:
  [{ id: 1 }, { id: 2n }] is one int64 column, the integral number read
  as bigint; only a fraction beside bigint rows is refused, naming the
  column and the value.
• Looking for a CSV reader: text/csv is not a record encoding; read lines
  with a rowheader regex, or convert upstream.

Language references

• references/rust.md - read for Rust: traits to import, RecordOptions builders, batch readers, Iceberg Table.
• references/python.md - read for Python: keyword properties, pyarrow readers, dataclass rows, lazy scans.
• references/javascript.md - read for Node.js: property objects, Arrow JS tables, bigint, copied IPC.
• references/formats.md - per-encoding table: media type and suffix, settings, pushdown, feature gate, limits.

Deeper

• Records surface (signatures, pushdown, limits, append and merge, commit cadence, lazy scans): https://platob.github.io/yggdryl/holder/#records
• Partitions (pruning, partition columns, derived columns): https://platob.github.io/yggdryl/holder/#partitions
• Media overview and options: https://platob.github.io/yggdryl/media/#read-and-write
• Per format: https://platob.github.io/yggdryl/media/#arrow-ipc, https://platob.github.io/yggdryl/media/#parquet, https://platob.github.io/yggdryl/media/#avro, https://platob.github.io/yggdryl/media/#plain-text, https://platob.github.io/yggdryl/media/#iceberg
• Required columns and the cast rule: https://platob.github.io/yggdryl/types/cast/
• Plans and write verbs: https://platob.github.io/yggdryl/expression/plans/
• Sibling skills: yggdryl-storage (handles, backends, codings), yggdryl-arrow (Serie, SerieReader, casts), yggdryl-expressions (filter/select grammar, Plan), yggdryl-uri (hive paths, globs), yggdryl-types (fields, dataclasses), yggdryl-documents (JSON/YAML/TOML/XML).