Yggdryl expressions

An expression is text - or a tree built method by method - over one grammar,
and it means nothing until it is bound against a Struct Field. Binding
only requires the schema's datatype to be a Struct; the root field's own
nullability is not checked (a non-null Struct root is a separate,
storage-side convention for record write roots, not for binding).
Parse once; bind once per stream, which resolves names to indices, converts
each literal into the type of the column it meets, folds constants and orders
and cheapest-first; then evaluate as many rows, batches or statistics as you
like from the compiled tree - or hand the unbound clause to a record read and
let the media prune. There is no second engine: one Bound answers a row, a
batch, a stream and a file's statistics, and the Arrow tier is asserted equal
to the row tier.

Pipeline: parse -> type -> simplify -> bind(schema) -> apply (row, batch,
reader, statistics, or media pushdown).

Choose the door

x is any clause: Filter, Selector, Plan or Expression, with two
exceptions: a Plan's output schema is plan.field_from(root) /
plan.fieldFrom(root) (Rust also apply_datatype), never apply_field; and
a Rust Plan answers only apply_arrow_reader and execute - convert with
Expression::from(plan) for apply_arrow_batch, apply_records or
apply_field.

Rules for fast, correct use

1. Parse once, bind once, evaluate many. Binding is the expensive,
   schema-dependent step (0.5-4.5 us); hoist it out of every row and batch
   loop and keep the Bound / BoundSelector.

2. Take the reader door. apply_arrow_reader binds once and wraps the
   stream lazily, with the output schema known before the first batch. The
   unbound clause's apply_arrow_batch binds on every call - right for one
   batch, a defect in a loop; Rust also exposes an unbound apply_scalar per
   row, at the same per-call bind cost. Python and JavaScript have no unbound
   per-row door: bind once and read Bound.matches / eval (Filter) or
   BoundSelector.apply_row / applyRow (Selector) per row instead.

3. Push where and select into the read instead of filtering after it.
   Record options carry them as sections: hive leaves prune by
   partition_pairs, Parquet row groups and Iceberg manifests by statistics,
   only read_columns() is decoded, and only surviving rows reach the host
   (see yggdryl-records).

4. Identity costs nothing. Shape the clause so the cheap rows of this
   table apply:

   Shape	Buffers
   select *, a self alias, a cast to the type a column has	skipped at bind: the reader or batch that came in
   a mask keeping every row	the input batch handed back (Rust, Python: the same object)
   a mask keeping some rows	copied, because a batch is dense
   a projection of bare columns	arrays reordered, no buffer touched
   offset, limit	slice views over the batches they cross
   order by	the stream collected once, one sort, one take
   any of the above from JavaScript	copied through Arrow IPC in and out

5. order by is the only section that collects. limit and offset are
   slices; execute pushes them into the read when nothing orders, as the
   record options' max_row_size and row_offset. Filter first and order
   only when the answer needs it.

6. Statistics answer false only when no row can match. true means
   "must read", never "matches"; a user function is unknown to statistics, so
   it forces a row read and never a wrong skip.

7. Null is unknown. A comparison with a null is unknown and a where
   keeps only exactly-true rows; test absence with is null, is distinct from or coalesce.

8. Best effort, then a named refusal. A constant coerces into the operand
   it meets (i = '1' on int64 binds as i = 1); operands with no common
   type compare as text (s > 1 on utf8 is s > '1'); a declared column
   casts safely unless it is not null, which refuses naming the column and
   value. Decimals never become floats - pass exact values, and write a
   fractional literal against a decimal typed (decimal(9,2) '9.50'): an
   untyped 9.5 is float64, shares no type with a decimal, and so compares
   as text.

9. Paths are resolved, never split. A FieldPath is parsed once at the
   boundary; quote a dotted name ("a.b"); never split a path string on ..

10. Parameters bind once. :name values go to bind_with / bind(root, params). A missing one is refused naming it; an extra one is silently
    ignored and a repeated one (Rust slice) takes its first value, so compare
    against parameters() yourself when a misspelt key must fail.

11. Canonical text is identity. Every layer's text re-parses to the same
    tree and stable_hash / stableHash hashes it - a safe key for a cache of
    parsed clauses; a cache of bound plans keys on the clause hash together
    with the schema's Field stable_hash, because a Bound is resolved
    against one schema.

12. A Field is a plan holder. into_field stores each derivation as
    TRANSFORM:function + TRANSFORM:sources (a call over plain columns) or
    TRANSFORM:expression, and Field::apply_arrow_batch /
    apply_arrow_reader (Python field.apply_arrow_batch(batch); not bound
    in JavaScript) recomputes them on a batch - it runs the field's cast
    first (cast rules: yggdryl-arrow references/cast-rules.md), then
    TRANSFORM:/PARTITION:, then DIGEST: holders (yggdryl-hashing). Keep the source columns in the selector: the stored
    field is what the recompute reads.

13. DuckDB names the vocabulary: * exclude (...), unnest/explode,
    in, between, is null, case when, asc/desc nulls first. Joins,
    aggregates, windows and regexes are refused, not emulated.

Pitfalls

Language references

• references/rust.md - read when writing Rust.
• references/python.md - read when writing Python.
• references/javascript.md - read when writing JavaScript / TypeScript.
• references/grammar.md - the grammar cheat sheet: every clause, verb,
  operator, path step and function, and the semantics that decide answers.

Deeper

• Overview and the best-effort rule: https://platob.github.io/yggdryl/expression/
• Grammar, functions, refusals: https://platob.github.io/yggdryl/expression/grammar/
• Terms, literals, predicate segments: https://platob.github.io/yggdryl/expression/terms/
• Selectors, * exclude, declared columns: https://platob.github.io/yggdryl/expression/selectors/
• Filters and pushdown levels: https://platob.github.io/yggdryl/expression/filters/
• User functions: https://platob.github.io/yggdryl/expression/functions/
• Plans, verbs, locations, execute: https://platob.github.io/yggdryl/expression/plans/
• Evaluation tiers, statistics, Iceberg scans: https://platob.github.io/yggdryl/expression/evaluate/
• Paths: https://platob.github.io/yggdryl/types/paths/
• Record options and partition pruning: https://platob.github.io/yggdryl/media/#options,
  https://platob.github.io/yggdryl/holder/#pruning-and-filtering
• Sibling skills: yggdryl-records (where the pushed-down read runs),
  yggdryl-types (the Field a clause binds against),
  yggdryl-arrow (BatchReader, Serie, casts; references/cast-rules.md
  for the cast Field::apply_arrow_batch runs before TRANSFORM:),
  yggdryl-hashing (stable_hash, the DIGEST: holders it fills last).