Yggdryl hashing

Two families over one digest engine. xxhash digests bytes, handles, values
and Arrow rows with XXH32, XXH64, XXH3-64 and XXH3-128; txhash couples a UTC
instant with one of those digests into one sortable TxHash, and defines no
second hash. The fact to hold: stable_hash is XXH3-64 over the canonical
value feed, everywhere - one value answers one number in Rust (u64),
Python (int) and JavaScript (bigint), whatever width or leaf holds it.
A Digest carries its algorithm with the number, so two algorithms never
compare equal. Not cryptographic, and not Iceberg's bucket[N] (murmur3).

A TxHash adds an 8-byte big-endian instant in front: 12, 16 or 24 bytes,
spelled <unix>@<unit>:<algorithm>:<hex>.

Choose the door

Rules for fast, correct use

1. Use stable_hash for cross-language identity. It is the same number in
   all three languages for equal values; a hand-rolled hash of a text or JSON
   rendering is not, and drifts with formatting.
2. The value feed is framed. stable_hash hashes a tag byte then the
   canonical form (lengths included), so it is not xxh3 of the value's
   bytes; the bare payload is as_value_bytes (Rust, Python).
3. Equal values, one digest; different values, apart. int8 1 and
   int64 1, float32/float64 1.5, decimal(1.00)/decimal(1), and every
   string leaf of the same characters hash equal; null vs "", "1" vs
   b"1", and a ccy vs a country with the same text stay apart.
4. Hash a stored resource where it lives. read_digest streams the handle
   through pstream_bytes one bounded chunk at a time, and no byte crosses
   into the host - true of every backend's returned chunk. The local backend
   is memory-mapped, though: it keeps one whole-file mapping open for the
   handle's life, so a large local file's touched pages stay resident (the OS
   page cache of that mapping) until the handle closes, even while each
   pread only copies out one bounded window. Buffer and the network-backed
   stores (HTTP, S3, ...) hold no such mapping and stay flat throughout. Never
   read_all_bytes then hash.
5. Hash in the pass you already make. Rust xxhash::reader / writer
   hash a copy in flight; Hashed<H> answers read_digest from its running
   state while writes are sequential from offset 0, and re-streams once after
   a positional write. In Python and JavaScript feed the chunks you copy to a
   state.
6. States are resumable. Any split answers the one-shot digest; reading a
   digest never consumes the state; clear() returns to the constructed seed
   and secret; JavaScript clone() forks.
7. Carry the algorithm. Store or send a Digest (xxh3-64:<hex>) or a
   TxHash rather than a bare integer: xxh64 and xxh3-64 are both 64 bits
   and differ.
8. Mark one holder, leave its sources ordinary columns. DIGEST:role = holder on the digest column; DIGEST:sources (a JSON array of paths
   relative to its own Struct) narrows the input, absent or ["*"] meaning
   every non-holder field. The schema pipeline fills holders last, after cast
   and transform, and a written (non-default) cell is preserved unless forced -
   so re-filling is idempotent and cheap.
9. Seedless or seeded, decide once. Field.apply_arrow_batch and the
   digest view fill with the seedless stable_hash; a state's
   apply_arrow_batch applies its seed and secret. A seed changes every
   digest, so a seeded key is only comparable with the same seed.
10. Secrets are XXH3 only: at least 136 bytes, refused below that, and
    consulted only for payloads longer than 240 bytes.
11. Small inputs pay the binding. A one-byte call costs about 166 ns in
    Python and 496 ns in Node over the hash itself; batch small values through
    a holder fill or row_digests instead of a call per value. A Python
    bytes is borrowed; bytearray/memoryview are copied (1.7x slower); a
    JavaScript string is UTF-8 encoded (7.5x slower than a Buffer).
12. A TxHash sorts by time only within one unit, one algorithm, one sign.
    Its bytes are the big-endian instant then the digest; value order is unit,
    then count, then digest; instants are never normalized across units.
13. Instants are UTC, microseconds by default. A zoned datetime already
    counts from the epoch, a naive one reads as UTC, a date is its midnight; a
    coarser restatement floors, overflow is refused. Set the resolution once
    on a TxHasher.
14. UUIDv7 needs a 64-bit digest and a non-negative instant. into_uuid
    keeps the microsecond and all 64 digest bits (nothing hashed again, both
    read back); into_sequenced_uuid(sequence, seed) is the graph-event
    layout; Rust into_ordered_bytes is the 16-byte chronological key with no
    UUID bits.
15. JavaScript Arrow crosses by copy. A state's or TxHasher's
    applyArrowBatch copies the batch through Arrow IPC both ways; row and
    coupled column functions are Rust and Python only.

Pitfalls

Language references

• references/rust.md - read when writing Rust.
• references/python.md - read when writing Python.
• references/javascript.md - read when writing JavaScript / TypeScript.

Deeper

• Contract, algorithms and bindings: https://platob.github.io/yggdryl/hashing/
• Streaming, seeds and secrets: https://platob.github.io/yggdryl/hashing/#streaming,
  https://platob.github.io/yggdryl/hashing/#seeds-and-secrets
• Handles and write-through: https://platob.github.io/yggdryl/hashing/#handles
• The value feed and its encoding: https://platob.github.io/yggdryl/hashing/#values,
  https://platob.github.io/yggdryl/hashing/#encoding
• Holders and row digests: https://platob.github.io/yggdryl/hashing/#digest-holders-and-row-digests
• TxHash, UUIDv7, instants, TxHasher: https://platob.github.io/yggdryl/hashing/#txhash-values,
  https://platob.github.io/yggdryl/hashing/#order-and-uuidv7-projection
• Coupled columns and holders: https://platob.github.io/yggdryl/hashing/#coupled-columns,
  https://platob.github.io/yggdryl/hashing/#coupled-holders
• Every edge: https://platob.github.io/yggdryl/hashing/#edges; measured costs:
  https://platob.github.io/yggdryl/hashing/#performance
• The digest protocol view on a field: https://platob.github.io/yggdryl/types/protocol/
• Sibling skills: yggdryl-storage (the IOBase handles read_digest
  streams), yggdryl-types (Scalar, Field metadata),
  yggdryl-expressions (Field.apply_arrow_batch, the cast -> transform ->
  digest pipeline),
  yggdryl-arrow (batches and readers), yggdryl-market-data (event
  identities built on TxHash and UUIDv7), yggdryl-expressions
  (stable_hash of a plan as a cache key).