Storage: handles, bytes, codings, charsets

Every storage backend is one positional IOBase handle; Holder is the Rust
enum over all of them and the bindings expose it as one IOBase class. A
caller writes the same calls whatever it holds: pread/pwrite are the
primitives and every whole read, stream, digest, coding and record surface
derives from them. Hold one mental model: a call is a round trip - one
syscall, one object-store request, one lock per wrapper - so the fastest code
is the one that makes the fewest calls, and construction makes none.

Records on a handle (read_arrow_reader, overwrite/append/merge,
RecordOptions, partitions) are yggdryl-records; JSON/YAML/TOML/XML codecs
are yggdryl-documents; parsing a URL, a glob or a Hive path is yggdryl-uri.
Install and cross-language conventions are in yggdryl.

Choose the door

Rules for fast, correct use

1. Count calls, not bytes. read_all_bytes is one call whatever the size;
   a footer is read_range_bytes(size - 8, 8) - one ranged GET, not the
   object; size itself is one HEAD on a closed store handle and free on a
   listed handle, inside open(), or after S3File::with_known_size(n).
   Slice later needs out of a read you already hold.
2. Construction touches nothing. Building a handle, a child
   (child_by_path, /, joinpath), a media type or a partition costs zero
   calls on every backend, S3/GCS/Azure included.
3. EAFP - never guard. A read of something absent is empty (b"", size
   0), a write creates the resource and every missing parent, remove of
   something absent succeeds. exists()/is_dir()/mkdir() before an
   operation is a wasted round trip and a race.
4. Stream bounded chunks. pstream_bytes hands out 64 KiB owned chunks
   and never asks for size; digests (read_digest) stream too. Reach for
   read_all_bytes only when the whole value is the answer.
5. One scan, one stream on a coded handle. A pread at a compressed
   offset rebuilds a decoder from the start: sixteen ranged reads of a gzip
   value cost sixteen decodes. Drain one pstream_bytes, or open() it
   (caches the decoded value), or wrap it in buffered.
6. Open a scope for metadata-heavy work. open caches the descriptor and
   mapping (local), the object's metadata (stores: size is a HEAD closed,
   free open), a Parquet footer, a decoded value; close publishes and
   releases. Outside a scope each call fetches fresh - right for a resource
   another writer changes.
7. The name declares the coding. trades.json.gz is base application/json
   plus coding gzip: media_type names the decoded representation, codec
   the stored coding. In-memory bytes are sniffed; declare a type the bytes
   cannot prove (CSV) with with_media_type / the media_type setter.
8. Know which bytes a handle presents. Python IOBase(name) composes what
   the name declares (Text(Gzip(LocalPath))) and reads decoded bytes; the
   role classes (LocalPath, LocalFile, S3File, FsPath) address stored
   bytes. JavaScript byte methods always address stored bytes; the coding is
   applied by readScalar/writeScalar, the record surface and
   compressInto/decompressInto. Rust Holder::local, Holder::from_url
   and the Local* roles address stored bytes; holder.into_coded() (or
   into_declared_media(), which also puts the record encoding on top)
   composes what the name declares - the Rust equivalent of Python IOBase(name).
9. Wrappers compose over a handle, never inside it. Buffered<Coded<_>>
   caches decoded pages, Coded<Buffered<_>> the encoded transport; wrapping
   twice reconfigures the one layer. Python conversions (buffered,
   into_coded) consume the handle they took.
10. Text crosses the charset boundary once. Decode at intake - a
    Transcoded handle, a media type's ;charset=, or Charset::decode - and
    work in UTF-8 after. No reader takes a charset argument. iso-8859-1 is
    never windows-1252; bare utf-16 is refused (say utf-16le/utf-16be).
11. Listings are lazy, sorted, and free to stop. Building a listing costs
    nothing and stopping early stops the walk: taking the first entries reads
    one directory level (all its names, sorted - never the tree) locally, or
    one 1000-entry page on a store. A recursive walk skips .git, .venv,
    .DS_Store unless include_private. A glob descends its fixed prefix
    (year=2024/**/*.parquet) rather than listing and filtering; an object
    listing states every size, so a listed object is never re-asked.
12. Join nested children in one string. child_by_path("sub/inner.bin")
    creates the parents on write; joining segment by segment needs each
    intermediate to already be a container.
13. A local file is memory-mapped. Another process truncating it raises
    SIGBUS; copy_into a Buffer when the file may change underneath.
    flush/close publish the logical length; append_bytes and
    write_all_bytes already end with a flush and publish on return, on
    every backend (a remote object written, the mapped file's growth slack
    trimmed) - only bare pwrite stages without publishing.
14. Object stores state their request count. Ranged read: one GET;
    whole read/drain/digest: one GET; whole write: one PUT; listing: one
    request per 1000 entries; remove: one DELETE, no probe. with_known_size
    skips the HEAD; open() before buffered on a remote handle.
15. Credentials resolve lazily, explicit wins. Unset knobs come from the
    URL, the environment (AWS_, GOOGLE_, AZURE_, YGGDRYL_), the store's
    files, then defaults; the AWS identity is botocore's chain through
    aws::Session. with_environment(false) seals everything but explicit
    values - see references/backends.md.
16. HTTP is a handle and a client. An http/https URL is a leaf: a
    whole read is one GET, a range one ranged GET, size one HEAD
    (none inside open()). A body read whole or streamed resumes a cut
    transfer from its cursor (Range + If-Range), for a successful uncoded
    GET only. Retries touch only what cannot act twice: an idempotent
    method, or a request no connection took; a POST answered 503 is
    handed back. Fan out with send_all on one session - one pool, answers
    in order, nothing held past the requests in flight - never a thread pool
    of gets. HTTP/2 and HTTP/3 are negotiated (ALPN h2, then HTTP/3 once
    Alt-Svc advertises it) and multiplex one connection per origin; a plain
    http:// origin speaks h2c only when asked, and a proxied request
    HTTP/1.1. Proxies, CA bundles and .netrc come from the environment as
    curl and requests read them.
17. Pagination is one ladder, seven spellings. Pagination::Auto (the
    default) tries the Link header, then a few common next-page headers,
    then a next URL or cursor in the body, in a fixed order; name one
    explicitly with link, header:<name>, url:<path>,
    cursor:<path>:<parameter>, offset:<parameter>:<size> or
    page:<parameter>:<start>. .pages() walks a Request that has not
    been sent - session.pages(request) / request.pages() in Rust,
    session.pages(url, pagination=...) / request.pages(pagination=...)
    in Python, session.pages(url, { pagination }) /
    request.withPagination(...).pages() in JavaScript - never an
    already-sent Response: Python's and JavaScript's session.get(url)
    sends immediately and answers one with no .pages(); Rust's
    session.get(url) alone is the unsent Request itself, only
    .send()'s answer has none. Every page lays out as one Arrow batch
    through Pages::into_arrow_reader, .intoArrowReader, or Python's
    .read_arrow().

Pitfalls

Language references

• references/rust.md - read when writing Rust (Holder, Buffer, Coded, Transcoded, Counted, s3, zip).
• references/python.md - read when writing Python (IOBase, role classes, yggdryl.gzip, yggdryl.charset).
• references/javascript.md - read when writing Node.js (IOBase, gzip/zlib/zstd/charset namespaces, handler protocol).
• references/backends.md - configuration and cost tables for Local, Filesystems, S3 / GCS / Azure, HTTP, Buffered and ZIP.

Deeper

• Handles, bytes, values, call counts, every backend: https://platob.github.io/yggdryl/holder/
• Laziness and kinds: https://platob.github.io/yggdryl/holder/#laziness-and-kinds
• Media type and codings: https://platob.github.io/yggdryl/holder/#media-type-and-codings
• Object stores: https://platob.github.io/yggdryl/holder/#object-stores
• HTTP, HTTP/2 and HTTP/3: https://platob.github.io/yggdryl/holder/#http
• Pagination and Pages: https://platob.github.io/yggdryl/holder/#pages
• ZIP: https://platob.github.io/yggdryl/holder/#zip
• Compression (gzip, zlib, zstd): https://platob.github.io/yggdryl/media/#compression
• Charsets: https://platob.github.io/yggdryl/media/#charsets
• Sibling skills: yggdryl-records (rows on a handle, partitions), yggdryl-uri
  (URLs, globs, Hive paths), yggdryl-documents (JSON/YAML/TOML/XML),
  yggdryl-hashing (digest values).