uw-ssec

zarr-fundamentals

Work with the Zarr array storage format for chunked, compressed, N-dimensional arrays. Covers array creation, hierarchical groups, metadata/attributes, advanced indexing modes, data types, thread/process safety, sharding, and Zarr v2 vs v3 differences. Use when the user asks about Zarr arrays, .zarr files, zarr-python, zarr stores, chunked array storage, or cloud-optimized scientific array storage.

uw-ssec 32 14 Updated 21h ago

Resources

2
GitHub

Install

npx skillscat add uw-ssec/savvy/zarr-fundamentals

Install via the SkillsCat registry.

SKILL.md

Zarr Fundamentals

Zarr-Python 3 API for chunked, compressed, N-dimensional array storage in both v2 and v3 formats.

Docs: https://zarr.readthedocs.io/ | Spec: https://zarr-specs.readthedocs.io/ | numcodecs: https://numcodecs.readthedocs.io/

Resources

Resource Purpose
references/patterns.md Patterns: hierarchical stores, remote access, appending, advanced indexing, sharding, concurrency
references/examples.md Full working examples with real-world datasets
references/common-issues.md v2/v3 confusion, metadata persistence, memory errors, concurrency bugs
assets/zarr-quickstart.py Runnable quickstart demo

Installation

pixi add zarr numpy numcodecs   # pixi
pip install zarr[extra]          # pip (includes numcodecs + optional deps)
pip install zarr[remote]         # adds cloud backends (s3fs, gcsfs, adlfs)

Core Operations

import zarr
import numpy as np

# Create (v3 default)
z = zarr.create_array("data.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="float32")
z[:] = np.random.randn(10000, 10000).astype("float32")

# Open existing (read-only)
z = zarr.open_array("data.zarr", mode="r")
subset = z[0:100, 0:100]

# Groups
root = zarr.open_group("experiment.zarr", mode="w")
obs = root.create_group("observations")
arr = obs.create_array("temperature", shape=(365, 180, 360), chunks=(30, 90, 180), dtype="float32")
root.attrs["project"] = "Climate Study 2025"
arr.attrs["units"] = "Celsius"
print(root.tree())

I/O Modes

Mode Description
'r' Read-only (error if not found)
'r+' Read/write (must exist)
'w' Write (overwrite if exists)
'w-' Create (error if exists — safe default for new stores)
'a' Append (create or open)

Zarr v2 vs v3

Feature v2 v3
Metadata file .zarray / .zgroup / .zattrs zarr.json (single file)
Default compressor Blosc Zstd
Sharding No Yes
Async I/O No Yes (native asyncio)
Chunk key format 0.0.0 (dot-separated) c/0/0/0 (path)
Codec system Single compressor + filters Composable pipeline

Default to v3 for new projects. Use zarr_format=2 only for backward compatibility.

Indexing Modes

Mode Syntax Returns
Basic slicing z[0:100, 0:100] Contiguous subarray
Coordinate z.vindex[rows, cols] Values at (row, col) pairs
Mask z.vindex[bool_mask] 1D array of True-masked values
Orthogonal z.oindex[rows, cols] Cartesian product (rows × cols)
Block z.blocks[0, 0] Entire chunk blocks by index

Thread and Process Safety

  • Reads: thread-safe without synchronization.
  • Writes to non-overlapping chunks: safe concurrently.
  • Writes to the same chunk: use ThreadSynchronizer (v2) or partition work so chunks don't overlap.
  • Blosc + multiprocessing: always set blosc.use_threads = False before forking to prevent silent data corruption.
  • v3 async concurrency: zarr.config.set({"async.concurrency": 64}).

Safe concurrent write workflow:

from numcodecs import blosc
blosc.use_threads = False  # set before any forked workers start

# Partition by non-overlapping index ranges, then write concurrently
# e.g. worker 0 writes z[0:500, :], worker 1 writes z[500:1000, :]

Sharding (v3 Only)

Reduces cloud object count by grouping multiple inner chunks into larger shard files.

z = zarr.create_array("sharded.zarr", shape=(10000, 10000),
                      chunks=(2500, 2500), shards=(500, 500), dtype="float32")
# 100 GB @ 1 MB inner chunks: 100,000 objects without sharding → ~100 with sharding

Best Practices

Area Guidance
Chunk size 1–10 MB for cloud, 100 KB–1 MB for local
Fill value Use np.nan for floats when 0 is a valid data value
Precision Prefer float32 over float64 when sufficient (2× storage savings)
Metadata Set units, long_name; follow CF Conventions for scientific data
Cloud Use sharding (v3) + zarr.consolidate_metadata() for v2
Safety Use mode="r" for reads; mode="w-" to prevent accidental overwrites