Work with the Zarr array storage format for chunked, compressed, N-dimensional arrays. Covers array creation, hierarchical groups, metadata/attributes, advanced indexing modes, data types, thread/process safety, sharding, and Zarr v2 vs v3 differences. Use when the user asks about Zarr arrays, .zarr files, zarr-python, zarr stores, chunked array storage, or cloud-optimized scientific array storage.
Resources
2Install
npx skillscat add uw-ssec/savvy/zarr-fundamentals Install via the SkillsCat registry.
Zarr Fundamentals
Zarr-Python 3 API for chunked, compressed, N-dimensional array storage in both v2 and v3 formats.
Docs: https://zarr.readthedocs.io/ | Spec: https://zarr-specs.readthedocs.io/ | numcodecs: https://numcodecs.readthedocs.io/
Resources
| Resource | Purpose |
|---|---|
| references/patterns.md | Patterns: hierarchical stores, remote access, appending, advanced indexing, sharding, concurrency |
| references/examples.md | Full working examples with real-world datasets |
| references/common-issues.md | v2/v3 confusion, metadata persistence, memory errors, concurrency bugs |
| assets/zarr-quickstart.py | Runnable quickstart demo |
Installation
pixi add zarr numpy numcodecs # pixi
pip install zarr[extra] # pip (includes numcodecs + optional deps)
pip install zarr[remote] # adds cloud backends (s3fs, gcsfs, adlfs)Core Operations
import zarr
import numpy as np
# Create (v3 default)
z = zarr.create_array("data.zarr", shape=(10000, 10000), chunks=(1000, 1000), dtype="float32")
z[:] = np.random.randn(10000, 10000).astype("float32")
# Open existing (read-only)
z = zarr.open_array("data.zarr", mode="r")
subset = z[0:100, 0:100]
# Groups
root = zarr.open_group("experiment.zarr", mode="w")
obs = root.create_group("observations")
arr = obs.create_array("temperature", shape=(365, 180, 360), chunks=(30, 90, 180), dtype="float32")
root.attrs["project"] = "Climate Study 2025"
arr.attrs["units"] = "Celsius"
print(root.tree())I/O Modes
| Mode | Description |
|---|---|
'r' |
Read-only (error if not found) |
'r+' |
Read/write (must exist) |
'w' |
Write (overwrite if exists) |
'w-' |
Create (error if exists — safe default for new stores) |
'a' |
Append (create or open) |
Zarr v2 vs v3
| Feature | v2 | v3 |
|---|---|---|
| Metadata file | .zarray / .zgroup / .zattrs |
zarr.json (single file) |
| Default compressor | Blosc | Zstd |
| Sharding | No | Yes |
| Async I/O | No | Yes (native asyncio) |
| Chunk key format | 0.0.0 (dot-separated) |
c/0/0/0 (path) |
| Codec system | Single compressor + filters | Composable pipeline |
Default to v3 for new projects. Use zarr_format=2 only for backward compatibility.
Indexing Modes
| Mode | Syntax | Returns |
|---|---|---|
| Basic slicing | z[0:100, 0:100] |
Contiguous subarray |
| Coordinate | z.vindex[rows, cols] |
Values at (row, col) pairs |
| Mask | z.vindex[bool_mask] |
1D array of True-masked values |
| Orthogonal | z.oindex[rows, cols] |
Cartesian product (rows × cols) |
| Block | z.blocks[0, 0] |
Entire chunk blocks by index |
Thread and Process Safety
- Reads: thread-safe without synchronization.
- Writes to non-overlapping chunks: safe concurrently.
- Writes to the same chunk: use
ThreadSynchronizer(v2) or partition work so chunks don't overlap. - Blosc + multiprocessing: always set
blosc.use_threads = Falsebefore forking to prevent silent data corruption. - v3 async concurrency:
zarr.config.set({"async.concurrency": 64}).
Safe concurrent write workflow:
from numcodecs import blosc
blosc.use_threads = False # set before any forked workers start
# Partition by non-overlapping index ranges, then write concurrently
# e.g. worker 0 writes z[0:500, :], worker 1 writes z[500:1000, :]Sharding (v3 Only)
Reduces cloud object count by grouping multiple inner chunks into larger shard files.
z = zarr.create_array("sharded.zarr", shape=(10000, 10000),
chunks=(2500, 2500), shards=(500, 500), dtype="float32")
# 100 GB @ 1 MB inner chunks: 100,000 objects without sharding → ~100 with shardingBest Practices
| Area | Guidance |
|---|---|
| Chunk size | 1–10 MB for cloud, 100 KB–1 MB for local |
| Fill value | Use np.nan for floats when 0 is a valid data value |
| Precision | Prefer float32 over float64 when sufficient (2× storage savings) |
| Metadata | Set units, long_name; follow CF Conventions for scientific data |
| Cloud | Use sharding (v3) + zarr.consolidate_metadata() for v2 |
| Safety | Use mode="r" for reads; mode="w-" to prevent accidental overwrites |