Profiling Materialize

CPU profiling

Prefer profiling with samply, which produces Firefox Profiler-compatible flame graphs.

• Attach to a running process: samply record -p PID
• Record environmentd from startup: bin/environmentd --wrapper "samply record"
• Record clusterd processes from startup: bin/environmentd -- --orchestrator-process-wrapper "samply record"

Samply opens a browser tab with an interactive flame graph when recording finishes.

On macOS, enable codesigning for Instruments support: bin/environmentd --enable-mac-codesigning

Memory profiling

Valgrind doesn't work on Materialize as it's too slow to yield results.
Heaptrack works, but is slow.

Run heaptrack on environmentd processes: bin/environmentd --reset --no-default-features --wrapper "heaptrack --raw"
Run heaptrack on clusterd processes: bin/environmentd --reset --no-default-features -- --orchestrator-process-wrapper "heaptrack --raw"
Use heaptrack --interpret on the .raw.zst file to analyze.

Set MALLOC_CHECK_=3 MALLOC_PERTURB_=123 (and compile with --no-default-features on Linux to use the system allocator instead of jemalloc) to detect some memory corruption issues.

Analyze the size of binary functions

nm --demangle=rust --print-size --size-sort --radix=d BINARY to determine the binary size of symbols in BINARY.
Produces a potentially large output, which should be redirected and analyzed separately.
Filter with cut -d' ' -f2- | sort | sort -k3 | uniq to just learn about the size of symbols.