Test Data Generation & Validation

This skill provides guidance on generating real Cassandra 5.0 test data and validating parsing correctness.

When to Use This Skill

• Generating test data with specific schemas
• Creating test fixtures for property tests
• Exporting SSTables from Cassandra
• Validating parsed data against sstabledump
• Managing test datasets
• Creating reproducible test scenarios

Overview

CQLite uses real Cassandra 5.0 instances to generate test data, ensuring:

• Format correctness (real Cassandra writes)
• Edge case coverage (nulls, empty values, large values)
• Compression validation (actual compressed SSTables)
• Schema variety (all CQL types)

Test Data Workflow

See dataset-generation.md for complete workflow details.

Quick Start

Full corpus regeneration (recommended):

# Regenerates nb + oa + da keyspaces in one command
bash test-data/scripts/regenerate-datasets.sh

# With custom row count
bash test-data/scripts/regenerate-datasets.sh --rows 200

Compose-stack workflow (interactive / partial):

cd test-data

# 1. Start clean Cassandra 5 with schemas
./scripts/start-clean.sh

# 2. (Optional) run CQL inserts manually via cqlsh
docker exec -it cqlite-cassandra-5-0 cqlsh

# 3. Export SSTables + generate JSONL goldens
./scripts/export.sh

# 4. Shutdown and clean volumes
./scripts/shutdown-clean.sh

Generation Scripts

regenerate-datasets.sh (primary entrypoint)

Full corpus regeneration: nb + oa + da keyspaces, single Docker container.

What it does:

1. Starts cassandra:5.0.2 container (cqlite-regen)
2. Applies schemas and inserts rows for nb keyspaces (CASSANDRA_4 compat mode)
3. Restarts with storage_compatibility_mode: NONE, creates oa keyspace
4. Restarts with sstable.selected_format: bti, creates da keyspace
5. Exports SSTables via tar stream + generates sstabledump JSONL goldens
6. Writes metadata.yml

Options:

• --rows N - Rows per nb table (default: 50)
• --out <dir> - Output directory (default: test-data/datasets)
• --dry-run - Print commands without executing

Example:

bash test-data/scripts/regenerate-datasets.sh
bash test-data/scripts/regenerate-datasets.sh --rows 200 --out /tmp/ds
bash test-data/scripts/regenerate-datasets.sh --dry-run

start-clean.sh

Starts Cassandra 5.0 container (via compose) and applies schemas.

What it does:

1. Starts cassandra-5-0 container via docker-compose
2. Waits for Cassandra to be healthy
3. Applies schemas from schemas/core.list

Environment variables:

• SCHEMA_SET=core - Use curated schema list (default)
• SCHEMA_SET=all - Use all *.cql files

Example:

./scripts/start-clean.sh
SCHEMA_SET=all ./scripts/start-clean.sh

export.sh

Exports SSTables from the running Cassandra container and generates reference files.

What it does:

1. Runs nodetool flush to write all memtables to disk
2. Streams SSTables from container to datasets/sstables/ via tar
3. Generates sstabledump JSONL golden files and sstablemetadata statistics
4. Writes metadata.yml using cqlsh (no external generator container needed)
5. Writes references.yml manifest

Output structure:

test-data/datasets/
├── metadata.yml
├── references.yml
└── sstables/
    ├── test_basic/
    │   └── simple_table-<hash>/
    │       ├── *-Data.db
    │       ├── *-Data.db.jsonl   # sstabledump golden
    │       ├── *-Index.db
    │       ├── *-Statistics.db
    │       └── *-TOC.txt
    ├── test_collections/
    └── test_timeseries/

shutdown-clean.sh

Stops Cassandra and removes Docker volumes.

What it does:

1. Stops all containers
2. Removes Docker volumes (clean slate)
3. Prepares for next generation cycle

Use when:

• Done with current dataset
• Want to regenerate from scratch
• Cleaning up after tests

Test Schemas

Schemas in test-data/schemas/:

basic-types.cql

Simple table with all primitive types:

• Partition key: uuid
• No clustering
• Columns: int, text, timestamp, boolean, etc.

collections.cql

Collection types:

• list
• set
• map<text, int>
• Nested frozen collections

time-series.cql

Time-series pattern:

• Partition key: sensor_id
• Clustering: timestamp (DESC)
• Columns: temperature, humidity, pressure

wide-rows.cql

Wide partition testing:

• Single partition key
• Many clustering rows (1000+)
• Tests pagination and offset handling

Custom Schemas

Add your own:

# Create schema
echo "CREATE TABLE test_keyspace.my_table (...);" > schemas/my-schema.cql

# Add to core.list
echo "my-schema.cql" >> schemas/core.list

# Generate
bash test-data/scripts/regenerate-datasets.sh

Validation Workflow

See validation-workflow.md for complete validation process.

Validate Against sstabledump

# 1. Generate sstabledump reference
sstabledump test-data/datasets/sstables/keyspace/table/*-Data.db \
    > reference.json

# 2. Parse with cqlite
cargo run --bin cqlite -- \
    --data-dir test-data/datasets/sstables/keyspace/table \
    --schema test-data/schemas/schema.cql \
    --out json > cqlite.json

# 3. Compare (ignoring formatting)
jq -S '.' reference.json > ref-sorted.json
jq -S '.' cqlite.json > cql-sorted.json
diff ref-sorted.json cql-sorted.json

Automated Validation

Run validation script:

# Validate all test tables
cargo test --test sstable_validation

# Validate specific table
cargo test --test sstable_validation -- simple_table

Property Testing

Generate random data for property tests:

use proptest::prelude::*;

proptest! {
    #[test]
    fn test_row_parsing_roundtrip(
        partition_key in any::<i32>(),
        text_value in "\\PC*",  // Any valid unicode
        int_value in any::<i32>(),
    ) {
        // Generate test data in Cassandra
        insert_test_row(partition_key, &text_value, int_value)?;
        flush_memtable()?;
        
        // Parse with cqlite
        let parsed = parse_sstable()?;
        
        // Validate roundtrip
        assert_eq!(parsed.get_int("partition_key"), partition_key);
        assert_eq!(parsed.get_text("text_col"), text_value);
        assert_eq!(parsed.get_int("int_col"), int_value);
    }
}

Dataset Packaging

Package datasets for CI or distribution:

# Package current dataset
./scripts/package_datasets.sh

# Output: test-data/cqlite-test-data-v5.0-<date>.tar.gz

Contents:

• All SSTables
• metadata.yml
• Schema files
• README with generation parameters

CI Integration

Smoke Test

Quick validation in CI:

# Use packaged dataset
tar xzf cqlite-test-data-v5.0.tar.gz

# Run core tests
./scripts/ci-one-shot-smoke.sh

# Validates:
# - Basic parsing
# - All CQL types
# - Compression
# - Collections

See test-data/scripts/CI_SMOKE_TEST_USAGE.md for details.

Common Scenarios

Scenario 1: Test New CQL Type

# 1. Add column to schema
echo "ALTER TABLE test_basic.simple_table ADD duration_col duration;" \
    >> schemas/basic-types.cql

# 2. Regenerate data
bash test-data/scripts/regenerate-datasets.sh

# 3. Validate parsing
cargo test --test sstable_validation

Scenario 2: Test Large Values

# Generate with higher row count
bash test-data/scripts/regenerate-datasets.sh --rows 10000

Scenario 3: Test Edge Cases

Add a new schema .cql file and CQL INSERT statements, then run
regenerate-datasets.sh. For small one-off tests, use the compose stack:

bash test-data/scripts/start-clean.sh
docker exec -it cqlite-cassandra-5-0 cqlsh -e "INSERT INTO test_basic.simple_table ..."
bash test-data/scripts/export.sh

PRD Alignment

Supports Milestone M1 (Core Reading Library):

• 95% test coverage goal
• All CQL types validated
• Real Cassandra data ensures format correctness

Supports All Milestones:

• Regression testing with frozen datasets
• Property-based testing for edge cases
• CI integration for PR validation

Troubleshooting

Cassandra Won't Start

# Check logs
docker logs cassandra-5-0

# Common issue: Port 9042 in use
lsof -i :9042
# Kill process or change port in docker-compose-cassandra5.yml

Generation Fails

# Check generator logs
cat test-data/logs/data_generation.log

# Verify schema applied
docker exec cassandra-5-0 cqlsh -e "DESCRIBE KEYSPACES;"

Export Produces No Files

# Verify data exists in container
docker exec cassandra-5-0 ls -la /var/lib/cassandra/data/

# Check if flush happened
docker logs cassandra-5-0 | grep flush

Dataset Repository

Packaged datasets available at:

https://github.com/pmcfadin/cqlite/releases/tag/test-data-v5.0

Download for:

• CI without Docker
• Reproducible benchmarks
• Offline development

Next Steps

When creating new tests:

1. Design schema in schemas/
2. Regenerate data with regenerate-datasets.sh
3. Write parser test
4. Validate with sstabledump
5. Add to CI smoke test suite

See documentation:

• dataset-generation.md - Full workflow
• validation-workflow.md - Validation process