Skip to content

TESTS.md

Latest commit

 

History

History
297 lines (210 loc) · 11 KB

TESTS.md

File metadata and controls

297 lines (210 loc) · 11 KB
Raw

Property-Based and Load/Stress Testing

This document describes the property-based testing and load/stress testing infrastructure added to the Rhiza project.

Overview

Rhiza now includes two additional types of testing:

  1. Property-Based Testing (using Hypothesis) - Tests that verify properties hold across a wide range of generated inputs
  2. Load/Stress Testing (using pytest-benchmark) - Tests that measure performance and verify stability under load

README Code Block Testing

The test file .rhiza/tests/sync/test_readme_validation.py automatically executes every python code block in README.md and checks that its output matches the adjacent ```result block. It also syntax-checks every bash block using bash -n.

Skipping individual blocks with +RHIZA_SKIP

To exclude a specific code block from being executed or syntax-checked, append +RHIZA_SKIP to the opening fence line:

```python +RHIZA_SKIP
# This block will NOT be executed or syntax-checked by the readme tests.
# Use it for illustrative examples, environment-specific code, or incomplete snippets.
from my_env import some_function
some_function()
```

```bash +RHIZA_SKIP
# This bash block will NOT be syntax-checked.
run-something --only-on-ci
```

Markdown renderers (including GitHub) ignore everything after the first word on a fence line, so the block still renders as a normal syntax-highlighted code block. All blocks that do not carry +RHIZA_SKIP continue to be validated as before.

Property-Based Testing

Property-based tests use the Hypothesis library to automatically generate test cases that verify certain properties always hold true.

Locations

In a standard Rhiza project, there are two relevant locations for property-based tests:

  • Project property-based tests live in tests/property/. These are part of your normal test suite and are discovered by pytest via pytest.ini:testpaths = tests.
  • Rhiza's own template/internal property-based tests (if present) live in .rhiza/tests/property/. These are not part of your project's main test suite by default.

Running Property-Based Tests

In a typical setup, the Make targets map to these suites as follows:

  • make test: runs pytest over everything under tests/ (including tests/property/).
  • make rhiza-test: runs Rhiza's internal tests under .rhiza/tests/ (including .rhiza/tests/property/ if any exist).

You can also invoke the corresponding pytest commands directly:

# Run all project property-based tests (what make test covers)
uv run pytest tests/property/ -v

# Run Rhiza's internal/template property-based tests (if you have any in .rhiza)
uv run pytest .rhiza/tests/property/ -v

# Run project property-based tests with more examples (increase coverage)
uv run pytest tests/property/ -v --hypothesis-max-examples=1000

# Run project property-based tests with verbose Hypothesis output
uv run pytest tests/property/ -v --hypothesis-verbosity=verbose

Opting in to live DEBUG logs

By default, the template disables live pytest CLI logging (log_cli = false) to keep normal test output concise. When you need detailed live logs for debugging, enable them per-run:

uv run pytest -o log_cli=true --log-cli-level=DEBUG

Example Tests

The following property-based tests are included as examples:

Generic Property Tests

  • test_sort_correctness_using_properties: Verifies that sorted() correctly orders lists and preserves all elements including duplicates

Load/Stress Testing

Load and stress tests use pytest-benchmark to measure performance and verify system stability under load.

Location

Benchmark and stress tests are located in tests/benchmarks/ (if present)

Running Benchmark Tests

# Run all benchmarks
make benchmark

# Or with pytest directly
uv run pytest tests/benchmarks/ -v

# Run benchmarks and generate histogram
uv run pytest tests/benchmarks/ --benchmark-histogram=_tests/benchmarks/histogram

# Run benchmarks and save results
uv run pytest tests/benchmarks/ --benchmark-json=_tests/benchmarks/results.json

# Skip benchmarks (for CI)
uv run pytest tests/benchmarks/ --benchmark-skip

# Run only stress tests (note: these don't run with make benchmark by default)
uv run pytest tests/benchmarks/ -m stress -v

# Skip stress tests (run only performance benchmarks)
uv run pytest tests/benchmarks/ -m "not stress" -v

Note: The make benchmark target runs with --benchmark-only, which means stress tests (that don't use the benchmark fixture) will be skipped. To run stress tests explicitly, use uv run pytest tests/benchmarks/ -m stress -v.

Benchmark Test Categories

1. Makefile Performance

Tests that measure the performance of common Makefile operations:

  • test_help_target_performance - Measures help target execution time
  • test_print_variable_performance - Measures variable printing performance
  • test_dry_run_install_performance - Measures dry-run install performance
  • test_makefile_parsing_overhead - Measures Makefile parsing overhead

2. File System Operations

Tests that benchmark file system operations:

  • test_directory_traversal_performance - Measures directory traversal speed
  • test_file_reading_performance - Measures file reading performance
  • test_multiple_file_checks_performance - Measures file existence checking

3. Subprocess Overhead

Tests that measure subprocess creation overhead:

  • test_subprocess_creation_overhead - Measures subprocess creation time
  • test_git_command_performance - Measures git command execution time

4. Stress Scenarios

Tests that verify stability under load (marked with @pytest.mark.stress):

  • test_repeated_help_invocations - Stress tests repeated help invocations (100 iterations)
  • test_concurrent_print_variable_stress - Tests concurrent Makefile invocations (deterministic)
  • test_file_system_stress - Tests rapid file creation/deletion (100 iterations)

Note: Stress tests can be slow and are marked with the stress marker. They don't use the benchmark fixture, so they won't run with make benchmark (which uses --benchmark-only). Use uv run pytest tests/benchmarks/ -m stress -v to run them explicitly.

Understanding Benchmark Results

Benchmark output includes:

  • Min/Max: Minimum and maximum execution times
  • Mean: Average execution time
  • StdDev: Standard deviation (consistency)
  • Median: Median execution time
  • IQR: Interquartile range
  • Outliers: Number of outlier measurements
  • OPS: Operations per second (1/Mean)

Example output:

--------------------------------------------------- benchmark: 1 tests ---------------------------------------------------
Name (time in ms)                    Min      Max     Mean  StdDev   Median     IQR  Outliers      OPS  Rounds  Iterations
--------------------------------------------------------------------------------------------------------------------------
test_help_target_performance     16.5255  18.0592  16.9294  0.3194  16.8354  0.4791      15;1  59.0689      55           1
--------------------------------------------------------------------------------------------------------------------------

Integration with CI/CD

Running in CI

The property-based tests run as part of the regular test suite:

# Run all tests including property-based tests
make test

Benchmarks can be run separately or as part of validation:

# Run benchmarks
make benchmark

Best Practices

Writing Property-Based Tests

  1. Focus on invariants: Test properties that should always hold true
  2. Use appropriate strategies: Choose Hypothesis strategies that generate realistic inputs
  3. Keep tests fast: Property tests run multiple times, so keep them quick
  4. Test edge cases: Use @example decorator to test specific known edge cases

Example:

from hypothesis import given, strategies as st, example

@given(version=st.from_regex(r"^\d+\.\d+\.\d+$", fullmatch=True))
@example(version="0.0.0")  # Test specific edge case
def test_version_parsing(version):
    parts = version.split(".")
    assert len(parts) == 3
    assert all(p.isdigit() for p in parts)

Writing Benchmark Tests

  1. Benchmark real operations: Test actual operations users will perform
  2. Use fixtures wisely: Use module or session-scoped fixtures for expensive setup
  3. Test multiple scenarios: Benchmark best case, average case, and worst case
  4. Monitor trends: Track benchmark results over time to detect regressions

Example:

def test_operation_performance(benchmark):
    def run_operation():
        # Operation to benchmark
        return perform_operation()
    
    result = benchmark(run_operation)
    assert result is not None

Writing Stress Tests

  1. Test realistic scenarios: Simulate real-world usage patterns
  2. Set reasonable thresholds: Allow small failure rates for resource contention
  3. Test concurrency: Use ThreadPoolExecutor or ProcessPoolExecutor for concurrent tests
  4. Monitor resource usage: Consider memory, CPU, and I/O in addition to time

Example:

def test_concurrent_operations(root):
    import concurrent.futures
    
    def operation():
        # Operation to stress test
        return perform_operation()
    
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        futures = [executor.submit(operation) for _ in range(100)]
        results = [f.result() for f in concurrent.futures.as_completed(futures)]
    
    success_rate = sum(results) / len(results)
    assert success_rate == 1.0  # Rhiza template stress tests require 100% success

Dependencies

The following dependencies are required for property-based and load/stress testing:

# Property-based testing
hypothesis>=6.150.0

# Benchmarking and performance testing
pytest-benchmark>=5.2.3
pygal>=3.1.0

These are provisioned on the fly by the relevant make targets via uv run --with … (e.g. make test, make benchmark), so no separate install step is required.

Troubleshooting

Hypothesis Hangs or Times Out

If Hypothesis tests hang, you can:

  1. Reduce the number of examples: pytest --hypothesis-max-examples=10
  2. Set a deadline: pytest --hypothesis-deadline=1000
  3. Use the CI profile: pytest --hypothesis-profile=ci

Benchmarks Vary Too Much

If benchmark results have high variance:

  1. Close other applications to reduce system load
  2. Increase the number of rounds: pytest --benchmark-min-rounds=10
  3. Run on a consistent environment (CI is preferred for accurate benchmarks)

Stress Tests Fail

If stress tests fail occasionally:

  1. Check system resources (memory, CPU)
  2. Increase acceptable failure rate if resource contention is expected
  3. Reduce iteration count for local development

References