roofline.md

Roofline Method (CPU baseline)

Goal

• Quantify whether the CPU kernel (ref/code.cpp) is memory-bound or compute-bound.
• Provide ceilings (bandwidth and compute) and plot measured points.

Assumptions for this stencil

• Per lattice update (section0, no sources):
  • Approx FLOPs: 30 (15 mul + 14 add + 1 div; division counted as 1 flop for simplicity).
  • Bytes: 64 B (15 float loads + 1 float store).
  • Arithmetic intensity I ≈ 30 / 64 ≈ 0.469 flop/byte.

Measure bandwidth ceiling

1. Build triad:
   • cd roofline && make
2. Run a few sizes, take the best:
   • ./stream_triad 8000000 50 (adjust N to sweep working set; report BW)

Estimate compute ceiling

• Use a practical upper bound (e.g., vendor docs, microbenchmarks, or sustained peak from a dense compute kernel).
• For quick plotting, pass a conservative estimate (e.g., 200–400 GF/s on modern AVX2 CPUs is unrealistic; typical sustained might be 50–150 GF/s depending on cores/freq). Choose per your CPU.

Produce roofline from bench CSV

1. Generate CSV via safe sweep:
   • cd cpu_bench && bash run_sweep.sh
2. Plot roofline (replace the ceilings with your measured values):
   • python ../roofline/plot_roofline.py sweep_small.csv --peak-bw-gbs 40 --peak-gflops 200
3. Output: cpu_bench/sweep_small.roofline.png

Interpretation

• If the measured point lies on the sloped (BW-limited) region, optimizations should focus on reducing bytes/update (tiling, reuse, vectorization) or increasing effective bandwidth (NUMA/core pinning).
• If it lies under the flat (compute-limited) roof, focus on instruction-level efficiency (vectorization, FMA utilization, register pressure).

Next (GPU roofline)

• After CUDA baseline, measure device bandwidth (e.g., via cudaMemcpy or a device triad), set SM compute peak (from nvidia-smi --query-gpu=clocks,fp32), and replot using the same arithmetic intensity (or a revised one if the CUDA kernel changes bytes/update via shared-memory tiling).