Skip to content

CLAUDE.md

Latest commit

 

History

History
131 lines (94 loc) · 7.41 KB

CLAUDE.md

File metadata and controls

131 lines (94 loc) · 7.41 KB
Raw

Pyramid Scale Guidance — SD WebUI Extension

Overview

SD WebUI extension that applies multi-scale pyramid guidance during denoising. Two pyramid modes are supported:

  • Gaussian Blur (default): Each pyramid level applies progressively stronger Gaussian blur (σ_k = max_σ × k/(K-1))
  • Down-Up Sampling: Each pyramid level downsamples to factor^k of original size (area averaging) then upsamples back (bicubic interpolation)

At each sampling step, the denoised latent is VAE-decoded to pixel space, degraded at multiple scales, VAE-encoded back, and each scale gets modified CLIP text conditioning and per-level CFG. The UNet runs once per pyramid level per step; outputs are weighted-averaged.

Architecture

Per Denoising Step:

  Denoised z ──┬── [Level 0: original] ── Standard UNet output ─────┐
               │                                                      │
               ├── VAE decode ──┬── Degrade(1) ── VAE encode ── UNet ┤
               │                ├── Degrade(2) ── VAE encode ── UNet ┤
               │                └── Degrade(K) ── VAE encode ── UNet ┤
               │                                                      │
               └──────────── Weighted Average ────────────────── Output

  Degrade = Blur(σ_k)               [Gaussian Blur mode]
          | Downsample(f^k) → Upsample  [Down-Up Sampling mode]

Hooks: on_cfg_denoiser (store refs) → cfg_after_cfg (run pyramid levels, weighted average into params.x)

Key Directories

  • pyramid_scale_guidance/ — Core library (types, blur, downsample, VAE, conditioning, denoiser, scheduling, tagger, frequency bands)
  • scripts/pyramid_scale_guidance.py — WebUI script (UI + hook registration)
  • tests/ — Unit tests (pytest), golden test runner (real SD model), parameter sweep
  • docs/ — UI screenshot
  • scripts/generate_readme_tables.py — Regenerates the image comparison section of README.md (rows=seeds, columns=configs, grouped by technique). Auto-run by serve_readme.sh.

Linting

./scripts/lint.sh          # auto-fix safe issues, then check
./scripts/lint.sh --check  # check only (no fixes)

Configured via ruff.toml (line-length=120, E402 ignored for files with sys.path manipulation).

Convention: Always use zip(..., strict=True) to catch length mismatches early. Ruff enforces this via B905.

Running Tests

# Unit tests (no GPU needed, ~60s)
# golden_test_runner.py and parameter_sweep.py are auto-excluded via conftest.py collect_ignore
./scripts/run_unit_tests.sh
# or equivalently:
python -m pytest tests/ -v

# Golden tests with real SD model (offline by default, run directly as scripts)
python tests/golden_test_runner.py --sd-steps 10
python tests/golden_test_runner.py --level 9 10 11  # specific levels
python tests/golden_test_runner.py --all-prompts     # all 12 prompts
python tests/golden_test_runner.py --online           # allow HF Hub access

# Multi-seed golden tests (Wilcoxon signed-rank test for statistical significance)
python tests/golden_test_runner.py --seeds 42,123,7,2024,555,8888,31415
python tests/golden_test_runner.py --n-seeds 7        # use first 7 from DEFAULT_SEEDS
python tests/golden_test_runner.py --n-seeds 7 --all-prompts  # full matrix

# Mock tests (math-only, no SD model)
python tests/golden_test_runner.py --mock

# Parameter sweep (offline by default, run directly as script)
python tests/parameter_sweep.py --seeds 42,123,7
python tests/parameter_sweep.py --mode downsample     # downsample factor sweep
python tests/parameter_sweep.py --mode both            # blur + downsample
python tests/parameter_sweep.py --online               # allow HF Hub access

HF Hub Offline Mode

All tests and the main extension default to offline mode — no HF Hub network requests. Models must be pre-cached locally.

  • Tests: Pass --online flag to golden_test_runner.py or parameter_sweep.py to allow downloads
  • Extension UI: "Allow tagger model download (HF Hub)" checkbox (unchecked by default) controls Florence-2 model download
  • Env vars: HF_HUB_OFFLINE=1, TRANSFORMERS_OFFLINE=1, HF_DATASETS_OFFLINE=1 are set by default in test scripts

Optimal Defaults (from parameter sweep)

Based on 81-config × 3-seed sweep (Gaussian blur mode): K=2, σ=2.0, threshold=0.5, decode_every=3, enable_frequency_bands=True

Lower sigma, fewer pyramid levels, higher activation threshold, and less frequent VAE decoding consistently perform best. Down-up sampling mode defaults: K=2, factor=0.5, threshold=0.5, decode_every=3 (not yet sweep-validated).

Prompt-type suitability (from 28-level × 12-prompt × 8-seed analysis): Pyramid guidance benefits atmospheric/mood scenes (89% positive), portraits (64%), and structured text (64%), but generally degrades CLIP alignment for detailed scenes, abstract patterns, macro photography, and minimalist compositions (~70% of configs show negative delta overall).

Tag Classification

Florence-2 tags are split into content (→ positive prompt) and quality/blur (→ negative prompt for ALL levels). Three-stage pipeline:

  1. Filter redundant tags (word stems already in prompt)
  2. Hardcoded blur/quality tag list (word-boundary matching)
  3. CLIP embedding distance fallback for unknown tags

Florence-2 Setup

Florence-2 runs in a separate venv (.venv-florence/) due to dependency conflicts with SD WebUI. The tests/florence2_tagger.py subprocess handles model loading and inference.

Future Work

Phase 1: Frequency-Band Weighting Validation (DONE)

  • Frequency band logic added to run_pyramid_euler in golden_test_runner.py
  • Golden test configs 14-18 cover blur+FB, downsample+FB, hi/lo emphasis, control
  • Results: FB modulation helps blur mode (reduces CLIP loss from -0.0131 to -0.0068), downsample+FB works well (+0.0023)
  • Band energies computed and logged in golden test results

Phase 2: Improvement Proposals

  • Algorithm: Adaptive freq-band weights from energy, step-dependent band weights, sigma-adaptive depth, pixel-space frequency recomposition
  • Quality metrics: LPIPS, FID/KID, per-band energy ratios (no MS-SSIM)
  • Performance: Latent-space blur (skip VAE), cached VAE decode, shared coarse UNet cache, half-precision VAE
  • New features: Anisotropic blur, wavelet (DWT) decomposition, self-similarity guidance (DINO), SDXL/SD2.x support, ControlNet-aware blur/downsample

Phase 3: Broader Prompt/Seed Testing (DONE)

  • PROMPTS expanded from 3 to 12 (abstract, text-heavy, macro, fog, lightning, architecture, multi-subject, watercolor, minimalist)
  • Multi-seed support via --seeds or --n-seeds flags; DEFAULT_SEEDS = [42, 123, 7, 2024, 555, 8888, 31415]
  • Wilcoxon signed-rank test for statistical significance (requires scipy, >= 6 non-zero diffs)
  • _aggregate_seed_results() computes mean/std CLIP delta, mean LPIPS, Wilcoxon p-value
  • Downsample parameter sweep via --mode downsample (factor grid: 0.3, 0.5, 0.7)

Other

  • Manual WebUI verification: Live end-to-end test with hooks, UI, and actual image generation
  • Live CLIP classification test: Verify _sd_clip_encode against real WebUI model hijack layer
  • Tagger backend alternatives: Test WD14 and CLIP Interrogator backends