benchmark-ctrs-experiments

benchmark-ctrs-experiments contains the method implementations and experiment configuration files used with the benchmark-ctrs framework for the thesis experiments on certified robustness with randomized smoothing.

Install this package next to benchmark-ctrs to make its models and losses available to the benchmark-ctrs CLI through Pluggy entry points.

Repository Layout

• src/benchmark_ctrs_experiments/models: training modules for thesis methods and reproduced baselines.
• src/benchmark_ctrs_experiments/losses: loss functions used by those modules.
• src/benchmark_ctrs_experiments/utilities: shared helpers for regularization, subsampling, and selective training.
• conf/cifar10: CIFAR-10 training, validation, and certification configs.
• conf/imagenet: ImageNet training, validation, and certification configs.
• conf/detailed-checkpointing.yaml: optional checkpointing override for saving every epoch.

Installation

Use Python 3.10 or newer. Install benchmark-ctrs first, then install this extension package.

git clone https://github.com/ratedali/benchmark-ctrs.git
git clone https://github.com/ratedali/benchmark-ctrs-experiments.git

cd benchmark-ctrs
python -m pip install -e .

cd ../benchmark-ctrs-experiments
python -m pip install -e .

For development tools:

python -m pip install -e ".[dev]"

The experiment configs use lightning.pytorch.loggers.wandb.WandbLogger. Install and configure Weights & Biases if you want online experiment tracking:

python -m pip install wandb
wandb login

For local reproduction without online logging, either set W&B offline:

export WANDB_MODE=offline

or append this override to any CLI command:

--trainer.logger=false

Data And Outputs

The core framework writes outputs to ./logs by default. Dataset caches default to ./datasets_cache, or to the path in DATASETS_CACHE when set.

export DATASETS_CACHE=/path/to/datasets

CIFAR-10 is downloaded automatically by torchvision. ImageNet must already exist under DATASETS_CACHE in torchvision's expected ImageNet directory layout.

The full experiment grid is computationally expensive. CIFAR-10 configs are reasonable on a single GPU. ImageNet configs in this repository assume a multi-GPU setup by default.

Usage

The package registers additional benchmark_ctrs entry points when installed. After installation, use the benchmark-ctrs command from the main package.

Configs are composed by passing multiple --config files. Later files override or extend earlier files.

Train a CIFAR-10 Gaussian-augmentation baseline with sigma 0.25:

benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/gaussian/sigma_0.25/train.yaml

Train a CIFAR-10 selective MACER variant:

benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/methods/selective/macer/sigma_0.25/train.yaml

Train an ImageNet baseline. The ImageNet default config uses DDP with four devices:

benchmark-ctrs fit \
  --config conf/imagenet/defaults.yaml \
  --config conf/imagenet/baselines/gaussian/sigma_0.25/train.yaml

For a smaller validation pass during development, add a validation override:

benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/gaussian/sigma_0.25/train.yaml \
  --config conf/cifar10/validation/reduced.yaml

Run certification or prediction from a trained checkpoint:

benchmark-ctrs predict \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/gaussian/sigma_0.25/train.yaml \
  --config conf/cifar10/certification/betting.yaml \
  --ckpt_path logs/lightning_logs/version_0/checkpoints/best.ckpt

Certification CSV files are written by benchmark_ctrs.callbacks.certified_radius_writer.CertifiedRadiusWriter when the selected certification config includes that callback.

Reproducing The Thesis Runs

The configuration hierarchy encodes the experiment grid. The general pattern is:

benchmark-ctrs fit \
  --config conf/<dataset>/defaults.yaml \
  --config conf/<dataset>/<family>/<method>/sigma_<sigma>/train.yaml

where:

• <dataset> is cifar10 or imagenet;
• <family> is baselines or methods;
• <method> is one of the method folders under that family;
• <sigma> is 0.25, 0.50, or 1.00.

Examples:

# CIFAR-10 MACER baseline, sigma 0.50
benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/macer/sigma_0.50/train.yaml

# CIFAR-10 projection MACER method, sigma 1.00
benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/methods/projection/macer/sigma_1.00/train.yaml

# ImageNet consistency baseline, sigma 0.25
benchmark-ctrs fit \
  --config conf/imagenet/defaults.yaml \
  --config conf/imagenet/baselines/consistency/sigma_0.25/train.yaml

After training, run certification with one of:

• conf/<dataset>/certification/clean.yaml
• conf/<dataset>/certification/full.yaml
• conf/<dataset>/certification/ub.yaml
• conf/<dataset>/certification/betting.yaml

Example:

benchmark-ctrs predict \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/macer/sigma_0.50/train.yaml \
  --config conf/cifar10/certification/full.yaml \
  --ckpt_path logs/lightning_logs/version_0/checkpoints/best.ckpt

For quick smoke tests, reduce runtime from the command line:

benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/gaussian/sigma_0.25/train.yaml \
  --trainer.max_epochs=1 \
  --data.init_args.validation=128 \
  --trainer.limit_train_batches=2 \
  --trainer.limit_val_batches=2 \
  --trainer.logger=false

Implemented Methods

Registered training modules include:

• benchmark_ctrs_experiments.models.adre.ADRE
• benchmark_ctrs_experiments.models.consistency.Consistency
• benchmark_ctrs_experiments.models.macer.MACER
• benchmark_ctrs_experiments.models.soft_macer.SoftMACER
• benchmark_ctrs_experiments.models.smoothadv.SmoothAdv
• benchmark_ctrs_experiments.models.reverse.standard.CIFARReverse
• benchmark_ctrs_experiments.models.reverse.standard.MNISTReverse
• benchmark_ctrs_experiments.models.reverse.gaussian_aug.GaussianAugReverse
• benchmark_ctrs_experiments.models.projection.module.RegularizedLossProjection
• benchmark_ctrs_experiments.models.selective.regularized.SelectionRegularized

Registered losses include:

• benchmark_ctrs_experiments.losses.adre.ADRELoss
• benchmark_ctrs_experiments.losses.consistency.ConsistencyLoss
• benchmark_ctrs_experiments.losses.macer.MACERLoss
• benchmark_ctrs_experiments.losses.soft_macer.SoftMACERLoss
• benchmark_ctrs_experiments.losses.perturbed_loss.PerturbedLossWrapper
• benchmark_ctrs_experiments.losses.rce.RCELoss

Extending Or Implementing New Methods

Most new methods should be added as a new BaseModule subclass and registered through the extension entry point.

1. Add the module under src/benchmark_ctrs_experiments/models/.
2. Return {"loss": loss, "predictions": predictions} from both training_step() and validation_step().
3. Register the class in src/benchmark_ctrs_experiments/register_modules.py.
4. Add a YAML config under conf/<dataset>/methods/<method>/sigma_<sigma>/.
5. Run benchmark-ctrs fit --print_config or a one-epoch smoke test.

Minimal model skeleton:

from typing import Any

from torch.nn import CrossEntropyLoss

from benchmark_ctrs.modules import BaseModule
from benchmark_ctrs.types import Batch, StepOutput


class MyMethod(BaseModule):
    def __init__(
        self,
        *args: Any,
        criterion: CrossEntropyLoss | None = None,
        **kwargs: Any,
    ):
        super().__init__(*args, **kwargs)
        self.save_hyperparameters(ignore=self.ignore_hyperparameters)
        self.criterion = criterion or CrossEntropyLoss()

    def training_step(self, batch: Batch, *args: Any, **kwargs: Any) -> StepOutput:
        inputs, targets = batch[:2]
        predictions = self.forward(inputs)
        loss = self.criterion(predictions, targets)
        return {"loss": loss, "predictions": predictions}

    def validation_step(self, batch: Batch, *args: Any, **kwargs: Any) -> StepOutput:
        inputs, targets = batch[:2]
        predictions = self.forward(inputs)
        loss = self.criterion(predictions, targets)
        return {"loss": loss, "predictions": predictions}

Register it:

from benchmark_ctrs import plugins
from benchmark_ctrs_experiments.models.my_method import MyMethod


@plugins.hookimpl
def register_models():
    return (MyMethod,)

Use it from YAML:

name: my_method
model:
  class_path: benchmark_ctrs_experiments.models.my_method.MyMethod
  init_args:
    sigma: 0.25
    certification_params:
      sigma: 0.25

For loss-only extensions, implement a callable torch.nn.Module loss and register it with register_criterions(). For methods that need custom certification, implement benchmark_ctrs.certification.CertificationMethod in the main framework style and register it with register_certification_methods().

Development Checks

Run Ruff:

ruff check .

Run basic package and plugin smoke checks after editing registration or metadata:

python -m pip install -e .
python -c "import benchmark_ctrs_experiments; print(benchmark_ctrs_experiments.__version__)"
benchmark-ctrs fit --print_config

Run a one-epoch CLI smoke test:

benchmark-ctrs fit \
  --config conf/cifar10/defaults.yaml \
  --config conf/cifar10/baselines/gaussian/sigma_0.25/train.yaml \
  --trainer.max_epochs=1 \
  --trainer.limit_train_batches=1 \
  --trainer.limit_val_batches=1 \
  --trainer.logger=false

License

This project is licensed under the MIT License. See LICENSE.