safeai.rgr.compare_rgr

safeai.rgr.compare_rgr(models, strengths, class_order, *, method: Literal['noise', 'adversarial', 'wasserstein_images', 'spatial_images'] = 'noise', y_true=None, y_true_dict=None, images=None, attack_model=None, preprocess_fn=None, attack_name: Literal['fgsm', 'pgd', 'square', 'hsj', 'simba'] = 'fgsm', base_attack_params=None, rga_dict=None, class_weights=None, fig_size=(12, 6), verbose=True, random_seed=None, save_path=None, show=False, max_iter=50, eps_step=0.01, num_translations=3, num_rotations=3)[source]

Compare several models using Rank Graduation Robustness (RGR) curves.

This is the main user-facing comparison function for RGR. It provides one unified interface for all supported robustness workflows:

  • Gaussian feature noise, using method=’noise’

  • Feature-space adversarial attacks, using method=’adversarial’

  • Image-level Wasserstein attacks, using method=’wasserstein_images’

  • Image-level spatial transformations, using method=’spatial_images’

The function returns one result dictionary per model and can optionally plot a comparison of the resulting RGR curves.

Parameters:

  • models (dict) –

    Dictionary containing model configurations.

    For method=’noise’ or method=’adversarial’, each entry must have the form:

    model_name -> (
    model,
    x_data,
    prob_original,
    model_class_order,
    model_type,
    device
)

    where:

    • model is a trained sklearn estimator or PyTorch module.

    • x_data is the feature matrix used for perturbation.

    • prob_original is the original probability matrix, or None.

    • model_class_order is the class order of the model probability output.

    • model_type is either ‘sklearn’ or ‘pytorch’.

    • device is the torch device for PyTorch models, or None for sklearn.

    For method=’wasserstein_images’ or method=’spatial_images’, each entry must have the form:

    model_name -> (
    model,
    prob_original,
    model_class_order,
    model_type,
    device
)

    In image-level workflows, perturbed images are first generated using attack_model, then converted to model-ready features through preprocess_fn.

  • strengths (array-like) –

    Perturbation strengths used to build the RGR curves.

    Their meaning depends on method:

    • method=’noise’:

      Gaussian noise standard deviations.

    • method=’adversarial’:

      Attack strengths for the selected ART attack.

    • method=’wasserstein_images’:

      Wasserstein attack eps values.

    • method=’spatial_images’:

      Spatial transformation strengths.

  • class_order (array-like) – Shared target class order used to align probability columns across all models.

  • method ({'noise', 'adversarial', 'wasserstein_images', 'spatial_images'}, default='noise') – Robustness workflow used for all models in the comparison.

  • y_true (array-like, optional) –

    Shared true labels.

    Required for method=’wasserstein_images’ and method=’spatial_images’.

    For method=’adversarial’, either y_true or y_true_dict must be provided.

  • y_true_dict (dict, optional) – Per-model true labels for adversarial evaluation. This is useful when different models are evaluated on different input arrays.

  • images (array-like or torch.Tensor, optional) – Original image tensor or image array used for image-level robustness evaluation. Required for method=’wasserstein_images’ and method=’spatial_images’.

  • attack_model (torch.nn.Module, optional) – PyTorch model used by ART to generate image-level adversarial examples or spatial transformations. Required for image-level methods.

  • preprocess_fn (callable, optional) – Function that maps perturbed images to model-ready feature matrices. Required for image-level methods.

  • attack_name ({'fgsm', 'pgd', 'square', 'hsj', 'simba'}, default='fgsm') – ART adversarial attack used when method=’adversarial’.

  • base_attack_params (dict, optional) – Additional fixed parameters passed to the ART attack constructor. The current strength value is inserted automatically using the appropriate parameter name for the selected attack.

  • rga_dict (dict, optional) – Mapping from model name to full RGA score. If provided, each RGR curve is rescaled by the corresponding RGA value.

  • class_weights (array-like, optional) – Weights used to aggregate per-class RGR values. If None, uniform class weights are used.

  • fig_size (tuple, default=(12, 6)) – Figure size used for the comparison plot.

  • verbose (bool, default=True) – Whether to print progress and summary information.

  • random_seed (int, optional) – Random seed used for Gaussian noise generation when method=’noise’.

  • save_path (str, optional) – Path where the comparison plot should be saved. If None and show=False, no plot is saved.

  • show (bool, default=False) – Whether to display the comparison plot with plt.show().

  • max_iter (int, default=50) – Maximum number of iterations used by the Wasserstein image attack.

  • eps_step (float, default=0.01) – Step size used by the Wasserstein image attack.

  • num_translations (int, default=3) – Number of translations tested by the spatial transformation attack.

  • num_rotations (int, default=3) – Number of rotations tested by the spatial transformation attack.

Returns:

Mapping from model name to RGR result dictionary.

Each result contains:

  • ’rgr_scores’np.ndarray or list

    Raw RGR scores at each perturbation strength.

  • ’rgr_rescaled’np.ndarray or list

    RGR scores after optional rescaling by rga_dict.

  • ’aurgr’float

    Area under the RGR curve.

  • ’per_class_rgr’np.ndarray or list

    Per-class RGR values at each perturbation strength.

  • ’class_order’np.ndarray or list

    Class order used for probability alignment.

  • ’method’str

    Robustness workflow used in the comparison.

Depending on method, each result also includes one of:

  • ’noise_levels’

  • ’attack_strengths’

For adversarial methods, results also include:

  • ’attack_name’

Return type:

dict