Experiment

Experiment

A single experiment, characterized by a confusion matrix.

It is responsible for generating synthetic confusion matrices, and computing metrics.

It is typically part of an ExperimentGroup, which in turn is part of a Study. The Study and ExperimentGroup are responsible for passing properly validated initialization parameters.

Parameters:

• name (str) –

  the name of this experiment

• rng (RNG) –

  the RNG used to control randomness

• confusion_matrix (Int[ndarray, 'num_classes num_classes']) –

  the confusion matrix for this experiment.

• prevalence_prior (str | float | Float[ArrayLike, 'num_classes'], default: 0 ) –

  the prior over the prevalence counts for this experiments. Defaults to 0, Haldane's prior.

• confusion_prior (str | float | Float[ArrayLike, 'num_classes num_classes'], default: 0 ) –

  the prior over the confusion counts for this experiments Defaults to 0, Haldane's prior.

Attributes

num_classes property

The number of classes in this experiment.

num_predictions property

The total number of predictions in the observed confusion matrix.

confusion_matrix = confusion_matrix instance-attribute

prevalence_prior = dirichlet_prior(strategy=prevalence_prior, shape=(self.num_classes,)) instance-attribute

confusion_prior = dirichlet_prior(strategy=confusion_prior, shape=(self.num_classes, self.num_classes)) instance-attribute

Sampling

sample

Sample synthetic confusion matrices for this experiment.

Parameters:

• sampling_method (SamplingMethod) –

  the sampling method used to generate the metric values. Must a member of the SamplingMethod enum

• num_samples (int) –

  the number of synthetic confusion matrices to sample

Returns:

• dict[MetricLike, ExperimentResult] –

  dict[MetricLike, ExperimentResult]: a dictionary of RootMetric instances

sample_metrics

Computes metrics over the synthetic confusion matrices.

Parameters:

• metrics (MetricCollection) –

  the metrics needed to be computed on the synthetic confusion matrices

• sampling_method (SamplingMethod) –

  the sampling method used to generate the metric values. Must a member of the SamplingMethod enum

• num_samples (int) –

  the number of synthetic confusion matrices to sample

Returns:

• dict[MetricLike, ExperimentResult] –

  dict[MetricLike, ExperimentResult]: a mapping from metric instance to an ExperimentResult instance

ExperimentResult dataclass

A simple wrapper class for results from an experiment.

Essentially just combines the metadata of an Experiment with a Metric, and stores its output.

Parameters:

• experiment (Experiment) –

  the experiment which produced these results

• metric (MetricLike) –

  the metric instance that produced these results

• values (Float[ndarray, ' num_samples #num_classes']) –

  the actual produced values

Attributes

is_multiclass property

Whether the metric that produced this result is binary or multiclass.

bounds property

The minimum and maximum possible values of the metric that produced this result.

num_classes property

The number of classes in the experiment that poduced this result.

num_samples property

How many confusion matrices were sampled.