Evaluation pipeline

Testing time augmentation

Get aggregation function

eztorch.evaluation.get_test_time_augmentation_fn(name, **kwargs)

Retrieve the test time augmentation given its name and optional arguments.

Parameters:

name (str) – Name of the testing time augmentation. Options are: 'avg_group', 'same_avg', 'same_max'.

Raises:

NotImplementedError – If name given is not supported.

Return type:

Callable

Returns:

The test time augmentation function.

Aggregation functions

eztorch.evaluation.average_group(values, labels, ids)

Perform average from values and aggregation of labels based on different ids.

Parameters:

• values (Tensor) – The values to average.

• labels (Tensor) – The labels associated to the values.

• ids (Tensor) – The different ids to group the values and labels.

Return type:

Tuple[Tensor]

Returns:

The average of values associated to its labels and ids.

eztorch.evaluation.average_same_num_aug(values, labels, ids, num_aug=30)

Perform average from values and aggregation of labels and ids on split tensors by specified number of augmentations.

Parameters:

• values (Tensor) – The values to average.

• labels (Tensor) – The labels associated to the values.

• ids (Tensor) – The different ids to group the values and labels.

• num_aug (int, optional) – The number of augmentations performed for test time augmentation.

  Default: 30

Return type:

Tuple[Tensor]

Returns:

The average of values associated to its labels and ids.

eztorch.evaluation.max_same_num_aug(values, labels, ids, num_aug=30)

Keep maximum from values and aggregation of labels and ids on split tensors by specified number of augmentations.

Parameters:

• values (Tensor) – The values to average.

• labels (Tensor) – The labels associated to the values.

• ids (Tensor) – The different ids to group the values and labels.

• num_aug (int, optional) – The number of augmentations performed for test time augmentation.

  Default: 30

Return type:

Tuple[Tensor]

Returns:

The average of values associated to its labels and ids.

Linear Classifier

class eztorch.evaluation.LinearClassifierEvaluation(trunk, classifier, optimizer, pretrained_trunk_path, trunk_pattern='^(trunk\\\\.)', train_transform=None, val_transform=None, test_transform=None, val_time_augmentation=None, test_time_augmentation=None)

Linear classifier evaluation for self-supervised learning.

Parameters:

• trunk (DictConfig) – Config to build a trunk.

• classifier (DictConfig) – Config to build a classifier.

• optimizer (DictConfig) – Config to build an optimizer.

• pretrained_trunk_path (str) – Path to the pretrained trunk file.

• trunk_pattern (str, optional) – Pattern to retrieve the trunk model in checkpoint state_dict and delete the key.

  Default: '^(trunk\.)'

• train_transform (Optional[DictConfig], optional) – Config to perform transformation on train input.

  Default: None

• val_transform (Optional[DictConfig], optional) – Config to perform transformation on val input.

  Default: None

• test_transform (Optional[DictConfig], optional) – Config to perform transformation on test input.

  Default: None

• val_time_augmentation (Optional[DictConfig], optional) – If not None, ensembling method for test time augmentation used at validation.

  Default: None

• test_time_augmentation (Optional[DictConfig], optional) – If not None, ensembling method for test time augmentation used at test.

  Default: None

Feature Extractor

class eztorch.evaluation.FeatureExtractor(trunk, pretrained_trunk_path, head=None, trunk_pattern='^(trunk\\\\.)', train_transform=None, val_transform=None, test_transform=None, train_time_augmentation=None, val_time_augmentation=None, test_time_augmentation=None, storage_prefix_name='')

Feature extractor.

Parameters:

• trunk (DictConfig) – Config to build a trunk.

• pretrained_trunk_path (str) – Path to the pretrained trunk file.

• head (Optional[DictConfig], optional) – Config to build a head such as for pooling, normalizing, ….

  Default: None

• trunk_pattern (str, optional) – Pattern to retrieve the trunk model in checkpoint state_dict and delete the key.

  Default: '^(trunk\.)'

• train_transform (Optional[DictConfig], optional) – Config to perform transformation on train input.

  Default: None

• val_transform (Optional[DictConfig], optional) – Config to perform transformation on val input.

  Default: None

• test_transform (Optional[DictConfig], optional) – Config to perform transformation on test input.

  Default: None

• train_time_augmentation (Optional[DictConfig], optional) – If not None, ensembling method for test time augmentation used for the training data.

  Default: None

• val_time_augmentation (Optional[DictConfig], optional) – If not None, ensembling method for test time augmentation used for the validation data.

  Default: None

• test_time_augmentation (Optional[DictConfig], optional) – If not None, ensembling method for test time augmentation used for the test data.

  Default: None

• storage_prefix_name (str, optional) – Prefix of the files to store features, labels and ids. The split is added before the prefix, example for train split: f”train{prefix}_features.pth”.

  Default: ''

class eztorch.evaluation.SoccerNetFeatureExtractor(trunk, dim_features, filename, pretrained_trunk_path, head=None, trunk_pattern='^(trunk\\\\.)', train_transform=None, val_transform=None, test_transform=None, fps_in=4, fps_out=2)

Feature extractor.

Parameters:

• trunk (DictConfig) – Config to build a trunk.

• dim_features (int) – Dimension of the features.

• filename (str) – Filename of the files to store features following the naming convention of SoccerNet Baidu features.

• pretrained_trunk_path (str) – Path to the pretrained trunk file.

• head (Optional[DictConfig], optional) – Config to build a head such as for pooling, normalizing, ….

  Default: None

• trunk_pattern (str, optional) – Pattern to retrieve the trunk model in checkpoint state_dict and delete the key.

  Default: '^(trunk\.)'

• train_transform (Optional[DictConfig], optional) – Config to perform transformation on train input.

  Default: None

• val_transform (Optional[DictConfig], optional) – Config to perform transformation on val input.

  Default: None

• test_transform (Optional[DictConfig], optional) – Config to perform transformation on test input.

  Default: None

• fps_in (int, optional) – Fps of the input.

  Default: 4

• fps_out (int, optional) – Fps of extraction.

  Default: 2

class eztorch.evaluation.SpotFeatureExtractor(trunk, dim_features, filename, pretrained_trunk_path, head=None, trunk_pattern='^(trunk\\\\.)', train_transform=None, val_transform=None, test_transform=None)

Feature extractor.

Parameters:

• trunk (DictConfig) – Config to build a trunk.

• dim_features (int) – Dimension of the features.

• filename (str) – Filename of the files to store features following the naming convention of SoccerNet Baidu features.

• pretrained_trunk_path (str) – Path to the pretrained trunk file.

• head (Optional[DictConfig], optional) – Config to build a head such as for pooling, normalizing, ….

  Default: None

• trunk_pattern (str, optional) – Pattern to retrieve the trunk model in checkpoint state_dict and delete the key.

  Default: '^(trunk\.)'

• train_transform (Optional[DictConfig], optional) – Config to perform transformation on train input.

  Default: None

• val_transform (Optional[DictConfig], optional) – Config to perform transformation on val input.

  Default: None

• test_transform (Optional[DictConfig], optional) – Config to perform transformation on test input.

  Default: None

• window_duration – Duration of each window. Used to initialize features dimension.

Example:

trunk = {...} # config to build a trunk
pretrained_trunk_path = ... # path where the trunk has been saved

model = FeatureExtractor(trunk, pretrained_trunk_path)

NMS

eztorch.evaluation.perform_all_classes_NMS(predictions, step_timestamp, window=60, threshold=0.5, min_weight=0.0, nms_type='hard')

Perform NMS on all classes.

Parameters:

• predictions (Tensor) – Predictions on which to perform NMS.

• step_timestamp (float) – The step between each timestamps in the video.

• window (int | list[int], optional) – The window used for NMS.

  Default: 60

• threshold (float, optional) – The threshold used for NMS.

  Default: 0.5

• min_weight (float, optional) – Minimum weight for decay function in soft nms.

  Default: 0.0

• nms_type (str, optional) – Type of NMS to apply, ‘soft’ or ‘hard’.

  Default: 'hard'

Return type:

Tuple[List[Tensor], List[Tensor]]

Returns:

The list of predictions per class aswell as their corresponding timestamp.

eztorch.evaluation.perform_soft_NMS(values, window, threshold=0.001, min_weight=0.0)

Perform soft NMS on some values.

Parameters:

• values (Tensor) – Values on which to perform soft NMS.

• window (int) – The window used for NMS.

• threshold (float, optional) – The threshold used for soft NMS.

  Default: 0.001

• min_weight (float, optional) – Minimum weight to decay the values.

  Default: 0.0

Returns:

The decayed values.

eztorch.evaluation.perform_hard_NMS(values, window=60, threshold=0.001)

Perform hard NMS on some values.

Parameters:

• values (Tensor) – Values on which to perform hard NMS.

• window (int, optional) – The window used for NMS.

  Default: 60

• threshold (float, optional) – The threshold used for hard NMS.

  Default: 0.001

Returns:

The boolean tensor of values to keep.

eztorch.evaluation.perform_hard_NMS_per_class(values, windows, threshold=0.001)

Perform hard NMS per class on some values.

Parameters:

• values (Tensor) – Values on which to perform hard NMS.

• windows (list[int]) – The windows used for NMS for each class.

• threshold (float, optional) – The threshold used for hard NMS.

  Default: 0.001

Returns:

The boolean tensor of values to keep.