SCE (WACV 2023)¶
Introduction¶
This repository contains the official Pytorch implementation of Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning (SCE) that has been published in the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023.
Data preparation¶
Data preparation details are available here.
Main results¶
The following results are the main ones reported in our paper:
| pretrain epochs | pretrain crops | ImageNet linear accuracy | ckpt |
| 100 | 2x224 | 72.1 | Download |
| 200 | 2x224 | 72.7 | Download |
| 300 | 2x224 | 73.3 | Download |
| 1000 | 2x224 | 74.1 | Download |
| 200 | 2x224 + 192 + 160 + 128 + 96 | 75.4 | Download |
You can find below the command lines to launch the configs to retrieve those results and above the checkpoint links.
SCE Pretraining¶
We launched our experiments on a computational cluster configured via SLURM using for the two crops configuration 8 A100-80G GPUs and for the multi-crop configuration 16 A100-80G GPUs.
We provide below the commands using the srun command from SLURM that was inside a SLURM script. Pytorch-Lightning directly detects SLURM is used and configures accordingly the distributed training. We strongly suggest you refer to Pytorch-Lightning’s documentation to correctly set up a command line without srun if you do not have access to a slurm cluster.
Pretraining for 100 epochs¶
output_dir=...
dataset_dir=...
config_path="../eztorch/configs/run/pretrain/sce/resnet50"
config_name="resnet50_imagenet"
seed=42
cd eztorch/run
srun --kill-on-bad-exit=1 python pretrain.py \
-cp $config_path -cn $config_name\
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='pretrain' \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8 \
trainer.max_epochs=100 \
model.optimizer.initial_lr=0.6
Pretraining for 200 epochs¶
srun --kill-on-bad-exit=1 python pretrain.py \
-cp $config_path -cn $config_name\
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='pretrain' \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8
Pretraining for 300 epochs¶
srun --kill-on-bad-exit=1 python pretrain.py \
-cp $config_path -cn $config_name\
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='pretrain' \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8 \
trainer.max_epochs=300
Pretraining for 1000 epochs¶
srun --kill-on-bad-exit=1 python pretrain.py \
-cp $config_path -cn $config_name\
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='pretrain' \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8 \
trainer.max_epochs=1000 \
model.optimizer.params.weight_decay=1.5e-6
Pretraining for 200 epochs with multi-crop¶
config_name="resnet50_imagenet_five_crops"
srun --kill-on-bad-exit=1 python pretrain.py \
-cp $config_path -cn $config_name \
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='pretrain' \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8 \
trainer.num_nodes=2
Linear classification¶
Same as for pretraining we launched our experiment on a SLURM cluster.
eval_config_path="../eztorch/configs/run/evaluation/linear_classifier/sce/resnet50"
eval_config_name="resnet50_imagenet"
pretrain_checkpoint=...
srun --kill-on-bad-exit=1 python linear_classifier_evaluation.py \
-cp $eval_config_path -cn $eval_config_name \
dir.data=$dataset_dir \
dir.root=$output_dir \
dir.exp='linear_classifier_evaluation' \
model.pretrained_trunk_path=$pretrain_checkpoint \
seed.seed=$seed \
datamodule.train.loader.num_workers=8 \
datamodule.val.loader.num_workers=8 \
trainer.devices=8
We consider by default you use checkpoints you pretrained yourselves.
If this is not the case and you downloaded the checkpoints we provide, do not forget to change the model.trunk_pattern config that searches the trunk pattern in the state dict:
srun --kill-on-bad-exit=1 python linear_classifier_evaluation.py
...
model.trunk_pattern="" \
...
Transfer Learning¶
For Transfer Learning evaluation we used code provided by several authors that we would like to thank for sharing their work. Below we redirect you to their Github for every transfer we have done.
All evaluations are based on the multi-crop checkpoint.
Linear classifier on other datasets¶
To evaluate the transferability of our pretrained checkpoints on various datasets by training a linear classifier, we used the ssl-transfer repository.
| Method | Food101 | CIFAR10 | CIFAR100 | SUN397 | Cars | Aircraft | VOC2007 | DTD | Pets | Caltech101 | Flowers | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SimCLR | 72.8 | 90.5 | 74.4 | 60.6 | 49.3 | 49.8 | 81.4 | 75.7 | 84.6 | 89.3 | 92.6 | 74.6 |
| BYOL | 75.3 | 91.3 | 78.4 | 62.2 | 67.8 | 60.6 | 82.5 | 75.5 | 90.4 | 94.2 | 96.1 | 79.5 |
| NNCLR | 76.7 | 93.7 | 79.0 | 62.5 | 67.1 | 64.1 | 83.0 | 75.5 | 91.8 | 91.3 | 95.1 | 80 |
| SCE | 77.7 | 94.8 | 80.4 | 65.3 | 65.7 | 59.6 | 84.0 | 77.1 | 90.9 | 92.7 | 96.1 | 80.4 |
| Supervised | 72.3 | 93.6 | 78.3 | 61.9 | 66.7 | 61.0 | 82.8 | 74.9 | 91.5 | 94.5 | 94.7 | 79.3 |
SVM classifier on PASCAL VOC 2007/2012¶
To evaluate the transferability of our pretrained checkpoints on PASCAL VOC by training an SVM classifier, we used the PCL repository.
| Method | K = 16 | K = 32 | K = 64 | full |
|---|---|---|---|---|
| MoCov2 | 76.14 | 79.16 | 81.52 | 84.60 |
| PCLv2 | 78.34 | 80.72 | 82.67 | 85.43 |
| ReSSL | 79.17 | 81.96 | 83.81 | 86.31 |
| SwAV | 78.38 | 81.86 | 84.40 | 87.47 |
| WCL | 80.24 | 82.97 | 85.01 | 87.75 |
| SCE | 79.47 | 83.05 | 85.47 | 88.24 |
Object detection and Mask Segmentation on COCO¶
To evaluate the transferability of our pretrained checkpoints on COCO by training a Mask R-CNN for object detection and mask segmentation, we used the triplet repository.
| Method | AP Box | AP Mask |
|---|---|---|
| MoCo | 40.9 | 35.5 |
| MoCov2 | 40.9 | 35.5 |
| SimCLR | 39.6 | 34.6 |
| BYOL | 40.3 | 35.1 |
| SCE | 41.6 | 36.0 |
| Truncated-Triplet | 41.7 | 36.2 |
| Supervised | 40.0 | 34.7 |
Issue¶
If you found an error, have trouble making this work or have any questions, please open an issue to describe your problem.
Acknowledgment¶
This work was made possible by the use of the Factory-AI supercomputer, financially supported by the Ile-de-France Regional Council.
Citation¶
If you found our work useful, please consider citing us:
@InProceedings{Denize_2023_WACV,
author = {Denize, Julien and Rabarisoa, Jaonary and Orcesi, Astrid and H\'erault, Romain and Canu, St\'ephane},
title = {Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning},
booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
month = {January},
year = {2023},
pages = {2706-2716}
}