ReXNet-tiny on CIFAR10 PyTorch

Implementation of ReXNet-tiny model on CIFAR10 using PyTorch

It's unofficial code! [origin paper]

• ReXNet-tiny-cifar
• Various optimizer (SGD, SGDP, SGDMix, Adam, AdamP, AdamW)
• Scheduler (step, multistep, cosine)
• Augmentation (CutMix, RandAugment, RandAugmentv2 with CutMix)
• Comparison with EfficientNet-tiny
• Model Compression
• Add other models (EfficientNet, GhostNet, MobileNeXt, WideResNet) (TODO)
• Hyperparameter (TODO)

Abstract

(1) simple and effective design principles to mitigate the representational bottleneck

(2) study the matrix rank on the features generated by ten thousand random networks

Method

Run

ReXNet-tiny

CUDA_VISIBLE_DEVICES=0 python train.py --epochs 100 \
--savepath ./checkpoint/cifar10/sgd

ReXNet-tiny with SGDMix

CUDA_VISIBLE_DEVICES=0 python train_sgdmix.py --epochs 100 \
--savepath ./checkpoint/cifar10/sgdmix \
--name sgdmix

ReXNet-tiny with RandAugment12 and cutmix 0.5

CUDA_VISIBLE_DEVICES=0 python train_sgd_randaugcutout_cutmix.py \
--savepath ./checkpoint/cifar10/sgd_randaugsubin_cutmix12 \
--name sgd_randaugsubin_cutmix12 \
--rand_n 1 --rand_m 2

Experiement

1. ReXNet-tiny-cifar with optimizer

Model	Input Res.	Optimizer	Top-1 acc.	Params
ReXNet-tiny-cifar	32x32	Adam	93.82	1.9M
ReXNet-tiny-cifar	32x32	AdamW	93.82	1.9M
ReXNet-tiny-cifar	32x32	AdamP	TODO	1.9M
ReXNet-tiny-cifar	32x32	SGD	94.90	1.9M
ReXNet-tiny-cifar	32x32	SGDP	94.43	1.9M
ReXNet-tiny-cifar	32x32	SGDMix	95.04	1.9M

2. Scheduler

Model	Input Res.	Optimizer	Top-1 acc.	Params
ReXNet-tiny-cifar, step 30	32x32	SGDMix	95.04	1.9M
ReXNet-tiny-cifar, multistep 30	32x32	SGDMix	95.13	1.9M
ReXNet-tiny-cifar, cosine 30	32x32	SGDMix	95.02	1.9M

Model	Input Res.	Optimizer	Top-1 acc.	Params
ReXNet-tiny-cifar, Multi step [30, 60, 90]	32x32	SGDMix	TODO	1.9M

3. Augmentation

Model	Input Res.	Optimizer	Top-1 acc.	Params
ReXNet-tiny-cifar	32x32	Adam	93.82	1.9M
ReXNet-tiny-cifar + cutmix	32x32	Adam	94.97	1.9M
ReXNet-tiny-cifar + cutmix	32x32	SGDMix	TODO	1.9M
ReXNet-tiny-cifar + randaug (1, 2)	32x32	Adam	94.	1.9M
ReXNet-tiny-cifar + randaug (1, 2) + cutmix	32x32	Adam	95.02	1.9M
ReXNet-tiny-cifar + randaug (1, 2) + cutmix	32x32	SGDMix	TODO	1.9M

4. Comparision with efficientnet-cifar

Model	Input Res.	Optimizer	Top-1 acc.	Params
Efficientnet-tiny-cifar	32x32	Adam	93.18
ReXNet-tiny-cifar	32x32	Adam	93.82	1.9M

5. Model Compression (TODO)

Author

• Subin Yang