Optimizers and tests

Every result is avg of 20 runs.

Dataset	LR Schedule	Imagenette size 128, 5 epoch	Imagewoof size 128, 5 epoch
Adam - baseline	OneCycle	0.8493	0.6125
RangerLars (RAdam + LARS + Lookahead)	Flat and anneal	0.8732	0.6523
Ralamb (RAdam + LARS)	Flat and anneal	0.8675	0.6367
Ranger (RAdam + Lookahead)	Flat and anneal	0.8594	0.5946
Novograd	Flat and anneal	0.8711	0.6126
Radam	Flat and anneal	0.8444	0.537
Lookahead	OneCycle	0.8578	0.6106
Lamb	OneCycle	0.8400	0.5597
DiffGrad	OneCycle	0.8527	0.5912
AdaMod	OneCycle	0.8473	0.6132

Name	Name	Last commit message	Last commit date
Latest commit mgrankin wip Nov 22, 2022 ac21bf9 · Nov 22, 2022 History 52 Commits
.github	.github	Create stale.yml	Feb 2, 2020
.gitignore	.gitignore	fastai -> lighning, apollo optimizer	Oct 12, 2020
Imagenette_128_5_epoch-bs64.ipynb	Imagenette_128_5_epoch-bs64.ipynb	wip	Nov 22, 2022
Imagewoof_128_5_epoch-bs64.ipynb	Imagewoof_128_5_epoch-bs64.ipynb	adan	Sep 2, 2022
LICENSE	LICENSE	Create LICENSE	Jan 19, 2020
README.md	README.md	AdaMod and DiffGrad	Jan 3, 2020
adabelief.py	adabelief.py	adabelief defaults	Oct 28, 2020
adamod.py	adamod.py	AdaMod and DiffGrad	Jan 3, 2020
adan.py	adan.py	adan	Sep 2, 2022
apollo.py	apollo.py	fastai -> lighning, apollo optimizer	Oct 12, 2020
diffgrad.py	diffgrad.py	AdaMod and DiffGrad	Jan 3, 2020
draft.ipynb	draft.ipynb	fastai -> lighning, apollo optimizer	Oct 12, 2020
environment.yml	environment.yml	wip	Nov 22, 2022
lamb.py	lamb.py	lamb optimizer, renaming	Aug 28, 2019
lookahead.py	lookahead.py	dataset v2	Jul 15, 2020
madam.py	madam.py	Madam optimiser https://github.com/jxbz/madam	Jul 13, 2020
madgrad.py	madgrad.py	madgrad and lighning	Apr 3, 2021
novograd.py	novograd.py	initial	Aug 25, 2019
radam.py	radam.py	Fix bug in RAdams buffer implementation	Sep 2, 2019
ralamb.py	ralamb.py	Fix bug in RAdams buffer implementation	Sep 2, 2019
ranger.py	ranger.py	start refactoring and renaming	Aug 27, 2019
rangerlars.py	rangerlars.py	fastai -> lighning, apollo optimizer	Oct 12, 2020
train.py	train.py	wip	Nov 22, 2022
xresnet.py	xresnet.py	xresnet revert to default	Aug 25, 2019

Provide feedback