cifar10_densenet.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "https://arxiv.org/abs/1608.06993"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.conv_learner import *\n",
    "from fastai.plots import *\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last run on: 2018-03-29\n",
      "PyTorch version: 0.3.1.post2\n",
      "fastai version: 0.6\n"
     ]
    }
   ],
   "source": [
    "print_info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH = 'data/'\n",
    "get_cifar10(PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats = (np.array([ 0.4914 ,  0.48216,  0.44653]), np.array([ 0.24703,  0.24349,  0.26159]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_sample(os.path.join(PATH, 'cifar10/train'), 0.05)\n",
    "create_sample(os.path.join(PATH, 'cifar10/test'), 0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Authors trained on entire train set so we will try to mimic their approach. I imagine they didn't use the test set to monitor progress but as I am trying to reimplement their architecture using the hyperparams they provide I think it is fine for me to do so. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz, bs, sample=False):\n",
    "    tfms = tfms_from_stats(stats, sz, aug_tfms=[RandomFlip()], pad=sz//8, pad_mode=cv2.BORDER_CONSTANT)\n",
    "    return ImageClassifierData.from_paths(\n",
    "        f'{PATH}cifar10/',\n",
    "        trn_name='train' if not sample else 'train_sample',\n",
    "        val_name='test' if not sample else 'test_sample',\n",
    "        tfms=tfms,\n",
    "        bs=bs,\n",
    "        num_workers=12\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64\n",
    "sample = False\n",
    "sz = 32"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_data(sz, bs, sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A nice reference implementation in PyTorch by the authors can be found here: https://github.com/gpleiss/efficient_densenet_pytorch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Small DenseNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the code / information I found, I was unable to infer what the architecture of a DenseNet with 40 layers and a growth rate of 12 is. I am guessing it consists of 3 blocks each with 12 layers but I might be wrong.\n",
    "\n",
    "Nonetheless, the plan is to implement something resembling a DenseNet, train it, and see if the results I am getting are in the correct ballpark."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_param_count(m):\n",
    "    return np.sum([o.numel() for o in m.parameters()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseLayer(nn.Sequential):\n",
    "    def __init__(self, ni, gr):\n",
    "        ''' gr: growth rate '''\n",
    "        super().__init__()\n",
    "        self.add_module('bn0', nn.BatchNorm2d(ni))\n",
    "        self.add_module('relu0', nn.ReLU(inplace=True))\n",
    "        # 4 x is arbitrary but that is what one of the implementations I looked at seems to be doing\n",
    "        self.add_module('conv0', nn.Conv2d(ni, 4*gr, 1, padding=0, bias=False)) \n",
    "        self.add_module('bn1', nn.BatchNorm2d(4*gr))\n",
    "        self.add_module('relu1', nn.ReLU(inplace=True))\n",
    "        self.add_module('conv1', nn.Conv2d(4*gr, gr, 3, padding=1, bias=False))\n",
    "        \n",
    "    def forward(self, x):\n",
    "        new_f = super().forward(x)\n",
    "        return torch.cat([x, new_f], 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseBlock(nn.Sequential):\n",
    "    def __init__(self, ni, gr, nl):\n",
    "        super().__init__()\n",
    "        for i in range(nl):\n",
    "            self.add_module(f'dense{i}', DenseLayer(ni + gr * i, gr))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Transition(nn.Sequential):\n",
    "    def __init__(self, ni, comp=0.5):\n",
    "        super().__init__()\n",
    "        self.add_module('bn', nn.BatchNorm2d(ni))\n",
    "        self.add_module('relu', nn.ReLU(inplace=True))\n",
    "        self.add_module('conv', nn.Conv2d(ni, int(ni*comp), 1, bias=False))\n",
    "        self.add_module('avg_pool', nn.AvgPool2d(2, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ni_per_block(i, no, nl, gr, comp=0.5):\n",
    "    ni = no\n",
    "    for i in range(i):\n",
    "        ni += nl * gr\n",
    "        ni = int(comp * ni)\n",
    "    return ni"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DenseNet40_12(nn.Module):\n",
    "    def __init__(self, c):\n",
    "        super().__init__()\n",
    "        no = 16 # count of output features from the initial convolution\n",
    "        nb = 3 # count of blocks\n",
    "        nl = 12 # count of layers per block\n",
    "        gr = 12 # growth rate, amount of features output by each layer\n",
    "        comp=0.5\n",
    "        \n",
    "        self.conv0 = nn.Conv2d(3, no, 3, padding=1, bias=False)\n",
    "        \n",
    "        self.conv_blocks = nn.ModuleList()\n",
    "        for i in range(nb):\n",
    "            self.conv_blocks.add_module(f'block_{i}', DenseBlock(ni_per_block(i, no, nl, gr, comp), gr, nl))\n",
    "            \n",
    "        \n",
    "        self.trans = nn.ModuleList([\n",
    "            Transition(ni_per_block(i, no, nl, gr, comp) + nl * gr) for i in range(nb-1)\n",
    "        ])\n",
    "        \n",
    "        n_f_final = ni_per_block(nb-1, no, nl, gr, comp) + nl * gr\n",
    "        self.bn_final = nn.BatchNorm2d(n_f_final)\n",
    "        self.classifier = nn.Linear(n_f_final, c)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.conv0(x)\n",
    "        \n",
    "        for i, b in enumerate(self.conv_blocks):\n",
    "            if i is not 0: x = self.trans[i-1](x)\n",
    "            x = b(x)\n",
    "            \n",
    "        x = self.bn_final(x)\n",
    "      \n",
    "        x = x.view(x.shape[0], x.shape[1], -1).mean(2)\n",
    "        x = self.classifier(x)\n",
    "        \n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(DenseNet40_12(10), data, opt_fn=SGD_Momentum(0.9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "475850"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_param_count(learn.model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paper mentions that the DenseNet I tried to recreate has 1 million parameters. I was unable to pinpoint in the paper what could be the reason for such a big difference in parameter count."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "wds = 10e-4 # paper mentions this wd but I think this might be incorrect, it might actually be 1e-4\n",
    "            # (it references the training or resnets where 1e-4 was used)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Paper suggests training for 300 epochs, starting with lr 0.1 and decaying it by 10 at 50% and 75% of training. But as I have a smaller model I think I should be okay training with training for a lesser number of epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ae2a638e39174e1f83ed578355190d19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=60), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                   \n",
      "    0      1.311386   1.274939   0.532643  \n",
      "    1      1.037473   1.401766   0.523587                   \n",
      "    2      0.92245    1.369378   0.591162                    \n",
      "    3      0.880907   1.360747   0.581111                    \n",
      "    4      0.804627   1.027813   0.656847                    \n",
      "    5      0.800998   1.177325   0.62918                     \n",
      "    6      0.766326   1.048657   0.648189                    \n",
      "    7      0.752539   1.077507   0.650179                    \n",
      "    8      0.76203    0.913143   0.692178                    \n",
      "    9      0.729287   1.052956   0.65824                     \n",
      "    10     0.772939   1.009882   0.653463                    \n",
      "    11     0.761346   0.918471   0.698846                    \n",
      "    12     0.742009   1.012124   0.660231                    \n",
      "    13     0.759809   1.062543   0.656051                    \n",
      "    14     0.749707   1.032562   0.657245                    \n",
      "    15     0.754506   1.002115   0.676553                    \n",
      "    16     0.717773   0.875453   0.70203                     \n",
      "    17     0.756766   0.990452   0.65625                     \n",
      "    18     0.745545   1.098775   0.649582                    \n",
      "    19     0.738318   1.106669   0.631967                    \n",
      "    20     0.733782   2.458608   0.487759                    \n",
      "    21     0.72059    1.248363   0.60619                     \n",
      "    22     0.727743   0.812367   0.7291                      \n",
      "    23     0.727803   0.996741   0.676055                    \n",
      "    24     0.711432   1.202433   0.631568                    \n",
      "    25     0.727688   1.140613   0.649084                    \n",
      "    26     0.72357    0.947216   0.689391                    \n",
      "    27     0.716943   0.928464   0.688595                    \n",
      "    28     0.711832   0.781481   0.727408                    \n",
      "    29     0.732292   1.131282   0.621417                    \n",
      "    30     0.70962    1.158594   0.64789                     \n",
      "    31     0.743452   1.043119   0.644009                    \n",
      "    32     0.723189   2.192703   0.44576                     \n",
      "    33     0.716297   0.998239   0.663714                    \n",
      "    34     0.726558   0.896145   0.690983                    \n",
      "    35     0.744842   0.745758   0.74582                     \n",
      "    36     0.701339   0.90998    0.704817                    \n",
      "    37     0.711872   1.299468   0.598527                    \n",
      "    38     0.720599   0.869885   0.703025                    \n",
      "    39     0.70447    1.530598   0.589072                    \n",
      "    40     0.704579   1.136153   0.637142                    \n",
      "    41     0.723872   1.27993    0.587281                    \n",
      "    42     0.696522   1.982901   0.512042                    \n",
      "    43     0.715856   1.176117   0.647094                    \n",
      "    44     0.697483   1.101382   0.639232                    \n",
      "    45     0.733253   1.090172   0.645601                    \n",
      "    46     0.730704   0.934285   0.677846                    \n",
      "    47     0.733017   0.977352   0.676752                    \n",
      "    48     0.746544   1.059559   0.653861                    \n",
      "    49     0.711871   1.512755   0.578125                    \n",
      "    50     0.722568   1.238156   0.597731                    \n",
      "    51     0.733907   0.840433   0.706111                    \n",
      "    52     0.712938   0.899108   0.694168                    \n",
      "    53     0.708582   0.99433    0.676951                    \n",
      "    54     0.722301   1.556477   0.540307                    \n",
      "    55     0.717263   1.049002   0.655951                    \n",
      "    56     0.743509   1.271472   0.606688                    \n",
      "    57     0.735525   1.259266   0.605693                    \n",
      "    58     0.744683   0.81168    0.723826                    \n",
      "    59     0.737412   1.199391   0.603901                    \n",
      "\n",
      "CPU times: user 1h 11min 20s, sys: 12min 27s, total: 1h 23min 47s\n",
      "Wall time: 1h 6min 29s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[1.1993911, 0.6039012738853503]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "learn.fit(1e-1, 60, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "272d7b0d6214419ba73191cfd5daf116",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=30), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.447679   0.461868   0.841063  \n",
      "    1      0.419699   0.423838   0.855494                    \n",
      "    2      0.403035   0.469202   0.839371                    \n",
      "    3      0.374985   0.425033   0.8542                      \n",
      "    4      0.374666   0.438647   0.850318                    \n",
      "    5      0.404271   0.459892   0.846935                    \n",
      "    6      0.384292   0.450538   0.842058                    \n",
      "    7      0.381985   0.456771   0.845442                    \n",
      "    8      0.386762   0.442477   0.846338                    \n",
      "    9      0.372147   0.502554   0.830016                    \n",
      "    10     0.365654   0.580266   0.807325                    \n",
      "    11     0.394753   0.498774   0.833798                    \n",
      "    12     0.395801   0.451188   0.84365                     \n",
      "    13     0.377641   0.507878   0.829319                    \n",
      "    14     0.362975   0.534201   0.82295                     \n",
      "    15     0.359549   0.430242   0.854299                    \n",
      "    16     0.350749   0.470165   0.835589                    \n",
      "    17     0.3751     0.4934     0.833002                    \n",
      "    18     0.362943   0.3978     0.865048                    \n",
      "    19     0.345836   0.454516   0.844745                    \n",
      "    20     0.366416   0.52708    0.829618                    \n",
      "    21     0.352806   0.452744   0.846338                    \n",
      "    22     0.353892   0.435232   0.850717                    \n",
      "    23     0.333811   0.482825   0.832902                    \n",
      "    24     0.33716    0.396809   0.861067                    \n",
      "    25     0.3454     0.467896   0.841461                    \n",
      "    26     0.34068    0.431413   0.853603                    \n",
      "    27     0.366389   0.452121   0.849323                    \n",
      "    28     0.340739   0.452658   0.84783                     \n",
      "    29     0.332731   0.422637   0.856489                    \n",
      "\n",
      "CPU times: user 35min 34s, sys: 6min 13s, total: 41min 48s\n",
      "Wall time: 33min 9s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4226373, 0.8564888535031847]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "learn.fit(1e-2, 30, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83db57076a7d4a528d6240c45bb32bd0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=30), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.224567   0.291461   0.898985  \n",
      "    1      0.217853   0.284613   0.900478                    \n",
      "    2      0.19508    0.280558   0.902568                    \n",
      "    3      0.190859   0.278302   0.901971                    \n",
      "    4      0.181993   0.280493   0.906051                    \n",
      "    5      0.187025   0.279654   0.904359                    \n",
      "    6      0.176685   0.281269   0.904857                    \n",
      "    7      0.189919   0.280698   0.904558                    \n",
      "    8      0.17278    0.282082   0.905852                    \n",
      "    9      0.1589     0.271804   0.908639                    \n",
      "    10     0.157584   0.276442   0.908141                    \n",
      "    11     0.161629   0.277764   0.907942                    \n",
      "    12     0.155887   0.280883   0.906648                    \n",
      "    13     0.148017   0.283008   0.905852                    \n",
      "    14     0.150339   0.287987   0.901971                    \n",
      "    15     0.161001   0.2901     0.902568                    \n",
      "    18     0.14293    0.291549   0.903165                    \n",
      "    19     0.146908   0.286995   0.90635                     \n",
      " 93%|█████████▎| 726/782 [00:58<00:04, 12.51it/s, loss=0.128]"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    22     0.133391   0.289553   0.905752                    \n",
      "    23     0.142564   0.285936   0.906449                    \n",
      "    24     0.130048   0.291292   0.904857                    \n",
      "    25     0.127702   0.293783   0.905255                    \n",
      "    26     0.137578   0.29019    0.907146                    \n",
      "    27     0.141066   0.293704   0.903762                    \n",
      "    28     0.134391   0.300314   0.899881                    \n",
      "    29     0.123537   0.290525   0.90426                     \n",
      "\n",
      "CPU times: user 35min 33s, sys: 6min 13s, total: 41min 47s\n",
      "Wall time: 33min 9s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.29052508, 0.9042595541401274]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "learn.fit(1e-3, 30, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9038"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_np(*learn.predict_with_targs())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                             \r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9111"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_preds,y = learn.TTA(True)\n",
    "preds = np.mean(np.exp(log_preds),0)\n",
    "accuracy_np(preds,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network is much smaller vs the one mentioned in the paper and it might be that there are also some other differences that I have missed.\n",
    "\n",
    "Just to experiment more with the architecture, I will try to train a bigger model. I might also add dropout as we seem to be overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class CustomDenseNet(nn.Module):\n",
    "    def __init__(self, c, no=16, nb=3, nl=12, gr=12, comp=0.5):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.conv0 = nn.Conv2d(3, no, 3, padding=1, bias=False)\n",
    "        \n",
    "        self.conv_blocks = nn.ModuleList()\n",
    "        for i in range(nb):\n",
    "            self.conv_blocks.add_module(f'block_{i}', DenseBlock(ni_per_block(i, no, nl, gr, comp), gr, nl))\n",
    "            \n",
    "        \n",
    "        self.trans = nn.ModuleList([\n",
    "            Transition(ni_per_block(i, no, nl, gr, comp) + nl * gr) for i in range(nb-1)\n",
    "        ])\n",
    "        \n",
    "        n_f_final = ni_per_block(nb-1, no, nl, gr, comp) + nl * gr\n",
    "        self.bn_final = nn.BatchNorm2d(n_f_final)\n",
    "        self.classifier = nn.Linear(n_f_final, c)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        x = self.conv0(x)\n",
    "        \n",
    "        for i, b in enumerate(self.conv_blocks):\n",
    "            if i is not 0: x = self.trans[i-1](x)\n",
    "            x = b(x)\n",
    "            \n",
    "        x = self.bn_final(x)\n",
    "      \n",
    "        x = x.view(x.shape[0], x.shape[1], -1).mean(2)\n",
    "        x = self.classifier(x)\n",
    "        \n",
    "        return F.log_softmax(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(CustomDenseNet(10, 24, 4, 15, 12), data, opt_fn=SGD_Momentum(0.9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1007332"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_param_count(learn.model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "wds=1e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "%%capture\n",
    "learn.fit(1e-1, 150, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "%%capture\n",
    "learn.fit(1e-2, 75, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "%%capture\n",
    "learn.fit(1e-3, 75, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8917"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_np(*learn.predict_with_targs())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('densenet_cifar_fully_trained')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "562e40e327724a6da0b49a3bff0225be",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                    \n",
      "    0      0.163783   0.367921   0.888834  \n",
      "\n",
      "CPU times: user 1min 33s, sys: 9.88 s, total: 1min 43s\n",
      "Wall time: 1min 28s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.36792144, 0.8888335987261147]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "learn.fit(1e-3, 1, wds=wds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds, targs = predict_with_targs(learn.model, learn.data.trn_dl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95372"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy_np(preds, targs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I am without a doubt overfitting here extremely severly. A model I was trying to recreate seems to achieve an error rate of 5.24 with data augmentation (which we do here).\n",
    "\n",
    "Going forward there are a couple of things that could be attempted here:\n",
    "* training with logging the val accuracy over for each epoch and seeing if at any point we are close to the error rate from the paper\n",
    "* checking the model architecture again against the paper (not sure that would get me far though)\n",
    "* adding dropout (despite it being stated in the paper that dropout was not used when training with data augmentation)\n",
    "* create a training schedule that allows for much quicker training using the ideas from the Leslie Smith [paper](https://arxiv.org/abs/1803.09820) (this sounds like a very fun and educational project!)\n",
    "* implement the extreme bottlenecking and depth (100 layers with 0.8 million params!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I reread the paper and now I think that it doesn't do the 1x1 convolutions layers! This might mean that more information is retained and possibly improve results! (going so many times from 200+ channels to 48 sounds a bit crazy). I do not know what this will do to parameter count but I think this is the next thing I should explore if I do continue working on this.\n",
    "\n",
    "Quite interesting though that this is not the architecture as implemented in the PyTorch repo authored by the authors of the paper."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}