assignment2/Dropout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "def rel_error(x, y):\n",
    "  \"\"\" returns relative error \"\"\"\n",
    "  return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "  print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean())\n",
    "  print('Mean of train-time output: ', out.mean())\n",
    "  print('Mean of test-time output: ', out_test.mean())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n",
    "预测值就会比训练值大很多"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. After doing so, run the following to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 5.37e-09\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss)\n",
    "  \n",
    "  # Relative errors should be around e-6 or less; Note that it's fine\n",
    "  # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. We will then visualize the training and validation accuracies of the two networks over time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.708000; val_acc: 0.299000\n",
      "(Epoch 6 / 25) train acc: 0.722000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.878000; val_acc: 0.269000\n",
      "(Epoch 9 / 25) train acc: 0.902000; val_acc: 0.275000\n",
      "(Epoch 10 / 25) train acc: 0.888000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.926000; val_acc: 0.278000\n",
      "(Epoch 12 / 25) train acc: 0.960000; val_acc: 0.302000\n",
      "(Epoch 13 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 14 / 25) train acc: 0.966000; val_acc: 0.309000\n",
      "(Epoch 15 / 25) train acc: 0.976000; val_acc: 0.288000\n",
      "(Epoch 16 / 25) train acc: 0.988000; val_acc: 0.301000\n",
      "(Epoch 17 / 25) train acc: 0.988000; val_acc: 0.305000\n",
      "(Epoch 18 / 25) train acc: 0.990000; val_acc: 0.308000\n",
      "(Epoch 19 / 25) train acc: 0.988000; val_acc: 0.311000\n",
      "(Epoch 20 / 25) train acc: 0.990000; val_acc: 0.311000\n",
      "(Iteration 101 / 125) loss: 0.006070\n",
      "(Epoch 21 / 25) train acc: 0.998000; val_acc: 0.313000\n",
      "(Epoch 22 / 25) train acc: 0.976000; val_acc: 0.322000\n",
      "(Epoch 23 / 25) train acc: 0.986000; val_acc: 0.314000\n",
      "(Epoch 24 / 25) train acc: 0.990000; val_acc: 0.310000\n",
      "(Epoch 25 / 25) train acc: 0.994000; val_acc: 0.305000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318478\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.402000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.502000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.528000; val_acc: 0.298000\n",
      "(Epoch 5 / 25) train acc: 0.562000; val_acc: 0.297000\n",
      "(Epoch 6 / 25) train acc: 0.626000; val_acc: 0.290000\n",
      "(Epoch 7 / 25) train acc: 0.628000; val_acc: 0.298000\n",
      "(Epoch 8 / 25) train acc: 0.686000; val_acc: 0.310000\n",
      "(Epoch 9 / 25) train acc: 0.722000; val_acc: 0.289000\n",
      "(Epoch 10 / 25) train acc: 0.724000; val_acc: 0.300000\n",
      "(Epoch 11 / 25) train acc: 0.760000; val_acc: 0.305000\n",
      "(Epoch 12 / 25) train acc: 0.772000; val_acc: 0.280000\n",
      "(Epoch 13 / 25) train acc: 0.814000; val_acc: 0.303000\n",
      "(Epoch 14 / 25) train acc: 0.814000; val_acc: 0.341000\n",
      "(Epoch 15 / 25) train acc: 0.856000; val_acc: 0.352000\n",
      "(Epoch 16 / 25) train acc: 0.838000; val_acc: 0.303000\n",
      "(Epoch 17 / 25) train acc: 0.840000; val_acc: 0.291000\n",
      "(Epoch 18 / 25) train acc: 0.844000; val_acc: 0.315000\n",
      "(Epoch 19 / 25) train acc: 0.864000; val_acc: 0.325000\n",
      "(Epoch 20 / 25) train acc: 0.862000; val_acc: 0.308000\n",
      "(Iteration 101 / 125) loss: 5.259476\n",
      "(Epoch 21 / 25) train acc: 0.896000; val_acc: 0.320000\n",
      "(Epoch 22 / 25) train acc: 0.872000; val_acc: 0.298000\n",
      "(Epoch 23 / 25) train acc: 0.906000; val_acc: 0.317000\n",
      "(Epoch 24 / 25) train acc: 0.906000; val_acc: 0.320000\n",
      "(Epoch 25 / 25) train acc: 0.910000; val_acc: 0.325000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "  'X_train': data['X_train'][:num_train],\n",
    "  'y_train': data['y_train'][:num_train],\n",
    "  'X_val': data['X_val'],\n",
    "  'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([500], dropout=dropout)\n",
    "  print(dropout)\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=25, batch_size=100,\n",
    "                  update_rule='adam',\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-4,\n",
    "                  },\n",
    "                  verbose=True, print_every=100)\n",
    "  solver.train()\n",
    "  solvers[dropout] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4gAAAJNCAYAAABz4lt9AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XucXXV97//XxzDIqJjhEhEmgQRJAzQJBIdgoVwkxaAIBLARVCy2ilgQT3tOKpxfmxNzaqWGir8caD1pRcALkNIYg4JR8Va0IhODiYCRgGBmQA2XRNABhvA5f+w1YWecmWSS2XvtmXk9H4957FnftfZan83sDPs938uKzESSJEmSpJeVXYAkSZIkqTEYECVJkiRJgAFRkiRJklQwIEqSJEmSAAOiJEmSJKlgQJQkSZIkAQZESZKIiDER8UxEHFh2LZIklSm8D6IkabiJiGeqNl8BPAdsKbbfn5mfr39VkiQNfwZESdKwFhEPA+/NzG8McMxumflC/aqqn5H82iRJ9ecQU0nSiBMRfx8RN0fEjRHxNPCuiPijiPhBRGyKiMciYnFENBXH7xYRGRETi+3PFftvj4inI+K/ImJSP9d6WUTcEhG/LM797Yg4rGr/KyLiqoj4RURsjojvRsTLi30nFDVtjogNEXF+0X5nRFxQdY73RsS3e9X6lxGxHvhp0X51RHRExG8i4u6IOLbq+btFxN9FxIPF/vaIOCAi/m9E/GOv13NbRHxw138KkqThyIAoSRqpzgK+AIwFbgZeAD4E7AscB5wKvH+A578D+Dtgb+AXwP8e4NgvA5OB1wI/AT5bte8qYDpwTHGu/wm8WATO24BPAPsAM4C1g3h9ZwBHA9OK7buK6+wN3AL8e08QBeYBb6PymluA9wLPAtcD50VEAETEfsBJwI2DqEOSNIIYECVJI9WdmXlrZr6YmV2ZeXdm3pWZL2TmQ8AS4MQBnn9LZrZnZjfweeDIvg4qzn9dZj6dmc8CC4DXR8QrI2IMcAFwaWY+lplbMvPO4pzvAm7PzKVFTY9n5j2DeH3/kJlPZWZXUcdnM/PJYrjpx4FXA4cUx74X+J+Z+UBR7z3Fsd+nEhRPKo47D/hGZj4+iDokSSOIAVGSNFJtqN6IiEMj4ivFUNDfAAup9Cb255dV3/8OeFVfBxUroH48Ih4qzru+2LUvsB+wO/BgH0+d0E/7jur9+v4mIn4aEZuBp4BX8tLrG+haN1AJqxSPn+3nOEnSKGBAlCSNVL1XYfu/VIZ/HpKZrwbmAzEE13k38BbgZCrDWXt67QL4FfA88Lo+nrehn3aA31JZnbXHa/s4Zuvri4g3An8NnENlCOlewDO89PoGutZngbMiYkZxzK39HCdJGgUMiJKk0WJPYDPw22IRmYHmHw72vM8BT1AJdR/t2ZGZW4DrgE9GxGuL3sbjisVxPgecGhHnFIvI7BsRRxRPvQc4JyKaI+IPgD/fgRpeAB4HmqgMc31l1f5/A/4+Il4XFUdGxN5FjY8AP6YyH/Hfi2GykqRRyoAoSRot/jvwZ8DTVHoTbx6i834GeLT4uhf4fq/9fwXcD6wCngT+gcptpn4OnA58uGj/ES8tOHMllR7CXwPXUgmTA7kN+AbwAPAw8Bvgsar9i4DlwB3FviXAHlX7ry+u7fBSSRrlvA+iJEmjXEScDHwaODj9YCBJo5o9iJIkjWIRsTuV23/8q+FQkmRAlCRplIqIaVRWPN0bWFxyOZKkBuAQU0mSJEkSYA+iJEmSJKlgQJQkSZIkAbBb2QXUw7777psTJ04suwxJkiRJKsWqVasez8xx2ztuVATEiRMn0t7eXnYZkiRJklSKiHhkR45ziKkkSZIkCTAgSpIkSZIKBkRJkiRJEmBAlCRJkiQVDIiSJEmSJKDBAmJEXBsRv46In/SzPyJicUSsj4g1EXFUvWuUJEmSpJGq0W5zcR1wNXBDP/vfDEwuvo4B/qV4lCRJkhrS8tWdLFq5jkc3dXFASzPzZk9hzozWssuS+tRQATEzvxsREwc45EzghsxM4AcR0RIR+2fmY3UpUJIkSRqE5as7uXzZWrq6twDQuamLy5etBTAkjjAj5Q8BDRUQd0ArsKFqu6NoMyBKkiQNwkj5MDsYZbzmRSvXbQ2HPbq6t7Bo5boR/9+7DGW9r0fSHwKGW0DcYRFxIXAhwIEHHlhyNZIkqZGV+aFytH2YHW2v+dFNXYNqHynK+DmX+b4eSX8IaKhFanZAJzChant80fZ7MnNJZrZlZtu4cePqUpwkSRp+ej5Udm7qInnpQ+Xy1X1+xBj214WBP8zW0mh8zQe0NA+qfSQo6+dc1s8YRtYfAoZbQFwBvLtYzfQNwGbnH0qSpF1R1ofK0fhhdjS+5nmzp9DcNGabtuamMcybPaWm1y1TWT/nMkPaSPpDQEMNMY2IG4GTgH0jogP4X0ATQGZ+CrgNeAuwHvgd8J5yKpUkSUOtrKGHZX2oLPvDbGcf16n1h9nR+Jp73sOjaQhzWT/nsn7GUPlDQPXwVhi+fwhoqICYmedtZ38CF9epHEmSVCdlzh0q60PlaPwwOxpfM1Tew/WehzYa/02V/TOGcv4QMNSG2xBTSZI0ApU59LCsIYBlDj2cM6OVj509jdaWZgJobWnmY2dPq/mH2dH4mssyGv9Nlf0znjOjle9ddjI/v+I0vnfZycP2vdVQPYiSJKl8ZQxLK3PoYVl/+S+7x6GMXq3R+JrLMhr/TfVce7T8jGvFgChJUoMaTcvElzn0EMr7UDkaP8yOxtdchtH6b0q7ziGmkiQNYPnqTo674ptMuuwrHHfFN+uyHH/PdUfTMvGjcaVHqZb8N6WdZUCUJKkfo/GebWUNSyt77pA00vhvSjvLIaaSJPVjoJA2UufklTkszSFp0tDy35R2hj2IkiT1o+x7tg2mfag4LE2SRjcDoiRJ/SgrpMHoXSZeklQuh5hKktSP0XrTZYelSdLoZUCUJKkf3rNNkjTaGBAlSRqAIU2SNJo4B1GSJEmSBBgQJUmSJEkFh5hKkoaF5as7S5sLKEnSaGFAlCQ1vOWrO7dZTbRzUxeXL1sLYEiUJGkIOcRUktTwFq1ct82tJgC6urewaOW6kiqSJGlkMiBKkhreo5u6BtUuSZJ2jgFRktTwDmhpHlS7JEl1t2YpXDUVFrRUHtcsLbuinWJAlCQ1vHmzp9DcNGabtuamMcybPaWkiiRJAyorLJV53Vsvhc0bgKw83nrpsAyJDRcQI+LUiFgXEesj4rI+9h8UEXdExJqI+HZEjC+jTkkq0/LVnRx3xTeZdNlXOO6Kb7J8dWfZJdXUnBmtfOzsabS2NBNAa0szHzt7mgvUSFIjKisslRnS7lgI3b2mPXR3VdqHmcjMsmvYKiLGAD8DTgE6gLuB8zLzvqpj/h34cmZeHxEnA+/JzPMHOm9bW1u2t7fXsHJJqp/eK3pCpTfNwCRJaghXTS1CWi9jJ8Bf/WTkXRcqPZb0lasCFmyq7bV3UESsysy27R3XaD2IM4H1mflQZj4P3ASc2euYw4FvFt9/q4/9kjSiuaKnJA0zI2Ru2g7b3DG49uF+XYCx/Qxq7K+9gTVaQGwFqmN/R9FW7cfA2cX3ZwF7RsQ+dahNkhqCK3pK0jAyguam7bCywlKZIW3WfGjqtXBaU3OlfZhptIC4I/4HcGJErAZOBDqBLb0PiogLI6I9Ito3btxY7xolqWbKXNFztM19lKRdNoLmpu2wssJSmSFt+lw4fXFlOCtReTx9caV9mNmt7AJ66QQmVG2PL9q2ysxHKXoQI+JVwDmZ+XsDezNzCbAEKnMQa1WwJNXbvNlT+pyDWOsVPXvPfezc1MXly9YCOPdRkvpT5rDHsvSEojsWVl7n2PGVkFbrsFTWdauvPwwDYW+NFhDvBiZHxCQqwfBc4B3VB0TEvsCTmfkicDlwbd2rlKQS9YSxRSvX8eimLg5oaWbe7Ck1D2kDzX00IEpSP8aO72fhlOE3N21QygpLIySklamhAmJmvhARlwArgTHAtZl5b0QsBNozcwVwEvCxiEjgu8DFpRUsaVRbvrqz7iGtx5wZrXUPZc59lKSdMGt+Zc5h9TDTYTo3TaNDQwVEgMy8DbitV9v8qu9vAW6pd12SVG00Drc8oKWZzj7CYD3mPkrSsFX2sEdpkBouIErScDAah1uWNfdRkoa9soY9rllqMNWgGRAlaSeMxuGWZc19lCTthJ7ba/QMbe25vQYYEjUgA6Ik7YTROtyyjLmPkkYYe7XqY6Dba/jfWwMYjvdBlKTSzZs9heamMdu0OdxSkraj7JvGr1kKV02FBS2Vx5F8s/rReHsNDQkDoiTthDkzWvnY2dNobWkmgNaWZj529jR716RdMZo+vI9WZd40vuxwWm/93UZjpN9eQ7vMIaaStJMcbikNIedLjQ5l9mqNtiGX3l5DO8keREmSVL4ye5ZUP2X2ao22IZfT58Lpi2HsBCAqj6cvHplhWEPKHkRJklS+0fbhfbQqs1dr7PhieGkf7SNVWbfX0LBmD6IkSSqf86Xqr4w5n2X2as2aXwmj1RxyKf0eexAlSVL5Rut8qbJu+VDmnM+yerV6ruktNqQBRWaWXUPNtbW1ZXt7e9llSJKkgZR5f7wyrt07pEElFNejR+2qqf0Mt5wAf/WT2l5bUikiYlVmtm3vOHsQJQ17y1d3smjlOh7d1MUBLc3Mmz3F1UWl4aisnqWyetPKXFXTOZ+S+uEcREnD2vLVnVy+bC2dm7pIoHNTF5cvW8vy1Z1llyZpuChrBdUyQ5pzPiX1w4AoaVhbtHIdXd1btmnr6t7CopXrSqpI0rBTVlArM6S5YIukfhgQJQ1rj27qGlS7JP2esoJamSHNe+RJ6odzECUNawe0NNPZRxg8oKW5j6MlqQ9lraBa9qqa3iNPUh8MiJKGtXmzp3D5srXbDDNtbhrDvNlTSqxKGiJl3gJhNN0KoMygZkiT1GAMiJKGtZ7VSl3FVCNOWStrlnl/vDIZ1CQJ8D6IkiQ1prLuU+f98SRpRNrR+yA23CI1EXFqRKyLiPURcVkf+w+MiG9FxOqIWBMRbymjTkmSaqqslTW9P54kjWoNFRAjYgxwDfBm4HDgvIg4vNdhfwsszcwZwLnAP9e3SkmS6qCslTW9P54kjWoNFRCBmcD6zHwoM58HbgLO7HVMAq8uvh8LPFrH+iRJqo+yboHg/fEkaVRrtIDYClRPfOgo2qotAN4VER3AbcAH61OaJEl1VNZ96rw/niSNasNxFdPzgOsy858i4o+Az0bE1Mx8sfqgiLgQuBDgwAMPLKFMSZJ2UVkra7qipySNWo3Wg9gJTKjaHl+0VfsLYClAZv4XsAewb+8TZeaSzGzLzLZx48bVqFxJkiRJGjkaLSDeDUyOiEkRsTuVRWhW9DrmF8AsgIg4jEpA3FjXKiVJkiRpBGqogJiZLwCXACuB+6msVnpvRCyMiDOKw/478L6I+DFwI3BBjoabOUqSJElSjTXcHMTMvI3K4jPVbfOrvr8POK7edUmSJEnSSNdQPYiSJDWcNUvhqqmwoKXyuGZp2RVJklQzDdeDKElSw1izFG69FLq7KtubN1S2wVU+JUkjkj2IkiT1546FL4XDHt1dlXZJkkYgA6IkSf3Z3DG4dkmShjkDoiRpeChjLuDY8YNrlyRpmDMgSpIaX89cwM0bgHxpLmCtQ+Ks+dDUvG1bU3OlXZKkEciAKElqfGXNBZw+F05fDGMnAFF5PH2xC9RIkkYsVzGVJDW+MucCTp9rIJQkjRr2IEqSGp9zASVJqgsDoiSp8TkXUJKkujAgSpIan3MBJUmqC+cgSpKGB+cCSpJUc/YgSpIkSZIAA6IkSZIkqeAQU0lDYvnqThatXMejm7o4oKWZebOnMGdGa9lljVxrllbuAbi5o7KS56z59Rl+WdZ1JUlSXRgQJe2y5as7uXzZWrq6twDQuamLy5etBTAk1sKapXDrpS/dOH7zhso21DaslXVdSZJUNw4xlbTLFq1ctzUc9ujq3sKiletKqmiEu2PhSyGtR3dXpX0kXleSJNWNAVHSLnt0U9eg2rWLNncMrn24X1eSJNWNQ0ylEaaMuYAHtDTT2UcYPKCluY+jtcvGjq8M7+yrfSReV5Ik1Y09iNII0jMXsHNTF8lLcwGXr+6s6XXnzZ5Cc9OYbdqam8Ywb/aUml531Jo1H5p6he+m5kr7SLyuJEmqm4YLiBFxakSsi4j1EXFZH/uvioh7iq+fRcSmMuqUGlFZcwHnzGjlY2dPo7WlmQBaW5r52NnTXKCmVqbPhdMXw9gJQFQeT19c+4ViyrquJEmqm8jMsmvYKiLGAD8DTgE6gLuB8zLzvn6O/yAwIzP/fKDztrW1ZXt7+1CXKzWcSZd9hb7+RQfw8ytOq3c5kiRJahARsSoz27Z3XKP1IM4E1mfmQ5n5PHATcOYAx58H3FiXyqRhoL85f84FlCRJ0o5otIDYClSvgNBRtP2eiDgImAR8sw51ScOCcwElSZK0K4bzKqbnArdk5pa+dkbEhcCFAAceeGA965JK0zPnr96rmEqSJGlkaLSA2AlMqNoeX7T15Vzg4v5OlJlLgCVQmYM4VAVKjW7OjFYDYb2sWVq5SfzmjsqtHmbNd8EWSZI0rDVaQLwbmBwRk6gEw3OBd/Q+KCIOBfYC/qu+5UlSYc1SuPVS6C7u/7h5Q2UbDImSJGnYqskcxIj4YETsNdjnZeYLwCXASuB+YGlm3hsRCyPijKpDzwVuykZaglXS6HLHwpfCYY/urkq7JEnSMFWrHsT9gLsj4kfAtcDKHQ1zmXkbcFuvtvm9thcMUZ2StHM2dwyuXZIkaRioSQ9iZv4tMBn4NHAB8EBE/ENEvK4W15Okuhs7fnDtkiRJw0DNbnNR9Bj+svh6gcqcwVsi4uO1uqYk1c2s+dDU6/6STc2VdkmSpGGqJkNMI+JDwLuBx4F/A+ZlZndEvAx4APibWlxXkuqmZyEaVzGVJEkjSK3mIO4NnJ2Zj1Q3ZuaLEfHWGl1Tkupr+lwDoSRJGlFqNcT0duDJno2IeHVEHAOQmffX6JqSJEmSpF1Qq4D4L8AzVdvPFG2SJEmSpAZVq4AY1be1yMwXqd1wVkmSJEnSEKhVQHwoIi6NiKbi60PAQzW6liRJkiRpCNQqIF4EHAt0Ah3AMcCFNbqWJEmSJGkI1GTYZ2b+Gji3FueWJEmSJNVGre6DuAfwF8AfAnv0tGfmn9fiepIkSZKkXVerIaafBV4LzAa+A4wHnq7RtSRJkiRJQ6BWAfGQzPw74LeZeT1wGpV5iJI09NYshaumwoKWyuOapWVXJEmSNCzV6tYT3cXjpoiYCvwSeE2NriWpEaxZCncshM0dMHY8zJoP0+fW57q3XgrdXZXtzRsq21Cf60uSJI0gtepBXBIRewF/C6wA7gP+sUbXklS2npC2eQOQL4W0evTk3bHwpXDYo7ur0i5JkqRBGfIexIh4GfCbzHwK+C5w8FBfQ1KDGSik1boXb3PH4NolSZLUryHvQczMF4G/GerzSmpgZYa0seMH1y5JkqR+1WqI6Tci4n9ExISI2Lvnq0bXklS2MkParPnQ1LxtW1NzpV2SJEmDUquA+HbgYipDTFcVX+01upakspUZ0qbPhdMXw9gJQFQeT1/sAjWSJEk7oSarmGbmpJ19bkScCvz/wBjg3zLzij6OmQssABL4cWa+Y2evJ2kI9ISxMlYx7bm+gVCSJGmX1SQgRsS7+2rPzBu287wxwDXAKUAHcHdErMjM+6qOmQxcDhyXmU9FhLfPUENavrqTRSvX8eimLg5oaWbe7CnMmdFadlm1Y0iTJEka9mp1H8Sjq77fA5gF/AgYMCACM4H1mfkQQETcBJxJ5TYZPd4HXFOskkpm/nqoipaGyvLVnVy+bC1d3VsA6NzUxeXL1gKM7JAoSZKkYa1WQ0w/WL0dES3ATTvw1FZgQ9V2B3BMr2P+oDjn96gMQ12QmV/d+Wqlobdo5bqt4bBHV/cWFq1cZ0CUJElSw6pVD2JvvwV2el5iL7sBk4GTgPHAdyNiWmZuqj4oIi4ELgQ48MADh+jS0o55dFPXoNolSZKkRlCrOYi3UllABiorpR4OLN2Bp3YCE6q2xxdt1TqAuzKzG/h5RPyMSmC8u/qgzFwCLAFoa2tLpDo6oKWZzj7C4AEtzX0cLUmSJDWGWvUgXln1/QvAI5m5I3fMvhuYHBGTqATDc4HeK5QuB84DPhMR+1IZcvrQrpcsDZ15s6dsMwcRoLlpDPNmTymxKkmSJGlgtQqIvwAey8xnASKiOSImZubDAz0pM1+IiEuAlVTmF16bmfdGxEKgPTNXFPveFBH3AVuAeZn5RI1eh7RTeuYZjqpVTCVJkjTsRebQj76MiHbg2Mx8vtjeHfheZh498DNro62tLdvb28u4tCRJkiSVLiJWZWbb9o57WY2uv1tPOAQovt+9RteSJEmSJA2BWgXEjRFxRs9GRJwJPF6ja0mSJEmShkCt5iBeBHw+Iq4utjuAd9foWpIkSZKkIVCTgJiZDwJviIhXFdvP1OI6kvqwZincsRA2d8DY8TBrPkyfW3ZVkiRJGgZqMsQ0Iv4hIloy85nMfCYi9oqIv6/FtSRVWbMUbr0UNm8AsvJ466WVdkmSJGk7ajUH8c2ZualnIzOfAt5So2tJ6nHHQuju2ratu6vSLkmSJG1HrQLimIh4ec9GRDQDLx/geElDYXPH4NolSZKkKrVapObzwB0R8RkggAuA62t0LUk9xo4vhpf20S5JkiRtR016EDPzH4G/Bw4DpgArgYNqcS1JVWbNh6bmbduamivtkiRJ0nbUaogpwK+ABP4UOBm4v4bXkgSV1UpPXwxjJwBReTx9sauYSpIkaYcM6RDTiPgD4Lzi63HgZiAy841DeR1JA5g+10AoSZKknTLUcxB/Cvwn8NbMXA8QEX81xNeQdtjy1Z0sWrmORzd1cUBLM/NmT2HOjNayy5IkSZIa0lAPMT0beAz4VkT8a0TMorJIjVR3y1d3cvmytXRu6iKBzk1dXL5sLctXd5ZdmiRJktSQhjQgZubyzDwXOBT4FvDfgNdExL9ExJuG8lrS9ixauY6u7i3btHV1b2HRynX1KWDNUrhqKixoqTx6s3pJkiQ1uFqtYvrbzPxCZp4OjAdWAx+uxbWk/jy6qWtQ7UNqzVK49dLilhNZebz1UkOiJEmSGlotVzEFIDOfyswlmTmr1teSqh3Q0jyo9iF1x0Lo7hVEu7sq7ZIkSVKDqnlAlMoyb/YUmpvGbNPW3DSGebOn1P7imzsG1y5JkiQ1gKFexVRqGD2rlZayiunY8cXw0j7aJUmSpAZlQNSINmdGazm3tZg1vzLnsHqYaVNzpV2SJElqUA03xDQiTo2IdRGxPiIu62P/BRGxMSLuKb7eW0ad0oCmz4XTF8PYCUBUHk9f7A3sJUmS1NAaqgcxIsYA1wCnAB3A3RGxIjPv63XozZl5Sd0L1PCzZmllYZjNHZXhnbPm1y+kTZ9rIJQkSdKw0mg9iDOB9Zn5UGY+D9wEnFlyTRquvNWEJEmSNCiNFhBbgeqVPTqKtt7OiYg1EXFLREyoT2kadrzVhCRJkjQojRYQd8StwMTMnA58Hbi+r4Mi4sKIaI+I9o0bN9a1QDUIbzUhSZIkDUqjBcROoLpHcHzRtlVmPpGZzxWb/wa8vq8TZeaSzGzLzLZx48bVpFg1uP5uKeGtJiRJkqQ+NVpAvBuYHBGTImJ34FxgRfUBEbF/1eYZwP11rE/Dyaz5lVtLVPNWE5IkSVK/GmoV08x8ISIuAVYCY4BrM/PeiFgItGfmCuDSiDgDeAF4ErigtIK1w5av7qz/Det7VhAtaxVTSZIkaZiJzCy7hppra2vL9vb2sssYtZav7uTyZWvp6t6yta25aQwfO3taOTexlyRJkkaZiFiVmW3bO67RhphqBFq0ct024RCgq3sLi1auK6kiSZIkSX0xIKrmHt3UNah2SZIkSeUwIKrmDmhpHlS7JEmSpHIYEFVz82ZPoblpzDZtzU1jmDd7SkkVSZIkSepLQ61iqpGpZyGauq9iKkmSJGlQDIiqizkzWg2EkiRJUoNziKkkSZIkCTAgSpIkSZIKBkRJkiRJEmBAlCRJkiQVDIiSJEmSJMCAKEmSJEkqGBAlSZIkSYABUZIkSZJUMCBKkiRJkgADoiRJkiSpsFvZBah+lq/uZNHKdTy6qYsDWpqZN3sKc2a0ll2WJEmSpAZhQBwllq/u5PJla+nq3gJA56YuLl+2FsCQKEmSJAlwiOmosWjluq3hsEdX9xYWrVxXUkWSJEmSGo0BcZR4dFPXoNolSZIkjT4NFxAj4tSIWBcR6yPisgGOOyciMiLa6lnfcHVAS/Og2ofcmqVw1VRY0FJ5XLO0PteVJEmStMMaKiBGxBjgGuDNwOHAeRFxeB/H7Ql8CLirvhUOX/NmT6G5acw2bc1NY5g3e0rtL75mKdx6KWzeAGTl8dZLDYmSJElSg2mogAjMBNZn5kOZ+TxwE3BmH8f9b+AfgWfrWdxwNmdGKx87exqtLc0E0NrSzMfOnlafBWruWAjdvYaydndV2iVJkiQ1jEZbxbQV2FC13QEcU31ARBwFTMjMr0TEvHoWN9zNmdFazoqlmzsG1y5JkiSpFI3WgzigiHgZ8Angv+/AsRdGRHtEtG/cuLH2xal/Y8cPrl2SJElSKRotIHYCE6q2xxdtPfYEpgLfjoiHgTcAK/paqCYzl2RmW2a2jRs3roYla7tmzYemXovhNDVX2iVJkiQ1jEYLiHcDkyNiUkTsDpwLrOjZmZmbM3PfzJyYmROBHwBnZGZ7OeVqh0yfC6cvhrETgKg8nr640i5JkiSpYTTUHMTMfCEiLgFWAmOAazPz3ohYCLRn5oqBz6ABrVlaWRhmc0dleOes+fULadPnGgglSZKkBtdQAREgM28DbuvV1udYxMw8qR41jQg9t5roWU2051YTYHCTJEmSBDTeEFPVireakCRJkrQdBsTRwltNSJIkSdoOA+Jo4a0mJEmSJG2HAXG08FYTkiRJkrbDgDhaeKsJSZIkSdvRcKuYqoa81YQkSZKkAdiDKEmSJEl2N4yvAAAgAElEQVQCDIiSJEmSpIIBUZIkSZIEGBAlSZIkSQUDoiRJkiQJMCBKkiRJkgoGREmSJEkSYECUJEmSJBUMiJIkSZIkwIAoSZIkSSoYECVJkiRJgAFRkiRJklQwIEqSJEmSgAYMiBFxakSsi4j1EXFZH/svioi1EXFPRNwZEYeXUackSZIkjTQNFRAjYgxwDfBm4HDgvD4C4Bcyc1pmHgl8HPhEncuUJEmSpBGpoQIiMBNYn5kPZebzwE3AmdUHZOZvqjZfCWQd65MkSZKkEWu3sgvopRXYULXdARzT+6CIuBj4a2B34OT6lCZJkiRJI1uj9SDukMy8JjNfB3wY+Nu+jomICyOiPSLaN27cWN8CJUmSJGkYarSA2AlMqNoeX7T15yZgTl87MnNJZrZlZtu4ceOGsERJkiRJGpkaLSDeDUyOiEkRsTtwLrCi+oCImFy1eRrwQB3rkyRJkqQRq6HmIGbmCxFxCbASGANcm5n3RsRCoD0zVwCXRMSfAN3AU8CflVexJEmSJI0cDRUQATLzNuC2Xm3zq77/UN2LkiRJkqRRoNGGmEqSJEmSSmJAlCRJkiQBBkRJkiRJUqHh5iBKkiRpdOnu7qajo4Nnn3227FKkYW+PPfZg/PjxNDU17dTzDYglWL66k0Ur1/Hopi4OaGlm3uwpzJnRWnZZkiRJpejo6GDPPfdk4sSJRETZ5UjDVmbyxBNP0NHRwaRJk3bqHA4xrbPlqzu5fNlaOjd1kUDnpi4uX7aW5as7yy5NkiSpFM8++yz77LOP4VDaRRHBPvvss0u98QbEOlu0ch1d3Vu2aevq3sKiletKqkiSJKl8hkNpaOzqvyUDYp09uqlrUO2SJEmqvT//8z/nNa95DVOnTu33mMzk0ksv5ZBDDmH69On86Ec/2rrv+uuvZ/LkyUyePJnrr79+u9f79re/zVvf+tYhqX1nXHfddTz66KOlXX+offWrX2XKlCkccsghXHHFFX0e84lPfILDDz+c6dOnM2vWLB555JGt+8aMGcORRx7JkUceyRlnnLHd6z388MMDvldqbfny5dx33301ObcBsc4OaGkeVLskSZJq74ILLuCrX/3qgMfcfvvtPPDAAzzwwAMsWbKED3zgAwA8+eSTfOQjH+Guu+7ihz/8IR/5yEd46qmndqqOF154YaeeN1gjKSBu2bKFiy++mNtvv5377ruPG2+8sc/wNGPGDNrb21mzZg1ve9vb+Ju/+Zut+5qbm7nnnnu45557WLFixU7XUq+fnwFxBJk3ewrNTWO2aWtuGsO82VNKqkiSJGl4Wb66k+Ou+CaTLvsKx13xzSFZy+GEE05g7733HvCYL33pS7z73e8mInjDG97Apk2beOyxx1i5ciWnnHIKe++9N3vttRennHJKn2Hzq1/9KoceeihHHXUUy5Yt29q+YMECzj//fI477jjOP/98nn32Wd7znvcwbdo0ZsyYwbe+9S2gEurOPPNMTjrpJCZPnsxHPvKRref4xCc+wdSpU5k6dSqf/OQngd/v5bryyitZsGABt9xyC+3t7bzzne/kyCOPpKurziPZ1iyFq6bCgpbK45qlu3S6H/7whxxyyCEcfPDB7L777px77rl86Utf+r3j3vjGN/KKV7wCgDe84Q10dHQM6jqrVq3iiCOO4IgjjuCaa67Z2n7ddddxxhlncPLJJzNr1iwyk3nz5jF16lSmTZvGzTffDFR6jU844QROO+00pkyZwkUXXcSLL74IwI033si0adOYOnUqH/7wh7ee+1WvetXW72+55RYuuOACvv/977NixQrmzZvHkUceyYMPPjio17E9rmJaZz2rlbqKqSRJ0uD1LPjXs6ZDz4J/QM0/T3V2djJhwoSt2+PHj6ezs7Pf9mrPPvss73vf+/jmN7/JIYccwtvf/vZt9t93333ceeedNDc380//9E9EBGvXruWnP/0pb3rTm/jZz34GVMLQT37yE17xildw9NFHc9pppxERfOYzn+Guu+4iMznmmGM48cQT2Wuvvfp8HW9729u4+uqrufLKK2lraxuq/zw7Zs1SuPVS6C5C6eYNlW2A6XN36pR9/fe/6667BnzOpz/9ad785jdv3X722Wdpa2tjt91247LLLmPOnDm/95z3vOc9XH311ZxwwgnMmzdvm30/+tGPWLNmDXvvvTf/8R//wT333MOPf/xjHn/8cY4++mhOOOEEoPLzu++++zjooIM49dRTWbZsGcceeywf/vCHWbVqFXvttRdvetObWL58eZ81ABx77LGcccYZvPWtb+Vtb3vbDv932lEGxBLMmdFqIJQkSdoJAy3418ifr376058yadIkJk+eDMC73vUulixZsnX/GWecQXNzZcrRnXfeyQc/+EEADj30UA466KCtAfGUU05hn332AeDss8/mzjvvJCI466yzeOUrX7m1/T//8z93aC5d3d2x8KVw2KO7q9K+kwFxsD73uc/R3t7Od77zna1tjzzyCK2trTz00EOcfPLJTJs2jde97nVb92/atIlNmzZtDXrnn38+t99++9b9PT3IUPn5nXfeeYwZM4b99tuPE088kbvvvptXv/rVzJw5k4MPPhiA8847jzvvvJOmpiZOOukkxo0bB8A73/lOvvvd7/YbEGvNIaZlGOJudUmSpNGizAX/Wltb2bBhw9btjo4OWltb+20fjJ5wtz29V6gcaMXK3XbbbesQRmCXbn0wZDb3M6yzv/YdMJj//t/4xjf46Ec/yooVK3j5y1++zTkADj74YE466SRWr149qBpq8fPrvb9ePz8DYr31dKtv3gDkS93qhkRJkqTtKnPBvzPOOIMbbriBzOQHP/gBY8eOZf/992f27Nl87Wtf46mnnuKpp57ia1/7GrNnz97muYceeigPP/zw1vliN954Y7/XOf744/n85z8PwM9+9jN+8YtfMGVKZb2Kr3/96zz55JN0dXWxfPlyjjvuOI4//niWL1/O7373O37729/yxS9+keOPP5799tuPX//61zzxxBM899xzfPnLX956jT333JOnn356qP8Tbd/Y8YNr3wFHH300DzzwAD//+c95/vnnuemmm/rsPV29ejXvf//7WbFiBa95zWu2tj/11FM899xzADz++ON873vf4/DDD9/muS0tLbS0tHDnnXcCbP359OX444/n5ptvZsuWLWzcuJHvfve7zJw5E6gMMf35z3/Oiy++yM0338wf//EfM3PmTL7zne/w+OOPs2XLFm688UZOPPFEAPbbbz/uv/9+XnzxRb74xS9uvUYtf34GxHobqFtdkiRJA6rVgn/nnXcef/RHf8S6desYP348n/70pwH41Kc+xac+9SkA3vKWt3DwwQdzyCGH8L73vY9//ud/BmDvvffm7/7u7zj66KM5+uijmT9//u8teLPHHnuwZMkSTjvtNI466qhtAkpvf/mXf8mLL77ItGnTePvb38511123tbdr5syZnHPOOUyfPp1zzjmHtrY2jjrqKC644AJmzpzJMcccw3vf+15mzJhBU1MT8+fPZ+bMmZxyyikceuihW69xwQUXcNFFF9V/kZpZ86GpV5hvaq6076TddtuNq6++mtmzZ3PYYYcxd+5c/vAP/xCA+fPnb12VdN68eTzzzDP86Z/+6Ta3s7j//vtpa2vjiCOO4I1vfCOXXXbZ7wVEgM985jNcfPHFHHnkkWRmv/WcddZZTJ8+nSOOOIKTTz6Zj3/847z2ta8FKmH2kksu4bDDDmPSpEmcddZZ7L///lxxxRW88Y1v5IgjjuD1r389Z555JgBXXHEFb33rWzn22GPZf//9t17j3HPPZdGiRcyYMWPIF6mJgV7cSNHW1pbt7e1ll1GxoAXo6795wIJN9a5GkiSpdPfffz+HHXbYDh+/fHXnqFzw77rrrqO9vZ2rr7667FJ2zZqllc6RzR2VnsNZ8+s2/7BM3/72t7nyyiu36cmtlb7+TUXEqszc7qpELlJTb2PHF8NL+2iXJEnSdrng3zA3fe6oCITDlUNM660G3eqSJEka+S644ILh33s4ip100kl16T3cVQbEeps+F05fDGMnAFF5PH2xf0WRJEmSVDqHmJbBbnVJkqRtZOZ2l/yXtH27usZMw/UgRsSpEbEuItZHxGV97P/riLgvItZExB0RcVAZdUqSJGlo7LHHHjzxxBO7/MFWGu0ykyeeeII99thjp8/RUD2IETEGuAY4BegA7o6IFZl5X9Vhq4G2zPxdRHwA+Djw9vpXK0mSpKEwfvx4Ojo62LhxY9mlSMPeHnvswfjxO78AZkMFRGAmsD4zHwKIiJuAM4GtATEzv1V1/A+Ad9W1QkmSJA2ppqYmJk2aVHYZkmi8IaatQPU9IDqKtv78BXB7TSuSJEmSpFGi0XoQd1hEvAtoA07sZ/+FwIUABx54YB0rkyRJkqThqdF6EDuBCVXb44u2bUTEnwD/H3BGZj7X14kyc0lmtmVm27hx42pSrCRJkiSNJNFIq0VFxG7Az4BZVILh3cA7MvPeqmNmALcAp2bmAzt43o3AI0Nf8S7bF3i87CI0Yvn+Ui35/lIt+f5SrfkeUy016vvroMzcbs9ZQwVEgIh4C/BJYAxwbWZ+NCIWAu2ZuSIivgFMAx4rnvKLzDyjpHJ3SUS0Z2Zb2XVoZPL9pVry/aVa8v2lWvM9ploa7u+vhpuDmJm3Abf1aptf9f2f1L0oSZIkSRoFGm0OoiRJkiSpJAbEci0puwCNaL6/VEu+v1RLvr9Ua77HVEvD+v3VcHMQJUmSJEnlsAdRkiRJkgQYEEsREadGxLqIWB8Rl5Vdj0aeiHg4ItZGxD0R0V52PRreIuLaiPh1RPykqm3viPh6RDxQPO5VZo0avvp5fy2IiM7id9g9xQrn0qBFxISI+FZE3BcR90bEh4p2f4dplw3w/hrWv8McYlpnETGGyr0eTwE6qNzr8bzMvK/UwjSiRMTDQFtmNuI9eDTMRMQJwDPADZk5tWj7OPBkZl5R/KFrr8z8cJl1anjq5/21AHgmM68sszYNfxGxP7B/Zv4oIvYEVgFzgAvwd5h20QDvr7kM499h9iDW30xgfWY+lJnPAzcBZ5ZckyT1KzO/CzzZq/lM4Pri++up/A9RGrR+3l/SkMjMxzLzR8X3TwP3A634O0xDYID317BmQKy/VmBD1XYHI+CNpIaTwNciYlVEXFh2MRqR9svMx4rvfwnsV2YxGpEuiYg1xRBUh/9pl0XERGAGcBf+DtMQ6/X+gmH8O8yAKI1Mf5yZRwFvBi4uhnBJNZGVuQrOV9BQ+hfgdcCRwGPAP5Vbjoa7iHgV8B/Af8vM31Tv83eYdlUf769h/TvMgFh/ncCEqu3xRZs0ZDKzs3j8NfBFKkObpaH0q2LuRc8cjF+XXI9GkMz8VWZuycwXgX/F32HaBRHRROXD++czc1nR7O8wDYm+3l/D/XeYAbH+7gYmR8SkiNgdOBdYUXJNGkEi4pXFRGki4pXAm4CfDPwsadBWAH9WfP9nwJdKrEUjTM8H98JZ+DtMOykiAvg0cH9mfqJql7/DtMv6e38N999hrmJagmKp208CY4BrM/OjJZekESQiDqbSawiwG/AF32PaFRFxI3ASsC/wK+B/AcuBpcCBwCPA3Mx0oRENWj/vr5OoDM1K4GHg/VXzxaQdFhF/DPwnsBZ4sWj+n1Tmifk7TLtkgPfXeQzj32EGREmSJEkS4BBTSZIkSVLBgChJkiRJAgyIkiRJkqSCAVGSJEmSBBgQJUmSJEkFA6IkSTsoIrZExD1VX5cN4bknRsSwuleWJGnk2a3sAiRJGka6MvPIsouQJKlW7EGUJGkXRcTDEfHxiFgbET+MiEOK9okR8c2IWBMRd0TEgUX7fhHxxYj4cfF1bHGqMRHxrxFxb0R8LSKaS3tRkqRRyYAoSdKOa+41xPTtVfs2Z+Y04Grgk0Xb/wGuz8zpwOeBxUX7YuA7mXkEcBRwb9E+GbgmM/8Q2AScU+PXI0nSNiIzy65BkqRhISKeycxX9dH+MHByZj4UEU3ALzNzn4h4HNg/M7uL9scyc9+I2AiMz8znqs4xEfh6Zk4utj8MNGXm39f+lUmSVGEPoiRJQyP7+X4wnqv6fguuFSBJqjMDoiRJQ+PtVY//VXz/feDc4vt3Av9ZfH8H8AGAiBgTEWPrVaQkSQPxL5OSJO245oi4p2r7q5nZc6uLvSJiDZVewPOKtg8Cn4mIecBG4D1F+4eAJRHxF1R6Cj8APFbz6iVJ2g7nIEqStIuKOYhtmfl42bVIkrQrHGIqSZIkSQLsQZQkSZIkFexBlCSpUNzYPiPCOfqSpFHJgChJGjEi4qsRsbCP9jMj4pcGP0mSBmZAlCSNJNcD74qI6NV+PvD5zHyhhJqGRFT4/21JUk35PxpJ0kiyHNgHOL6nISL2At4K3FBsnxYRqyPiNxGxISIW7OjJI+KyiHgwIp6OiPsi4qxe+98XEfdX7T+qaJ8QEcsiYmNEPBERVxftCyLic1XP32aIa0R8OyI+GhHfA34HHBwR76m6xkMR8f5eNZwZEfcUr+/BiDg1Iv40Ilb1Ou6vI+JLO/raJUmjgwFRkjRiZGYXsBR4d1XzXOCnmfnjYvu3xf4W4DTgAxExZwcv8SCV8DkW+AjwuYjYHyAi/hRYUJz71cAZwBMRMQb4MvAIMBFoBW4axMs6H7gQ2LM4x6+pBN5XU7mv4lVVQXQmlSA8r3h9JwAPAyuASRFxWK/z3jCIOiRJo4ABUZI00lwPvC0i9ii23120AZCZ387MtZn5YmauAW4ETtyRE2fmv2fmo8VzbwYeAGYWu98LfDwz786K9Zn5SLH/AGBeZv42M5/NzDsH8Xquy8x7M/OFzOzOzK9k5oPFNb4DfI2Xekz/Arg2M79e1NiZmT/NzOeAm4F3AUTEH1IJq18eRB2SpFHAgChJGlGK8PU4MCciXkcloH2hZ39EHBMR3yqGe24GLgL23ZFzR8S7i+GbmyJiEzC16rkTqPQw9jYBeGQX5j9u6FXDmyPiBxHxZFHDW3agBqiE5HcU8zPPB5YWwVGSpK0MiJKkkegGKj2H7wJWZuavqvZ9gcqQywmZORb4FNB7UZvfExEHAf8KXALsk5ktwE+qnrsBeF0fT90AHNjPCqq/BV5Rtf3aPo7ZesPiiHg58B/AlcB+RQ237UANZOYPgOep9Da+A/hsX8dJkkY3A6IkaSS6AfgT4H1UDS8t7Ak8mZnPFnP23rGD53wllbC2ESAi3kOlB7HHvwH/IyJeX6w4ekgRKn8IPAZcERGvjIg9IuK44jn3ACdExIERMRa4fDs17A68vKjhhYh4M/Cmqv2fBt4TEbMi4mUR0RoRh1btvwG4Guge5DBXSdIoYUCUJI04mfkw8H0qoW5Fr91/CSyMiKeB+VQWtdmRc94H/BPwX8CvgGnA96r2/zvwUSo9lE9TWVF178zcApwOHAL8AugA3l485+tU5gauAVaxnTmBmfk0cGlR81NUwu2Kqv0/pFi4BtgMfAc4qOoUn6USaj+HJEl9iMzc/lGSJGnYi4hmKqugHpWZD5RdjySp8diDKEnS6PEB4G7DoSSpP31NmJckSSNMRDxMZTGbHb3noyRpFHKIqSRJkiQJcIipJEmSJKlgQJQkSZIkAaNkDuK+++6bEydOLLsMSZIkSSrFqlWrHs/Mcds7blQExIkTJ9Le3l52GZIkSZJUioh4ZEeOc4ipJEmSJAkwIEqSJEmSCgZESZIkSRJgQJQkSZIkFQyIkiRJkiTAgChJkiRJKhgQJUlSY1izFK6aCgtaKo9rlpZdkSSNOjUNiBFxakSsi4j1EXFZH/svioi1EXFPRNwZEYcX7RMjoqtovyciPlX1nNcXz1kfEYsjImr5GiRJUh2sWQq3XgqbNwBZebz1UkOiJNVZzQJiRIwBrgHeDBwOnNcTAKt8ITOnZeaRwMeBT1TtezAzjyy+Lqpq/xfgfcDk4uvUWr0GSZJUJ3cshO6ubdu6uyrtkqS6qWUP4kxgfWY+lJnPAzcBZ1YfkJm/qdp8JZADnTAi9gdenZk/yMwEbgDmDG3ZkiSp7jZ3DK5dklQTtQyIrcCGqu2Oom0bEXFxRDxIpQfx0qpdkyJidUR8JyKOrzpn9f8p+jynJEkaZsaOH1y7JKkmSl+kJjOvyczXAR8G/rZofgw4MDNnAH8NfCEiXj2Y80bEhRHRHhHtGzduHNqiJUnS0Jo1H5qat21raq60S5LqppYBsROYULU9vmjrz00Uw0Uz87nMfKL4fhXwIPAHxfOr/5TY7zkzc0lmtmVm27hx43b6RUiSpDqYPhdOXwxjJwBReTx9caVdklQ3u9Xw3HcDkyNiEpUQdy7wjuoDImJyZj5QbJ4GPFC0jwOezMwtEXEwlcVoHsrMJyPiNxHxBuAu4N3A/6nha5AkSfUyfa6BUJJKVrOAmJkvRMQlwEpgDHBtZt4bEQuB9sxcAVwSEX8CdANPAX9WPP0EYGFEdAMvAhdl5pPFvr8ErgOagduLL0mSJEnSLorKYqAjW1tbW7a3t5ddhiRJkiSVIiJWZWbb9o4rfZEaSZIkSVJjMCBKkiRJkgADoiRJkiSpYECUJEmSJAEGREmS1NuapXDVVFjQUnlcs7TsiiRJdVLL+yBKkqThZs1SuPVS6O6qbG/eUNkG71EoSaOAPYiSJOkldyx8KRz26O6qtEuSRjwDoiRJesnmjsG1S5JGFAOiJEl6ydjxg2uXJI0oBkRJkv5fe3cfJmddH/r//SHZyAKa5SFSSACD5IRiEpN2Ez1Gng1BgRARI1RUfEKslHp6nUjya6XAqYcIVLwoXFWqAp4qECnEWIsBedDGKmYhaQKBSHhQdqGFAEHRBULy+f0xs2E27Cab7N57Z2fer+vaa+b+3A/zndl7Z+cz3+/3c+s1x54PTc3dY03Nlbgkqe6ZIEqSpNdMmgMnXQEjDwCicnvSFRaokaQGYRVTSZLU3aQ5JoSS1KDsQZQkSZIkASaIkiRJkqQqE0RJkiRJEmCCKEmSJEmqMkGUJEmSJAEmiJIkSZKkKhNESZIkSRJggihJkiRJqjJBlCRJkiQBJoiSJEmSpCoTREmSJEkSUHCCGBHHR8SaiFgbEfN6WH92RKyKiBURsTQiDqvGZ0TEvdV190bEMTX73F095orqz5uLfA6SJEmS1CiGF3XgiBgGXAXMANqBZRGxODNX12z23cz8WnX7WcBXgOOBdcBJmflkREwAlgCja/b7cGa2FdV2SZIkFWTlQrjjInihHUaOgWPPh0lzym6V1H91cm4XliAC04C1mfkoQETcAJwMbE4QM/O3NdvvDmQ1vrwm/gDQHBFvyMyXC2yvJEmSirRyIfzgXNjQWVl+4YnKMgzJD9LSZnV0bhc5xHQ08ETNcjvdewEBiIjPRcQjwCXAuT0c5wPAfVskh9dUh5d+MSJiIBstSZKkgtxx0WsfoLts6KzEpaGsjs7t0ovUZOZVmflW4Dzgb2rXRcTbgC8Dn6kJfzgzJwKHV38+0tNxI+KsiGiLiLZnnnmmmMZLkiSp715o3764+mXR8g6mL7iTsfN+yPQFd7JoeUfZTapfdXRuF5kgdgAH1CyPqcZ6cwMwu2shIsYAtwAfzcxHuuKZ2VG9/R3wXSpDWV8nM6/OzNbMbB01atQOPwlJkiQNkJFjti+uHbZoeQfzb15Fx/pOEuhY38n8m1eZJBaljs7tIhPEZcC4iBgbESOA04DFtRtExLiaxROAh6vxFuCHwLzM/FnN9sMjYp/q/SbgROD+Ap+DJEmSBsqx50NTc/dYU3MlrgF16ZI1dG7Y2C3WuWEjly5ZU1KL6lwdnduFFanJzFcj4hwqFUiHAd/KzAci4iKgLTMXA+dExHuADcDzwMequ58DHAKcHxFdr+pxwO+BJdXkcBjwY+CfinoOkiRJGkBdxTrqoNLjdimhuuWT6zu3K143yqokWkfndmRm2W0oXGtra7a1eVUMSZIkDbItq1tCpWfppCsKTR6mL7iTjh6SwdEtzfxs3jE97FEHSnqth4qIuDczW7e1XelFaiRJkqS6VVJ1y7kzx9PcNKxbrLlpGHNnji/0cUtVR5VEy1TkdRAlSZKkxlZSdcvZUypXl7t0yRqeXN/J/i3NzJ05fnO8cGUM9ayjSqJlMkGUJEmSijJyTOWi6T3FCzZ7yujBSwhrlXXR+BJf63riEFNJkiSpKHVU3bLPyhrq2YivdQFMECVJkhrRyoVw+QS4oKVyu3Jh2S2qT5PmVIqkjDwAiMptvRdNKWuoZyO+1gVwiKkkSVKjKWsIYKOaNKexXtcyh3o22mtdAHsQJUmSGo3VHgfVouUdTF9wJ2Pn/ZDpC+5k0fKOsptUqGVv/Qs6c0S3WGeOYNlb/6KkFml7mCBKkiQ1Gqs9DppFyzuYf/MqOtZ3kkDH+k7m37yqrpPEz68ex3kbPkX7pn3YlEH7pn04b8On+PzqcWU3TX3gEFNJkqRGY7XHQXPpkjV0btjYLda5YSOXLllTToXRQfDk+k46eDeLX3l3t3is7+xlD+1MTBAlSZIazbHnd5+DCINW7XHR8o7Srs1XxmM/2UtS1Fu8Huzf0kxHD89v/5bmHrbWzsYhppK0o6wAKGmoKqnaY5nDLct67N6SonpOlubOHE9z07BuseamYcydOb7wx260+Z5FsAdRknaEFQBVx8rs4dEgKqHaY5nDLct67LkzxzP/5lXdHnuwkqWydL2eg/0+0vUlQNdr3fUlQG2btG0miJK0I7ZWAdAEUUOYH7BUpDKHW5b12GUlS2WbPWX0oD/HRpzvWQQTREnaEVYAVJ3yA1bjKKOnuMy5afu3NPOnv72dLwxfyP6xjidzHy55dQ73vmlG4Y9dRrLUiBpxvmcRnIMoSTuit0p/VgDUEOcHrMZQ1ny8MuemffWwh/ly0zcYs8s6dgkYs8s6vtz0Db562MOFP7YGRzoHy5oAACAASURBVCPO9yyCCaIk7Yhjz69U/Ks1SBUApSL5AasxbK2nuEizp4zm4lMmMrqlmQBGtzRz8SkTB6V3beoj/0BzvNIt1hyvMPWRfyj8sTU4yvwCop44xFSSdkTXPMM7LqoMKx05ppIcOv+wOCsXNtzrXcYQwEYsqFGmsgoCldlTXNpwS6cG1L2y53vWS4EvE0RJ2lElVABsWA1YNbasYjFlf8BqJGUWBGrI69SNHFN57+gprrpR1hcQ9VTgyyGmkqSd39aqxtapsoYAQuXDzM/mHcNjC07gZ/OOGXIfboaKMn/HDTkUz6kBKlCZf88DzR5ESdLOrwGHhj25vpNZuyx9XcXFH6x/d9lNq08lDGEue5gnNFhPsVMDVKB6KvBlgihJ2vmVODSsrDklH9vjl3xhwzfYrVpUY0ysY0HTN9iraQRwQuGP31BKGsJc9jDPhrz0glMDVJCy/54HkkNMJUk7v5KGhpV1KQCALzTduDk57LJbvMIXmm4s/LEbTklDmBtymKdUp+rp77nQBDEijo+INRGxNiLm9bD+7IhYFRErImJpRBxWs25+db81ETGzr8eUJNWhSXPgpCtg5AFAVG5PuqLwnoAy55Ts1vlf2xXXjstehir3Fh8oZV7yQdLAqqe/58KGmEbEMOAqYAbQDiyLiMWZubpms+9m5teq288CvgIcX00UTwPeBuwP/Dgi/kd1n20dU5JUlDIvNVHC0LBS55RYcXHQ/Df78Ec800u8WA05zFOqU/Xy91xkD+I0YG1mPpqZrwA3ACfXbpCZv61Z3B3I6v2TgRsy8+XMfAxYWz3eNo8pNbpFyzuYvuBOxs77IdMX3DkoQ+HUILrmab3wBJCvzdNaubDslhWm1IvGW3Fx0Fz8ygf5Q47oFvtDjuDiVz5YUoskqTxFJoijgdqvPtursW4i4nMR8QhwCXDuNvbt0zGlncLKhXD5BLigpXI7CB+iy5wvVaoSXuuG1ICXmih1TklJw2obUdubZjBvw6do37QPmzJo37QP8zZ8irY3zSi7aVL/+P9RO6D0KqaZeRVwVUT8GfA3wMcG4rgRcRZwFsCBBx44EIeU+q6kiniXLlnDjI0/4QsjupfFv3TJiLoY8tCjBryAemka8FITpV8KwIqLg2LuzPHMv/kVFr/y2iVEmpuGcfEQLC4hbeb/R+2gIhPEDuCAmuUx1VhvbgD+sQ/79umYmXk1cDVAa2tr9rSNVJit9bQU+Kbc+tvbubjp9WXx5/8W4JjCHrdUJb3WDalB58TVy5wS9a70LwKkIvj/UTuoyARxGTAuIsZSSeJOA/6sdoOIGJeZD1cXTwC67i8GvhsRX6FSpGYc8EsgtnVMaadQUk/L/BHfYzdeXxZ//ojvARcX+tilacBerdIce373b6PBOXGqG34RoLrj/0ftoMISxMx8NSLOAZYAw4BvZeYDEXER0JaZi4FzIuI9wAbgearDS6vbLQRWA68Cn8vMjQA9HbOo5yDtsJJ6WvZl3XbF60KD9mqVousb57KqmEqS+s7/j9pBkVn/oy9bW1uzra2t7GaokWw57h8qPS1FF5i4fEIv/wwOgP91f3GPW6ayXmtJknZm/n/UFiLi3sxs3dZ2RVYxlRpXWdUHG7EsvpUeJUl6Pf8/agfZgyjVmzIvZC5JkqSdUl97EEu/zIWkAWZZfEmSJO0gh5hKkiSVxQuZS9rJ2IMoaWA4tFWSto8XMpe0E7IHUVL/dX3IeeEJIF/7kOM34cWx10Ea+rZ2IXNJKokJoqT+80PO4DIhl+qDFzKXtBMyQZTUf37IGVwm5KpTi5Z3MH3BnYyd90OmL7iTRcs7ym5SsXq7YLkXMpdUIhNESf3nh5zBZUKuOrRoeQfzb15Fx/pOEuhY38n8m1fVd5LYiNeulbTTM0GU1H9+yBlcJuSqQ5cuWUPnho3dYp0bNnLpkjUltWgQeCFzSTshq5hK6r+uDzNWMR0cx57fvfIhmJBryHtyfed2xeuG166VtJMxQZQ0MPyQM3hMyFWH9m9ppqOHZHD/luYetpYkFcUEUZKGIhNy1Zm5M8cz/+ZV3YaZNjcNY+7M8SW2SpIajwmiJEkq3ewpo4HKXMQn13eyf0szc2eO3xyXJA0OE0RJA2LR8g4/2Enql9lTRvu+IUklM0GU1G9d5em7hoZ1lacH/LAnSZI0hHiZC0n91pDl6SVJkuqQCaKkfmvY8vRqDCsXwuUT4IKWyu3KhWW3SJKkwmwzQYyIv4iIPQejMZKGpt7K0FueXkPeyoWVa06+8ASQldsfnGuSKEmqW33pQdwXWBYRCyPi+IiIohslaWiZO3M8zU3DusUsT6+Btmh5B9MX3MnYeT9k+oI7WbS8o/gHveMi2LBFT/iGzkpckqQ6tM0iNZn5NxHxReA44OPAlRGxEPhmZj5SdAOl/rCy5uCwPL2KVlohpBfaty8uSdIQ16cqppmZEfFfwH8BrwJ7AjdFxO2Z+YUiGyjtKCtrDi7L06tIWyuEVOh5N3JMdXhpD3FJkupQX+Yg/mVE3AtcAvwMmJiZnwX+FPjANvY9PiLWRMTaiJjXw/q/iojVEbEyIu6IiIOq8aMjYkXNz0sRMbu67tqIeKxm3eQdeN4aZGUMDbt0yRpmbPwJS0ecy6Nv+DOWjjiXGRt/YmVNaQgqrRDSsedD0xZzaZuaK3FJkupQX3oQ9wJOycxf1wYzc1NEnNjbThExDLgKmAG0U5nHuDgzV9dsthxozcw/RMRnqSShH8rMu4DJ1ePsBawFbqvZb25m3tSHtmsnUFZPXutvb+fipm+wW7wCwJhYx4KmbzD/twDHFPa40mBotOHT+7c009FDMlh4IaRJcyq3d1xUGVY6ckwlOeyKS5JUZ/pSpOZW4LmuhYh4U0S8AyAzH9zKftOAtZn5aGa+AtwAnFy7QWbelZl/qC7+AuhpzM6pwK0122mIKesaefNHfG9zcthlt3iF+SO+V+jjSkXr+tKlY30nyWtfugxK0ZaSlFoIadIc+F/3wwXrK7cmh5KkOtaXBPEfgRdrll+sxrZlNFA7caO9GuvNJ6kko1s6Dbh+i9iXqsNSL4+IN/ShLSpRWUPD9mXddsWloaKsL13KNHvKaC4+ZSKjW5oJYHRLMxefMrGue00lSSpDX4aYRmZm10J1aGmfitv0VUScAbQCR24R3w+YCCypCc+nUixnBHA1cB7wunrjEXEWcBbAgQceOJDN1XYqa2hY9FJcIiwuoSGutPl4JbMQkiRJxetLD+KjEXFuRDRVf/4SeLQP+3UAB9Qsj6nGuomI9wB/DczKzJe3WD0HuCUzN3QFMvOprHgZuIbKUNbXycyrM7M1M1tHjRrVh+aqKKUNDbO4hOpUb1+uFD4fT5Ik1b2+JIhnA++ikty1A++g2jO3DcuAcRExNiJGUBkqurh2g4iYAnydSnL4dA/HOJ0thpdWexWJiABmA/f3oS0qUWlDwybNgZOugJEHAFG5PemKQZs/VMpFvdUQSp2PJ0mS6lrUjB4d+INHvA/4KjAM+FZmfikiLgLaMnNxRPyYyhDSp6q7/CYzZ1X3fQuVy2ockJmbao55JzAKCGAFcHZm1s6RfJ3W1tZsa2sb0Ocmbc2WlVuh8gHeOVMaKI1WxVSSJPVPRNybma3b3G5bCWJE7EqlgMzbgF274pn5if42crCYIGqwTV9wZ4/zLke3NPOzeV5io16YpEmSpKGirwliX4aY/j/gj4CZwE+ozCX8Xf+aJ9W3Ri0i0kga8VITkiSp/vUlQTwkM78I/D4zrwNOoDIPUVIvLCJS/xrxUhOSJKn+9SVB7Koguj4iJgAjgTcX1yRp6LOISP2zl1iSJNWjvlzP8OqI2BP4GypVSPcAvlhoq6QhrmsemvPT6ldZ1/eUJEkq0lYTxIjYBfhtZj4P/BQ4eFBaJdUBL+pd3+bOHN9jpVp7iSVJ0lC21SGm1ctLfGGQ2iJJQ0Zp1/eUJEkqUF+GmP44Iv43cCPw+65gZj5XWKskaQiwl1iSJNWbviSIH6refq4mljjcVJIkSZLqyjYTxMwcOxgNkSRJkiSVa5sJYkR8tKd4Zn574JsjSdqZLVreYXVeSZLqWF+GmE6tub8rcCxwH2CCKEkNZNHyjm6VWzvWdzL/5lUAJomSJNWJvgwx/Yva5YhoAW4orEWSpJ3SpUvWdLusB0Dnho1cumSNCaIkSXViq5e56MXvAeclSlKDeXJ953bFJUnS0NOXOYg/oFK1FCoJ5WHAwiIbpWI4d0hSf+zf0kxHD8ng/i3NJbRGkiQVoS9zEC+ruf8q8OvMbC+oPSqIc4ck9dfcmeO7vY8ANDcNY+7M8SW2SpIkDaS+JIi/AZ7KzJcAIqI5It6SmY8X2jINKOcOSeqvrvcKRyJIklS/+pIgfg94V83yxmpsas+ba2fk3CFJA2H2lNEmhJIk1bG+JIjDM/OVroXMfCUiRhTYJhXAuUOqZ86vlSRJGhh9qWL6TETM6lqIiJOBdcU1SUWYO3M8zU3DusUGde7QyoVw+QS4oKVyu9I6RxoYXfNrO9Z3krw2v3bR8o6ymyZJkjTk9KUH8WzgOxFxZXW5HfhocU1SEUqdO7RyIfzgXNhQ7cF84YnKMsCkOcU/vuqa82slSZIGzjYTxMx8BHhnROxRXX6x8FapEKXNHbrjoteSwy4bOitxE0T1k/NrJUmSBs42h5hGxP+NiJbMfDEzX4yIPSPi7wajcaoTL/RyVZTe4tJ26G0erfNrJUmStl9f5iC+NzPXdy1k5vPA+/py8Ig4PiLWRMTaiJjXw/q/iojVEbEyIu6IiINq1m2MiBXVn8U18bERcU/1mDdaMGcIGDlm++LSdih9fq0kSVId6UuCOCwi3tC1EBHNwBu2sn3XdsOAq4D3AocBp0fEYVtsthxozcxJwE3AJTXrOjNzcvVnVk38y8DlmXkI8DzwyT48B5Xp2POhaYvenKbmSlzqp9lTRnPxKRMZ3dJMAKNbmrn4lInOP5QkSdoBfSlS8x3gjoi4BgjgTOC6Puw3DVibmY8CRMQNwMnA6q4NMvOumu1/AZyxtQNGRADHAH9WDV0HXAD8Yx/ao7J0zTO846LKsNKRYyrJofMPNUC8Np8kSdLA6EuRmi9HxH8C7wESWAIctPW9ABgNPFGz3A68YyvbfxK4tWZ514hoA14FFmTmImBvYH1mvlpzTD8VDgWT5pgQSpIkSTu5vvQgAvw3leTwg8BjwL8MZCMi4gygFTiyJnxQZnZExMHAnRGxCnhhO455FnAWwIEHHjiQzZUkSZKkutTrHMSI+B8R8bcR8RDwD8BvgMjMozPzyt72q9EBHFCzPKYa2/Jx3gP8NTArM1/uimdmR/X2UeBuYArwLNASEV2JbY/HrO53dWa2ZmbrqFGj+tBcSZIkSWpsWytS8xCV+X4nZua7M/MfgI1b2X5Ly4Bx1aqjI4DTgMW1G0TEFODrVJLDp2vie3YVxomIfYDpwOrMTOAu4NTqph8Dvr8dbZIkSZIk9WJrCeIpwFPAXRHxTxFxLJUiNX1SnSd4DpU5iw8CCzPzgYi4KCK6qpJeCuwBfG+Ly1n8MdBWnft4F5U5iF3Fbc4D/ioi1lKZk/jNvrZJkiRJktS7qHTKbWWDiN2pVB89nUqP4reBWzLztuKbNzBaW1uzra2t7GZIkiRJUiki4t7MbN3Wdtu8DmJm/j4zv5uZJ1GZ87ecSi+eJEmSJKmObDNBrJWZz1eLvxxbVINUoJUL4fIJcEFL5XblwrJbJEmSJGkn0tfLXGioW7kQfnAubOisLL/wRGUZvD6hJEmSJGA7exA1hN1x0WvJYZcNnZW4JEmSJGGC2DheaN++uCRJkqSGY4LYKEaO2b64JEmSpIZjgtgojj0fmpq7x5qaK3FJkiRJwgSxcUyaAyddASMPAKJye9IVFqiRJEmStJlVTBvJpDkmhJIkSZJ6ZQ+iJEmSJAkwQZQkSZIkVZkgSpIkSZIAE0RJkiRJUpUJoiRJkiQJMEGUJEmSJFWZIEqSJEmSABNESZIkSVKVCaIkSZIkCTBBlCRJkiRVmSBKkiRJkgATREmSJElSlQmiJEmSJAkoOEGMiOMjYk1ErI2IeT2s/6uIWB0RKyPijog4qBqfHBE/j4gHqus+VLPPtRHxWESsqP5MLvI5SJIkSVKjKCxBjIhhwFXAe4HDgNMj4rAtNlsOtGbmJOAm4JJq/A/ARzPzbcDxwFcjoqVmv7mZObn6s6Ko5yBJkiRJjaTIHsRpwNrMfDQzXwFuAE6u3SAz78rMP1QXfwGMqcZ/lZkPV+8/CTwNjCqwrZIkSZLU8IpMEEcDT9Qst1djvfkkcOuWwYiYBowAHqkJf6k69PTyiHjDQDRWkiRJkhrdTlGkJiLOAFqBS7eI7wf8P+DjmbmpGp4PHApMBfYCzuvlmGdFRFtEtD3zzDOFtV2SJEmS6kWRCWIHcEDN8phqrJuIeA/w18CszHy5Jv4m4IfAX2fmL7rimflUVrwMXENlKOvrZObVmdmama2jRjk6VZIkSZK2pcgEcRkwLiLGRsQI4DRgce0GETEF+DqV5PDpmvgI4Bbg25l50xb77Fe9DWA2cH+Bz0GSJEmSGsbwog6cma9GxDnAEmAY8K3MfCAiLgLaMnMxlSGlewDfq+R7/CYzZwFzgCOAvSPizOohz6xWLP1ORIwCAlgBnF3Uc5AkSZKkRhKZWXYbCtfa2pptbW1lN0OSJEmSShER92Zm67a22ymK1EiSJEmSymeCKEmSJEkCTBAlSZIkSVUmiJIkSZIkwARRkiRJklRlgihJkiRJAkwQJUmSJElVJoiSJEmSJMAEUZIkSZJUZYIoSZIkSQJMECVJkiRJVSaIkiRJkiTABFGSJEmSVGWCKEmSJEkCTBAlSZIkSVUmiJIkSZIkwARRkiRJklRlgihJkiRJAkwQJUmSJElVJoiSJEmSJMAEUZIkSZJUZYIoSZIkSQIKThAj4viIWBMRayNiXg/r/yoiVkfEyoi4IyIOqln3sYh4uPrzsZr4n0bEquoxr4iIKPI5SJIkSVKjKCxBjIhhwFXAe4HDgNMj4rAtNlsOtGbmJOAm4JLqvnsBfwu8A5gG/G1E7Fnd5x+BTwPjqj/HF/UcJEmSJKmRFNmDOA1Ym5mPZuYrwA3AybUbZOZdmfmH6uIvgDHV+zOB2zPzucx8HrgdOD4i9gPelJm/yMwEvg3MLvA5SJIkSVLDKDJBHA08UbPcXo315pPArdvYd3T1fl+PKUmSJEnqo+FlNwAgIs4AWoEjB/CYZwFnARx44IEDdVhJkiRJqltF9iB2AAfULI+pxrqJiPcAfw3MysyXt7FvB68NQ+31mACZeXVmtmZm66hRo3b4SUiSJElSoygyQVwGjIuIsRExAjgNWFy7QURMAb5OJTl8umbVEuC4iNizWpzmOGBJZj4F/DYi3lmtXvpR4PsFPgdJkiRJahiFDTHNzFcj4hwqyd4w4FuZ+UBEXAS0ZeZi4FJgD+B71atV/CYzZ2XmcxHxf6gkmQAXZeZz1ft/DlwLNFOZs3grkiRJkqR+i0ox0PrW2tqabW1tZTdDkiRJkkoREfdmZuu2tityiKkkSZIkaQgxQZQkSZIkASaIkiRJkqQqE0RJkiRJEmCCKEmSJEmqMkGUJEmSJAEmiJIkSZKkKhNESZIkSRJggihJkiRJqjJBlCRJkiQBJoiSJEmSpCoTREmSJEkSAMPLbkAjWrS8g0uXrOHJ9Z3s39LM3JnjmT1ldNnNkiRJKsWGDRtob2/npZdeKrsp0pC36667MmbMGJqamnZofxPEQbZoeQfzb15F54aNAHSs72T+zasATBIlSVJDam9v541vfCNvectbiIiymyMNWZnJs88+S3t7O2PHjt2hYzjEdJBdumTN5uSwS+eGjVy6ZE1JLZIkSSrXSy+9xN57721yKPVTRLD33nv3qzfeBHGQPbm+c7vikiRJjcDkUBoY/f1bMkEcZPu3NG9XXJIkScX7xCc+wZvf/GYmTJjQ6zaZybnnnsshhxzCpEmTuO+++zavu+666xg3bhzjxo3juuuu2+bj3X333Zx44okD0vYdce211/Lkk0+W9vgD7Uc/+hHjx4/nkEMOYcGCBT1u85WvfIXDDjuMSZMmceyxx/LrX/9687phw4YxefJkJk+ezKxZs7b5eI8//vhWz5WiLVq0iNWrVxdybBPEQTZ35niam4Z1izU3DWPuzPEltUiSJGloWbS8g+kL7mTsvB8yfcGdLFre0e9jnnnmmfzoRz/a6ja33norDz/8MA8//DBXX301n/3sZwF47rnnuPDCC7nnnnv45S9/yYUXXsjzzz+/Q+149dVXd2i/7VVqgrhyIVw+AS5oqdyuXNivw23cuJHPfe5z3HrrraxevZrrr7++x+RpypQptLW1sXLlSk499VS+8IUvbF7X3NzMihUrWLFiBYsXL97htgzW788EsY7MnjKai0+ZyOiWZgIY3dLMxadMtECNJElSH3QV/OtY30nyWsG//iaJRxxxBHvttddWt/n+97/PRz/6USKCd77znaxfv56nnnqKJUuWMGPGDPbaay/23HNPZsyY0WOy+aMf/YhDDz2UP/mTP+Hmm2/eHL/gggv4yEc+wvTp0/nIRz7CSy+9xMc//nEmTpzIlClTuOuuu4BKUnfyySdz1FFHMW7cOC688MLNx/jKV77ChAkTmDBhAl/96leB1/dyXXbZZVxwwQXcdNNNtLW18eEPf5jJkyfT2TmIU51WLoQfnAsvPAFk5fYH5/YrSfzlL3/JIYccwsEHH8yIESM47bTT+P73v/+67Y4++mh22203AN75znfS3t6+XY9z77338va3v523v/3tXHXVVZvj1157LbNmzeKYY47h2GOPJTOZO3cuEyZMYOLEidx4441Apdf4iCOO4IQTTmD8+PGcffbZbNq0CYDrr7+eiRMnMmHCBM4777zNx95jjz0237/ppps488wz+Y//+A8WL17M3LlzmTx5Mo888sh2PY9tsYppCWZPGW1CKEmStAO2VvCv6M9XHR0dHHDAAZuXx4wZQ0dHR6/xWi+99BKf/vSnufPOOznkkEP40Ic+1G396tWrWbp0Kc3Nzfz93/89EcGqVat46KGHOO644/jVr34FVJKh+++/n912242pU6dywgknEBFcc8013HPPPWQm73jHOzjyyCPZc889e3wep556KldeeSWXXXYZra2tA/Xy9M0dF8GGLRLSDZ2V+KQ5O3TInl7/e+65Z6v7fPOb3+S9733v5uWXXnqJ1tZWhg8fzrx585g9e/br9vn4xz/OlVdeyRFHHMHcuXO7rbvvvvtYuXIle+21F//yL//CihUr+M///E/WrVvH1KlTOeKII4DK72/16tUcdNBBHH/88dx88828613v4rzzzuPee+9lzz335LjjjmPRokU9tgHgXe96F7NmzeLEE0/k1FNP7fPr1Ff2IEqSJGnIGKoF/x566CHGjh3LuHHjiAjOOOOMbutnzZpFc3OlJsXSpUs3rz/00EM56KCDNieIM2bMYO+996a5uZlTTjmFpUuXsnTpUt7//vez++67s8cee3DKKafw7//+74P7BPvqhV567XqLF+Cf//mfaWtr65bk/frXv6atrY3vfve7fP7zn39dr9z69etZv3795kTvIx/5SLf1XT3IUPn9nX766QwbNox9992XI488kmXLlgEwbdo0Dj74YIYNG8bpp5/O0qVLWbZsGUcddRSjRo1i+PDhfPjDH+anP/1pkS/BVpkglmGAx11LkiQ1ijIL/o0ePZonnnhi83J7ezujR4/uNb49dt999z5tt2WFyq1VrBw+fPjmIYxAvy59MGBGjtm+eB9sz+v/4x//mC996UssXryYN7zhDd2OAXDwwQdz1FFHsXz58u1qQxG/vy3XD9bvr9AEMSKOj4g1EbE2Iub1sP6IiLgvIl6NiFNr4kdHxIqan5ciYnZ13bUR8VjNuslFPocBV8C4a0mSpEZRZsG/WbNm8e1vf5vM5Be/+AUjR45kv/32Y+bMmdx22208//zzPP/889x2223MnDmz276HHnoojz/++Oaeqeuvv77Xxzn88MP5zne+A8CvfvUrfvOb3zB+fOX53X777Tz33HN0dnayaNEipk+fzuGHH86iRYv4wx/+wO9//3tuueUWDj/8cPbdd1+efvppnn32WV5++WX+9V//dfNjvPGNb+R3v/vdQL9E23bs+dC0RTLf1FyJ76CpU6fy8MMP89hjj/HKK69www039FiJdPny5XzmM59h8eLFvPnNb94cf/7553n55ZcBWLduHT/72c847LDDuu3b0tJCS0sLS5cuBdj8++nJ4Ycfzo033sjGjRt55pln+OlPf8q0adOAyhDTxx57jE2bNnHjjTfy7ne/m2nTpvGTn/yEdevWsXHjRq6//nqOPPJIAPbdd18efPBBNm3axC233LL5MYr8/RU2BzEihgFXATOAdmBZRCzOzNpyO78BzgT+d+2+mXkXMLl6nL2AtcBtNZvMzcybimp7oQoYdy1JktQouuYZXrpkDU+u72T/lmbmzhzf7/mHp59+OnfffTfr1q1jzJgxXHjhhXzyk5/ka1/7GgBnn30273vf+/i3f/s3DjnkEHbbbTeuueYaAPbaay+++MUvMnXqVADOP//81xW82XXXXbn66qs54YQT2G233Tj88MN7/YD/53/+53z2s59l4sSJDB8+nGuvvXZzb9e0adP4wAc+QHt7O2ecccbmOYRnnnnm5iTkU5/6FFOmTNnclmnTpjF69GgOPfTQzY9x5plncvbZZ9Pc3MzPf/7zzcNbC9f1efeOiyrDSkeOqSSH/fgcPHz4cK688kpmzpzJxo0b+cQnPsHb3vY2oPL8W1tbmTVrFnPnzuXFF1/kgx/8IAAHHnggixcv5sEHH+Qzn/kMu+yyC5s2bWLevHmvSxAB5nbtgQAAB8dJREFUrrnmGj7xiU8QERx33HG9tuf9738/P//5z3n7299ORHDJJZfwR3/0Rzz00ENMnTqVc845h7Vr13L00Ufz/ve/n1122YUFCxZw9NFHk5mccMIJnHzyyQAsWLCAE088kVGjRtHa2sqLL74IwGmnncanP/1prrjiCm666Sbe+ta37vDrt6XIzAE7WLcDR/xP4ILMnFldng+QmRf3sO21wL/2lPRFxFnAkZn54W1t25vW1tZsa2vbkacx8C5oAXp6zQMuWD/YrZEkSSrdgw8+yB//8R+X3Yyd3rXXXktbWxtXXnll2U3RDrj77ru57LLLuvXkFqWnv6mIuDczt1mVqMghpqOBJ2qW26ux7XUasGUf/JciYmVEXB4Rb+hpp51WAeOuJUmSJGkg7NRFaiJiP2AisKQmPB84FJgK7AWc18OuRMRZEdEWEW3PPPNM4W3tswLGXUuSJKn+nXnmmfYeDmFHHXXUoPQe9leRCWIHcEDN8phqbHvMAW7JzA1dgcx8KiteBq4BpvW0Y2ZenZmtmdk6atSo7XzYAk2aAyddASMPAKJye9IVzj+UJEmSVLrCitQAy4BxETGWSmJ4GvBn23mM06n0GG4WEftl5lNRqfk6G7h/IBo7qCbNMSGUJEmqkZnbLPkvadv6W2OmsB7EzHwVOIfK8NAHgYWZ+UBEXBQRswAiYmpEtAMfBL4eEQ907R8Rb6HSA/mTLQ79nYhYBawC9gH+rqjnIEmSpOLtuuuuPPvss/3+YCs1uszk2WefZdddd93hYxRWxXRnslNVMZUkSVI3GzZsoL29fee4kLs0xO26666MGTOGpqambvG+VjEtcoipJEmStE1NTU2MHTu27GZIYievYipJkiRJGjwmiJIkSZIkwARRkiRJklTVEEVqIuIZ4Ndlt6MH+wDrym6E6pbnl4rk+aUieX6paJ5jKtLOen4dlJnbvEB8QySIO6uIaOtLJSFpR3h+qUieXyqS55eK5jmmIg3188shppIkSZIkwARRkiRJklRlgliuq8tugOqa55eK5PmlInl+qWieYyrSkD6/nIMoSZIkSQLsQZQkSZIkVZkgliAijo+INRGxNiLmld0e1Z+IeDwiVkXEiohoK7s9Gtoi4lsR8XRE3F8T2ysibo+Ih6u3e5bZRg1dvZxfF0RER/U9bEVEvK/MNmroiogDIuKuiFgdEQ9ExF9W476Hqd+2cn4N6fcwh5gOsogYBvwKmAG0A8uA0zNzdakNU12JiMeB1szcGa/BoyEmIo4AXgS+nZkTqrFLgOcyc0H1i649M/O8MtupoamX8+sC4MXMvKzMtmnoi4j9gP0y876IeCNwLzAbOBPfw9RPWzm/5jCE38PsQRx804C1mfloZr4C3ACcXHKbJKlXmflT4LktwicD11XvX0flH6K03Xo5v6QBkZlPZeZ91fu/Ax4ERuN7mAbAVs6vIc0EcfCNBp6oWW6nDk4k7XQSuC0i7o2Is8pujOrSvpn5VPX+fwH7ltkY1aVzImJldQiqw//UbxHxFmAKcA++h2mAbXF+wRB+DzNBlOrTuzPzT4D3Ap+rDuGSCpGVuQrOV9BA+kfgrcBk4Cng78ttjoa6iNgD+Bfg85n529p1voepv3o4v4b0e5gJ4uDrAA6oWR5TjUkDJjM7qrdPA7dQGdosDaT/rs696JqD8XTJ7VEdycz/zsyNmbkJ+Cd8D1M/REQTlQ/v38nMm6th38M0IHo6v4b6e5gJ4uBbBoyLiLERMQI4DVhccptURyJi9+pEaSJid+A44P6t7yVtt8XAx6r3PwZ8v8S2qM50fXCvej++h2kHRUQA3wQezMyv1KzyPUz91tv5NdTfw6xiWoJqqduvAsOAb2Xml0pukupIRBxMpdcQYDjwXc8x9UdEXA8cBewD/Dfwt8AiYCFwIPBrYE5mWmhE262X8+soKkOzEngc+EzNfDGpzyLi3cC/A6uATdXw/0dlnpjvYeqXrZxfpzOE38NMECVJkiRJgENMJUmSJElVJoiSJEmSJMAEUZIkSZJUZYIoSZIkSQJMECVJkiRJVSaIkiT1UURsjIgVNT/zBvDYb4mIIXWtLElS/RledgMkSRpCOjNzctmNkCSpKPYgSpLUTxHxeERcEhGrIuKXEXFINf6WiLgzIlZGxB0RcWA1vm9E3BIR/1n9eVf1UMMi4p8i4oGIuC0imkt7UpKkhmSCKElS3zVvMcT0QzXrXsjMicCVwFersX8ArsvMScB3gCuq8SuAn2Tm24E/AR6oxscBV2Xm24D1wAcKfj6SJHUTmVl2GyRJGhIi4sXM3KOH+OPAMZn5aEQ0Af+VmXtHxDpgv8zcUI0/lZn7RMQzwJjMfLnmGG8Bbs/McdXl84CmzPy74p+ZJEkV9iBKkjQwspf72+PlmvsbsVaAJGmQmSBKkjQwPlRz+/Pq/f8ATqve/zDw79X7dwCfBYiIYRExcrAaKUnS1vjNpCRJfdccEStqln+UmV2XutgzIlZS6QU8vRr7C+CaiJgLPAN8vBr/S+DqiPgklZ7CzwJPFd56SZK2wTmIkiT1U3UOYmtmriu7LZIk9YdDTCVJkiRJgD2IkiRJkqQqexAlSZIkSYAJoiRJkiSpygRRkiRJkgSYIEqSJEmSqkwQJUmSJEmACaIkSZIkqer/B8GixQ+DGvT5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "  solver = solvers[dropout]\n",
    "  train_accs.append(solver.train_acc_history[-1])\n",
    "  val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].train_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "  plt.plot(solvers[dropout].val_acc_history, 'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n",
    "我的结果显示，加了dropout的训练准确率有所下降，但是验证准确率上升，可以看到缩小了两者之间的gap， 所以说是dropout充当了regularization的作用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]减小每层隐藏层节点个数，会减小overfitting，如果这时还是overfitting的话，可以减小keep prob的p，如果已经underfiiting欠拟合的话可以适当增大keep prob的p\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}