notebooks/4.sentiment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第四课 情感分析\n",
    "\n",
    "褚则伟 zeweichu@gmail.com\n",
    "\n",
    "学习目标\n",
    "- 学习和训练文本分类模型\n",
    "- 学习torchtext的基本使用方法\n",
    "    - BucketIterator\n",
    "- 学习torch.nn的一些基本模型\n",
    "    - Conv2d\n",
    "\n",
    "本notebook参考了https://github.com/bentrevett/pytorch-sentiment-analysis\n",
    "\n",
    "在这份notebook中，我们会用PyTorch模型和TorchText再来做情感分析(检测一段文字的情感是正面的还是负面的)。我们会使用[IMDb 数据集](http://ai.stanford.edu/~amaas/data/sentiment/)，即电影评论。\n",
    "\n",
    "模型从简单到复杂，我们会依次构建：\n",
    "- Word Averaging模型\n",
    "- RNN/LSTM模型\n",
    "- CNN模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准备数据\n",
    "\n",
    "- TorchText中的一个重要概念是`Field`。`Field`决定了你的数据会被怎样处理。在我们的情感分类任务中，我们所需要接触到的数据有文本字符串和两种情感，\"pos\"或者\"neg\"。\n",
    "- `Field`的参数制定了数据会被怎样处理。\n",
    "- 我们使用`TEXT` field来定义如何处理电影评论，使用`LABEL` field来处理两个情感类别。\n",
    "- 我们的`TEXT` field带有`tokenize='spacy'`，这表示我们会用[spaCy](https://spacy.io) tokenizer来tokenize英文句子。如果我们不特别声明`tokenize`这个参数，那么默认的分词方法是使用空格。\n",
    "- 安装spaCy\n",
    "```\n",
    "pip install -U spacy\n",
    "python -m spacy download en\n",
    "```\n",
    "- `LABEL`由`LabelField`定义。这是一种特别的用来处理label的`Field`。我们后面会解释dtype。\n",
    "- 更多关于`Fields`，参见https://github.com/pytorch/text/blob/master/torchtext/data/field.py\n",
    "- 和之前一样，我们会设定random seeds使实验可以复现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchtext import data\n",
    "\n",
    "SEED = 1234\n",
    "\n",
    "torch.manual_seed(SEED)\n",
    "torch.cuda.manual_seed(SEED)\n",
    "torch.backends.cudnn.deterministic = True\n",
    "\n",
    "TEXT = data.Field(tokenize='spacy')\n",
    "LABEL = data.LabelField(dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- TorchText支持很多常见的自然语言处理数据集。\n",
    "- 下面的代码会自动下载IMDb数据集，然后分成train/test两个`torchtext.datasets`类别。数据被前面的`Fields`处理。IMDb数据集一共有50000电影评论，每个评论都被标注为正面的或负面的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from torchtext import datasets\n",
    "train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看每个数据split有多少条数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 25000\n",
      "Number of testing examples: 25000\n"
     ]
    }
   ],
   "source": [
    "print(f'Number of training examples: {len(train_data)}')\n",
    "print(f'Number of testing examples: {len(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看一个example。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'text': ['Brilliant', 'adaptation', 'of', 'the', 'novel', 'that', 'made', 'famous', 'the', 'relatives', 'of', 'Chilean', 'President', 'Salvador', 'Allende', 'killed', '.', 'In', 'the', 'environment', 'of', 'a', 'large', 'estate', 'that', 'arises', 'from', 'the', 'ruins', ',', 'becoming', 'a', 'force', 'to', 'abuse', 'and', 'exploitation', 'of', 'outrage', ',', 'a', 'luxury', 'estate', 'for', 'the', 'benefit', 'of', 'the', 'upstart', 'Esteban', 'Trueba', 'and', 'his', 'undeserved', 'family', ',', 'the', 'brilliant', 'Danish', 'director', 'Bille', 'August', 'recreates', ',', 'in', 'micro', ',', 'which', 'at', 'the', 'time', 'would', 'be', 'the', 'process', 'leading', 'to', 'the', 'greatest', 'infamy', 'of', 'his', 'story', 'to', 'the', 'hardened', 'Chilean', 'nation', ',', 'and', 'whose', 'main', 'character', 'would', 'Augusto', 'Pinochet', '(', 'Stephen', 'similarities', 'with', 'it', 'are', 'inevitable', ':', 'recall', ',', 'as', 'an', 'example', ',', 'that', 'image', 'of', 'the', 'senator', 'with', 'dark', 'glasses', 'that', 'makes', 'him', 'the', 'wink', 'to', 'the', 'general', 'to', 'begin', 'making', 'the', 'palace).<br', '/><br', '/>Bille', 'August', 'attends', 'an', 'exceptional', 'cast', 'in', 'the', 'Jeremy', 'protruding', 'Irons', ',', 'whose', 'character', 'changes', 'from', 'arrogance', 'and', 'extreme', 'cruelty', ',', 'the', 'hard', 'lesson', 'that', 'life', 'always', 'brings', 'us', 'to', 'almost', 'force', 'us', 'to', 'change', '.', 'In', 'Esteban', 'fully', 'applies', 'the', 'law', 'of', 'resonance', ',', 'with', 'great', 'wisdom', ',', 'Solomon', 'describes', 'in', 'these', 'words:\"The', 'things', 'that', 'freckles', 'are', 'the', 'same', 'punishment', 'that', 'will', 'serve', 'you', '.', '\"', '<', 'br', '/><br', '/>Unforgettable', 'Glenn', 'Close', 'playing', 'splint', ',', 'the', 'tainted', 'sister', 'of', 'Stephen', ',', 'whose', 'sin', ',', 'driven', 'by', 'loneliness', ',', 'spiritual', 'and', 'platonic', 'love', 'was', 'the', 'wife', 'of', 'his', 'cruel', 'snowy', 'brother', '.', 'Meryl', 'Streep', 'also', 'brilliant', ',', 'a', 'woman', 'whose', 'name', 'came', 'to', 'him', 'like', 'a', 'glove', 'Clara', '.', 'With', 'telekinetic', 'powers', ',', 'cognitive', 'and', 'mediumistic', ',', 'this', 'hardened', 'woman', ',', 'loyal', 'to', 'his', 'blunt', ',', 'conservative', 'husband', ',', 'is', 'an', 'indicator', 'of', 'character', 'and', 'self', '-', 'control', 'that', 'we', 'wish', 'for', 'ourselves', 'and', 'for', 'all', 'human', 'beings', '.', '<', 'br', '/><br', '/>Every', 'character', 'is', 'a', 'portrait', 'of', 'virtuosity', '(', 'as', 'Blanca', 'worthy', 'rebel', 'leader', 'Pedro', 'Segundo', 'unhappy', '...', ')', 'or', 'a', 'portrait', 'of', 'humiliation', ',', 'like', 'Stephen', 'Jr.', ',', 'the', 'bastard', 'child', 'of', 'Senator', ',', 'who', 'serves', 'as', 'an', 'instrument', 'for', 'the', 'return', 'of', 'the', 'boomerang', '.', '<', 'br', '/><br', '/>The', 'film', 'moves', 'the', 'bowels', ',', 'we', 'recreated', 'some', 'facts', 'that', 'should', 'not', 'ever', 'be', 'repeated', ',', 'but', 'that', 'absurdly', 'still', 'happen', '(', 'Colombia', 'is', 'a', 'sad', 'example', ')', 'and', 'another', 'reminder', 'that', ',', 'against', 'all', ',', 'life', 'is', 'wonderful', 'because', 'there', 'are', 'always', 'people', 'like', 'Isabel', 'Allende', 'and', 'immortalize', 'just', 'Bille', 'August', '.'], 'label': 'pos'}\n"
     ]
    }
   ],
   "source": [
    "print(vars(train_data.examples[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 由于我们现在只有train/test这两个分类，所以我们需要创建一个新的validation set。我们可以使用`.split()`创建新的分类。\n",
    "- 默认的数据分割是 70、30，如果我们声明`split_ratio`，可以改变split之间的比例，`split_ratio=0.8`表示80%的数据是训练集，20%是验证集。\n",
    "- 我们还声明`random_state`这个参数，确保我们每次分割的数据集都是一样的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import random\n",
    "train_data, valid_data = train_data.split(random_state=random.seed(SEED))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "检查一下现在每个部分有多少条数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training examples: 17500\n",
      "Number of validation examples: 7500\n",
      "Number of testing examples: 25000\n"
     ]
    }
   ],
   "source": [
    "print(f'Number of training examples: {len(train_data)}')\n",
    "print(f'Number of validation examples: {len(valid_data)}')\n",
    "print(f'Number of testing examples: {len(test_data)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 下一步我们需要创建 _vocabulary_ 。_vocabulary_ 就是把每个单词一一映射到一个数字。\n",
    "![](assets/sentiment5.png)\n",
    "- 我们使用最常见的25k个单词来构建我们的单词表，用`max_size`这个参数可以做到这一点。\n",
    "- 所有其他的单词都用`<unk>`来表示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TEXT.build_vocab(train_data, max_size=25000)\n",
    "# LABEL.build_vocab(train_data)\n",
    "TEXT.build_vocab(train_data, max_size=25000, vectors=\"glove.6B.100d\", unk_init=torch.Tensor.normal_)\n",
    "LABEL.build_vocab(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique tokens in TEXT vocabulary: 25002\n",
      "Unique tokens in LABEL vocabulary: 2\n"
     ]
    }
   ],
   "source": [
    "print(f\"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}\")\n",
    "print(f\"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 当我们把句子传进模型的时候，我们是按照一个个 _batch_ 穿进去的，也就是说，我们一次传入了好几个句子，而且每个batch中的句子必须是相同的长度。为了确保句子的长度相同，TorchText会把短的句子pad到和最长的句子等长。\n",
    "![](assets/sentiment6.png)\n",
    "- 下面我们来看看训练数据集中最常见的单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('the', 201455), (',', 192552), ('.', 164402), ('a', 108963), ('and', 108649), ('of', 100010), ('to', 92873), ('is', 76046), ('in', 60904), ('I', 54486), ('it', 53405), ('that', 49155), ('\"', 43890), (\"'s\", 43151), ('this', 42454), ('-', 36769), ('/><br', 35511), ('was', 34990), ('as', 30324), ('with', 29691)]\n"
     ]
    }
   ],
   "source": [
    "print(TEXT.vocab.freqs.most_common(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以直接用 `stoi`(**s**tring **to** **i**nt) 或者 `itos` (**i**nt **to**  **s**tring) 来查看我们的单词表。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['<unk>', '<pad>', 'the', ',', '.', 'a', 'and', 'of', 'to', 'is']\n"
     ]
    }
   ],
   "source": [
    "print(TEXT.vocab.itos[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看labels。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "defaultdict(<function _default_unk_index at 0x7fbec39a79d8>, {'neg': 0, 'pos': 1})\n"
     ]
    }
   ],
   "source": [
    "print(LABEL.vocab.stoi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 最后一步数据的准备是创建iterators。每个itartion都会返回一个batch的examples。\n",
    "- 我们会使用`BucketIterator`。`BucketIterator`会把长度差不多的句子放到同一个batch中，确保每个batch中不出现太多的padding。\n",
    "- 严格来说，我们这份notebook中的模型代码都有一个问题，也就是我们把`<pad>`也当做了模型的输入进行训练。更好的做法是在模型中把由`<pad>`产生的输出给消除掉。在这节课中我们简单处理，直接把`<pad>`也用作模型输入了。由于`<pad>`数量不多，模型的效果也不差。\n",
    "- 如果我们有GPU，还可以指定每个iteration返回的tensor都在GPU上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "BATCH_SIZE = 64\n",
    "\n",
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "\n",
    "train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(\n",
    "    (train_data, valid_data, test_data), \n",
    "    batch_size=BATCH_SIZE,\n",
    "    device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word Averaging模型\n",
    "\n",
    "- 我们首先介绍一个简单的Word Averaging模型。这个模型非常简单，我们把每个单词都通过`Embedding`层投射成word embedding vector，然后把一句话中的所有word vector做个平均，就是整个句子的vector表示了。接下来把这个sentence vector传入一个`Linear`层，做分类即可。\n",
    "\n",
    "![](assets/sentiment8.png)\n",
    "\n",
    "- 我们使用[`avg_pool2d`](https://pytorch.org/docs/stable/nn.html?highlight=avg_pool2d#torch.nn.functional.avg_pool2d)来做average pooling。我们的目标是把sentence length那个维度平均成1，然后保留embedding这个维度。\n",
    "\n",
    "![](assets/sentiment9.png)\n",
    "\n",
    "- `avg_pool2d`的kernel size是 (`embedded.shape[1]`, 1)，所以句子长度的那个维度会被压扁。\n",
    "\n",
    "![](assets/sentiment10.png)\n",
    "\n",
    "![](assets/sentiment11.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class WordAVGModel(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, output_dim, pad_idx):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)\n",
    "        self.fc = nn.Linear(embedding_dim, output_dim)\n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.embedding(text) # [sent len, batch size, emb dim]\n",
    "        embedded = embedded.permute(1, 0, 2) # [batch size, sent len, emb dim]\n",
    "        pooled = F.avg_pool2d(embedded, (embedded.shape[1], 1)).squeeze(1) # [batch size, embedding_dim]\n",
    "        return self.fc(pooled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "OUTPUT_DIM = 1\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = WordAVGModel(INPUT_DIM, EMBEDDING_DIM, OUTPUT_DIM, PAD_IDX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 2,500,301 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "def count_parameters(model):\n",
    "    return sum(p.numel() for p in model.parameters() if p.requires_grad)\n",
    "\n",
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],\n",
       "        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],\n",
       "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
       "        ...,\n",
       "        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],\n",
       "        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],\n",
       "        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pretrained_embeddings = TEXT.vocab.vectors\n",
    "model.embedding.weight.data.copy_(pretrained_embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch.optim as optim\n",
    "\n",
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "model = model.to(device)\n",
    "criterion = criterion.to(device)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "计算预测的准确率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def binary_accuracy(preds, y):\n",
    "    \"\"\"\n",
    "    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8\n",
    "    \"\"\"\n",
    "\n",
    "    #round predictions to the closest integer\n",
    "    rounded_preds = torch.round(torch.sigmoid(preds))\n",
    "    correct = (rounded_preds == y).float() #convert into float for division \n",
    "    acc = correct.sum()/len(correct)\n",
    "    return acc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train(model, iterator, optimizer, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    model.train()\n",
    "    \n",
    "    for batch in iterator:\n",
    "        optimizer.zero_grad()\n",
    "        predictions = model(batch.text).squeeze(1)\n",
    "        loss = criterion(predictions, batch.label)\n",
    "        acc = binary_accuracy(predictions, batch.label)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        epoch_loss += loss.item()\n",
    "        epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def evaluate(model, iterator, criterion):\n",
    "    \n",
    "    epoch_loss = 0\n",
    "    epoch_acc = 0\n",
    "    model.eval()\n",
    "    \n",
    "    with torch.no_grad():\n",
    "        for batch in iterator:\n",
    "            predictions = model(batch.text).squeeze(1)\n",
    "            loss = criterion(predictions, batch.label)\n",
    "            acc = binary_accuracy(predictions, batch.label)\n",
    "            epoch_loss += loss.item()\n",
    "            epoch_acc += acc.item()\n",
    "        \n",
    "    return epoch_loss / len(iterator), epoch_acc / len(iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "def epoch_time(start_time, end_time):\n",
    "    elapsed_time = end_time - start_time\n",
    "    elapsed_mins = int(elapsed_time / 60)\n",
    "    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))\n",
    "    return elapsed_mins, elapsed_secs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.685 | Train Acc: 56.84%\n",
      "\t Val. Loss: 0.622 |  Val. Acc: 71.09%\n",
      "Epoch: 02 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.642 | Train Acc: 71.31%\n",
      "\t Val. Loss: 0.510 |  Val. Acc: 75.48%\n",
      "Epoch: 03 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.573 | Train Acc: 78.31%\n",
      "\t Val. Loss: 0.449 |  Val. Acc: 79.52%\n",
      "Epoch: 04 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.503 | Train Acc: 82.78%\n",
      "\t Val. Loss: 0.419 |  Val. Acc: 82.72%\n",
      "Epoch: 05 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.440 | Train Acc: 85.84%\n",
      "\t Val. Loss: 0.408 |  Val. Acc: 84.75%\n",
      "Epoch: 06 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.389 | Train Acc: 87.59%\n",
      "\t Val. Loss: 0.413 |  Val. Acc: 86.02%\n",
      "Epoch: 07 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.352 | Train Acc: 88.85%\n",
      "\t Val. Loss: 0.425 |  Val. Acc: 86.92%\n",
      "Epoch: 08 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.320 | Train Acc: 89.93%\n",
      "\t Val. Loss: 0.440 |  Val. Acc: 87.54%\n",
      "Epoch: 09 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.294 | Train Acc: 90.74%\n",
      "\t Val. Loss: 0.456 |  Val. Acc: 88.09%\n",
      "Epoch: 10 | Epoch Time: 0m 2s\n",
      "\tTrain Loss: 0.274 | Train Acc: 91.27%\n",
      "\t Val. Loss: 0.468 |  Val. Acc: 88.49%\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 10\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'wordavg-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "\n",
    "def predict_sentiment(sentence):\n",
    "    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]\n",
    "    indexed = [TEXT.vocab.stoi[t] for t in tokenized]\n",
    "    tensor = torch.LongTensor(indexed).to(device)\n",
    "    tensor = tensor.unsqueeze(1)\n",
    "    prediction = torch.sigmoid(model(tensor))\n",
    "    return prediction.item()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5.568591932965664e-26"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(\"This film is terrible\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 142,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predict_sentiment(\"This film is great\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## RNN模型\n",
    "\n",
    "- 下面我们尝试把模型换成一个**recurrent neural network** (RNN)。RNN经常会被用来encode一个sequence\n",
    "$$h_t = \\text{RNN}(x_t, h_{t-1})$$\n",
    "- 我们使用最后一个hidden state $h_T$来表示整个句子。\n",
    "- 然后我们把$h_T$通过一个线性变换$f$，然后用来预测句子的情感。\n",
    "\n",
    "![](assets/sentiment1.png)\n",
    "\n",
    "![](assets/sentiment7.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class RNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, \n",
    "                 n_layers, bidirectional, dropout, pad_idx):\n",
    "        super().__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)\n",
    "        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers=n_layers, \n",
    "                           bidirectional=bidirectional, dropout=dropout)\n",
    "        self.fc = nn.Linear(hidden_dim*2, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, text):\n",
    "        embedded = self.dropout(self.embedding(text)) #[sent len, batch size, emb dim]\n",
    "        output, (hidden, cell) = self.rnn(embedded)\n",
    "        #output = [sent len, batch size, hid dim * num directions]\n",
    "        #hidden = [num layers * num directions, batch size, hid dim]\n",
    "        #cell = [num layers * num directions, batch size, hid dim]\n",
    "        \n",
    "        #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers\n",
    "        #and apply dropout\n",
    "        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1)) # [batch size, hid dim * num directions]\n",
    "        return self.fc(hidden.squeeze(0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "HIDDEN_DIM = 256\n",
    "OUTPUT_DIM = 1\n",
    "N_LAYERS = 2\n",
    "BIDIRECTIONAL = True\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, \n",
    "            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The model has 4,810,857 trainable parameters\n"
     ]
    }
   ],
   "source": [
    "print(f'The model has {count_parameters(model):,} trainable parameters')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
      "        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
      "        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],\n",
      "        ...,\n",
      "        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],\n",
      "        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],\n",
      "        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]],\n",
      "       device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "model.embedding.weight.data.copy_(pretrained_embeddings)\n",
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "\n",
    "print(model.embedding.weight.data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练RNN模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "optimizer = optim.Adam(model.parameters())\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 1m 29s\n",
      "\tTrain Loss: 0.676 | Train Acc: 57.69%\n",
      "\t Val. Loss: 0.694 |  Val. Acc: 53.40%\n",
      "Epoch: 02 | Epoch Time: 1m 29s\n",
      "\tTrain Loss: 0.641 | Train Acc: 63.77%\n",
      "\t Val. Loss: 0.744 |  Val. Acc: 49.22%\n",
      "Epoch: 03 | Epoch Time: 1m 29s\n",
      "\tTrain Loss: 0.618 | Train Acc: 65.77%\n",
      "\t Val. Loss: 0.534 |  Val. Acc: 73.72%\n",
      "Epoch: 04 | Epoch Time: 1m 30s\n",
      "\tTrain Loss: 0.634 | Train Acc: 63.79%\n",
      "\t Val. Loss: 0.619 |  Val. Acc: 66.85%\n",
      "Epoch: 05 | Epoch Time: 1m 29s\n",
      "\tTrain Loss: 0.448 | Train Acc: 79.19%\n",
      "\t Val. Loss: 0.340 |  Val. Acc: 86.63%\n"
     ]
    }
   ],
   "source": [
    "N_EPOCHS = 5\n",
    "best_valid_loss = float('inf')\n",
    "for epoch in range(N_EPOCHS):\n",
    "    start_time = time.time()\n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'lstm-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have noticed the loss is not really decreasing and the accuracy is poor. This is due to several issues with the model which we'll improve in the next notebook.\n",
    "\n",
    "Finally, the metric we actually care about, the test loss and accuracy, which we get from our parameters that gave us the best validation loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.load_state_dict(torch.load('lstm-model.pt'))\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CNN(nn.Module):\n",
    "    def __init__(self, vocab_size, embedding_dim, n_filters, \n",
    "                 filter_sizes, output_dim, dropout, pad_idx):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)\n",
    "        self.convs = nn.ModuleList([\n",
    "                                    nn.Conv2d(in_channels = 1, out_channels = n_filters, \n",
    "                                              kernel_size = (fs, embedding_dim)) \n",
    "                                    for fs in filter_sizes\n",
    "                                    ])\n",
    "        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "    def forward(self, text):\n",
    "        text = text.permute(1, 0) # [batch size, sent len]\n",
    "        embedded = self.embedding(text) # [batch size, sent len, emb dim]\n",
    "        embedded = embedded.unsqueeze(1) # [batch size, 1, sent len, emb dim]\n",
    "        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]\n",
    "            \n",
    "        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]\n",
    "        \n",
    "        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]\n",
    "        \n",
    "        #pooled_n = [batch size, n_filters]\n",
    "        \n",
    "        cat = self.dropout(torch.cat(pooled, dim=1))\n",
    "\n",
    "        #cat = [batch size, n_filters * len(filter_sizes)]\n",
    "            \n",
    "        return self.fc(cat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "INPUT_DIM = len(TEXT.vocab)\n",
    "EMBEDDING_DIM = 100\n",
    "N_FILTERS = 100\n",
    "FILTER_SIZES = [3,4,5]\n",
    "OUTPUT_DIM = 1\n",
    "DROPOUT = 0.5\n",
    "PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]\n",
    "\n",
    "\n",
    "model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)\n",
    "model.embedding.weight.data.copy_(pretrained_embeddings)\n",
    "UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]\n",
    "\n",
    "model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)\n",
    "model = model.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 01 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.645 | Train Acc: 62.12%\n",
      "\t Val. Loss: 0.485 |  Val. Acc: 79.61%\n",
      "Epoch: 02 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.423 | Train Acc: 80.59%\n",
      "\t Val. Loss: 0.360 |  Val. Acc: 84.63%\n",
      "Epoch: 03 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.302 | Train Acc: 87.33%\n",
      "\t Val. Loss: 0.320 |  Val. Acc: 86.59%\n",
      "Epoch: 04 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.222 | Train Acc: 91.20%\n",
      "\t Val. Loss: 0.306 |  Val. Acc: 87.17%\n",
      "Epoch: 05 | Epoch Time: 0m 11s\n",
      "\tTrain Loss: 0.161 | Train Acc: 93.99%\n",
      "\t Val. Loss: 0.325 |  Val. Acc: 86.82%\n"
     ]
    }
   ],
   "source": [
    "optimizer = optim.Adam(model.parameters())\n",
    "criterion = nn.BCEWithLogitsLoss()\n",
    "criterion = criterion.to(device)\n",
    "\n",
    "N_EPOCHS = 5\n",
    "\n",
    "best_valid_loss = float('inf')\n",
    "\n",
    "for epoch in range(N_EPOCHS):\n",
    "\n",
    "    start_time = time.time()\n",
    "    \n",
    "    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)\n",
    "    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)\n",
    "    \n",
    "    end_time = time.time()\n",
    "\n",
    "    epoch_mins, epoch_secs = epoch_time(start_time, end_time)\n",
    "    \n",
    "    if valid_loss < best_valid_loss:\n",
    "        best_valid_loss = valid_loss\n",
    "        torch.save(model.state_dict(), 'CNN-model.pt')\n",
    "    \n",
    "    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')\n",
    "    print(f'\\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')\n",
    "    print(f'\\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Loss: 0.336 | Test Acc: 85.66%\n"
     ]
    }
   ],
   "source": [
    "model.load_state_dict(torch.load('CNN-model.pt'))\n",
    "test_loss, test_acc = evaluate(model, test_iterator, criterion)\n",
    "print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}