evaluate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "86fe17dc-eaca-49ae-81b8-8c372bff4fb8",
   "metadata": {},
   "source": [
    "# Custom Evaluation with LlamaIndex"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13376312-ee85-470c-bb8a-ec13d6c38b2f",
   "metadata": {},
   "source": [
    "In this notebook, we evaluate 3 different embedding models: \n",
    "1. proprietary OpenAI embedding,\n",
    "2. open source `BAAI/bge-small-en`, and\n",
    "3. our finetuned embedding model.\n",
    "\n",
    "We consider 2 evaluation approaches:\n",
    "1. a simple custom **hit rate** metric\n",
    "2. using `InformationRetrievalEvaluator` from sentence_transformers\n",
    "\n",
    "We show that finetuning on synthetic (LLM-generated) dataset significantly improve upon an opensource embedding model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "id": "3f249133-d7ca-42e8-ad41-c5ef8fe5849b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from tqdm.notebook import tqdm\n",
    "import pandas as pd\n",
    "\n",
    "from llama_index import ServiceContext, VectorStoreIndex\n",
    "from llama_index.schema import TextNode\n",
    "from llama_index.embeddings import OpenAIEmbedding"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355b4619-4a01-427e-97f3-ec7cac6d2bfd",
   "metadata": {},
   "source": [
    "### Load data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1591bd3a-8be4-4596-acb8-d9d007239c32",
   "metadata": {},
   "source": [
    "First, let's load the synthetic dataset we automatically generated from our corpus (without having access to any labellers)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "id": "e73cb5c7-a71c-4e70-ba24-ce196e7177af",
   "metadata": {},
   "outputs": [],
   "source": [
    "TRAIN_DATASET_FPATH = './data/train_dataset.json'\n",
    "VAL_DATASET_FPATH = './data/val_dataset.json'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "f5a85bae-0f40-4d24-b96f-a5f884893310",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(TRAIN_DATASET_FPATH, 'r+') as f:\n",
    "    train_dataset = json.load(f)\n",
    "\n",
    "with open(VAL_DATASET_FPATH, 'r+') as f:\n",
    "    val_dataset = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8232ec0e-40bf-4910-bdad-cca9d7de0619",
   "metadata": {},
   "source": [
    "### Define eval function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25744f5d-cd89-4f63-b1c9-238519b74bc8",
   "metadata": {},
   "source": [
    "**Option 1**: We use a simple **hit rate** metric for evaluation:\n",
    "* for each (query, relevant_doc) pair,\n",
    "* we retrieve top-k documents with the query,  and \n",
    "* it's a **hit** if the results contain the relevant_doc.\n",
    "\n",
    "This approach is very simple and intuitive, and we can apply it to both the proprietary OpenAI embedding as well as our open source and fine-tuned embedding models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "75e79771-fe51-44a0-807f-ee9697064ef6",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(\n",
    "    dataset,\n",
    "    embed_model,\n",
    "    top_k=5,\n",
    "    verbose=False,\n",
    "):\n",
    "    corpus = dataset['corpus']\n",
    "    queries = dataset['queries']\n",
    "    relevant_docs = dataset['relevant_docs']\n",
    "\n",
    "    service_context = ServiceContext.from_defaults(embed_model=embed_model)\n",
    "    nodes = [TextNode(id_=id_, text=text) for id_, text in corpus.items()] \n",
    "    index = VectorStoreIndex(\n",
    "        nodes, \n",
    "        service_context=service_context, \n",
    "        show_progress=True\n",
    "    )\n",
    "    retriever = index.as_retriever(similarity_top_k=top_k)\n",
    "\n",
    "    eval_results = []\n",
    "    for query_id, query in tqdm(queries.items()):\n",
    "        retrieved_nodes = retriever.retrieve(query)\n",
    "        retrieved_ids = [node.node.node_id for node in retrieved_nodes]\n",
    "        expected_id = relevant_docs[query_id][0]\n",
    "        is_hit = expected_id in retrieved_ids  # assume 1 relevant doc\n",
    "        \n",
    "        eval_result = {\n",
    "            'is_hit': is_hit,\n",
    "            'retrieved': retrieved_ids,\n",
    "            'expected': expected_id,\n",
    "            'query': query_id,\n",
    "        }\n",
    "        eval_results.append(eval_result)\n",
    "    return eval_results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b282f2b8-3486-4e11-b314-4c2c52d2ea02",
   "metadata": {},
   "source": [
    "**Option 2**: We use the `InformationRetrievalEvaluator` from sentence_transformers.\n",
    "\n",
    "This provides a more comprehensive suite of metrics, but we can only run it against the sentencetransformers compatible models (open source and our finetuned model, *not* the OpenAI embedding model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "id": "8bce34cd-d51d-44c4-900b-9bded5e1ec70",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers.evaluation import InformationRetrievalEvaluator\n",
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "def evaluate_st(\n",
    "    dataset,\n",
    "    model_id,\n",
    "    name,\n",
    "):\n",
    "    corpus = dataset['corpus']\n",
    "    queries = dataset['queries']\n",
    "    relevant_docs = dataset['relevant_docs']\n",
    "\n",
    "    evaluator = InformationRetrievalEvaluator(queries, corpus, relevant_docs, name=name)\n",
    "    model = SentenceTransformer(model_id)\n",
    "    return evaluator(model, output_path='results/')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0871d4-5a1a-4e5d-97d2-3fa9ae68af54",
   "metadata": {},
   "source": [
    "### Run Evals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62183ec-ac63-4efc-9b9d-9a4c091bee22",
   "metadata": {},
   "source": [
    "#### OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94922539-197e-459e-b5c0-16a515f2f100",
   "metadata": {},
   "source": [
    "Note: this might take a few minutes to run since we have to embed the corpus and queries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "4aa63f1f-0d8d-4533-bc40-d7791c523bf7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "faf17aad3b474cce97166a9ba83f72a5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a356dca7b93b4b77b366b5b0020d7e52",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/790 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ada = OpenAIEmbedding()\n",
    "ada_val_results = evaluate(val_dataset, ada)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "3cebfdf9-4d7b-417a-95df-01fcc5ac6589",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ada = pd.DataFrame(ada_val_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "20980c2f-c4ca-44f3-9c96-b5cef61c4f2c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8708860759493671"
      ]
     },
     "execution_count": 144,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hit_rate_ada = df_ada['is_hit'].mean()\n",
    "hit_rate_ada"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b63f51d-3bf9-4360-8ff8-fd2593ea7eb5",
   "metadata": {},
   "source": [
    "### BAAI/bge-small-en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "7d33e9fe-cddf-44f2-a300-ebe5dbf1dec8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a154480bdfed4dd1abf8b97f0935f610",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1ea9f45195414ea19017cba3fa87418d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/790 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bge = \"local:BAAI/bge-small-en\"\n",
    "bge_val_results = evaluate(val_dataset, bge)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "8e633ef4-4681-45bf-9712-ef6abbc1769d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_bge = pd.DataFrame(bge_val_results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "8fb28364-ffa3-49af-b140-ffc8430756a4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7873417721518987"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hit_rate_bge = df_bge['is_hit'].mean()\n",
    "hit_rate_bge"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "id": "f18ee8ee-6437-4c69-aa00-153c81b798ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6154761680813499"
      ]
     },
     "execution_count": 195,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_st(val_dataset, \"BAAI/bge-small-en\", name='bge')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79c85066-27bd-4376-8f7b-b6cda3a99371",
   "metadata": {},
   "source": [
    "### Finetuned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "id": "cfb73b2c-5d69-42d5-9bba-bf5920e86b14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "74d5dd4899f44e9c9648f7bee4bc6803",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating embeddings:   0%|          | 0/395 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "26f14361f95e44549dfb21289ea1616b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/790 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "finetuned = \"local:exp_finetune\"\n",
    "val_results_finetuned = evaluate(val_dataset, finetuned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "c5861d26-ee50-47cb-9a34-3d9295a99754",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_finetuned = pd.DataFrame(val_results_finetuned)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "id": "1ee59dda-b96b-44a8-836d-63970c2c4957",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8443037974683544"
      ]
     },
     "execution_count": 152,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hit_rate_finetuned = df_finetuned['is_hit'].mean()\n",
    "hit_rate_finetuned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "id": "2ba92a7d-414c-4f25-9caa-f29922a25c9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7127630344044761"
      ]
     },
     "execution_count": 196,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "evaluate_st(val_dataset, \"exp_finetune\", name='finetuned')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2debfccd-542b-42b7-a603-4b4272435130",
   "metadata": {},
   "source": [
    "### Summary of Results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76700270-cce3-4def-b545-bfefe94222b1",
   "metadata": {},
   "source": [
    "#### Hit rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 170,
   "id": "1c66604a-a844-4efe-9d64-9aaccad2fbbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ada['model'] = 'ada'\n",
    "df_bge['model'] = 'bge'\n",
    "df_finetuned['model'] = 'fine_tuned'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b7c04d3-1f29-4a91-a52d-561f4f498e7c",
   "metadata": {},
   "source": [
    "We can see that fine-tuning our small open-source embedding model drastically improve its retrieval quality (even approaching the quality of the proprietary OpenAI embedding)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 171,
   "id": "c2d338ec-b130-4467-b6be-4a6bad60e205",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>is_hit</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ada</th>\n",
       "      <td>0.870886</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>bge</th>\n",
       "      <td>0.787342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fine_tuned</th>\n",
       "      <td>0.844304</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              is_hit\n",
       "model               \n",
       "ada         0.870886\n",
       "bge         0.787342\n",
       "fine_tuned  0.844304"
      ]
     },
     "execution_count": 171,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all = pd.concat([df_ada, df_bge, df_finetuned])\n",
    "df_all.groupby('model').mean('is_hit')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f87314-9790-40d7-9c72-ea1ccb4b789b",
   "metadata": {},
   "source": [
    "### InformationRetrievalEvaluator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "id": "19a777a0-a9c1-4d5c-9da6-fde91b5bb712",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_st_bge = pd.read_csv('results/Information-Retrieval_evaluation_bge_results.csv')\n",
    "df_st_finetuned = pd.read_csv('results/Information-Retrieval_evaluation_finetuned_results.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f2bf70c-f3c5-4c68-b741-946e08d5567b",
   "metadata": {},
   "source": [
    "We can see that embedding finetuning improves metrics consistently across the suite of eval metrics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "id": "2d30e52e-38e3-4164-9c31-26f47ac811db",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epoch</th>\n",
       "      <th>steps</th>\n",
       "      <th>cos_sim-Accuracy@1</th>\n",
       "      <th>cos_sim-Accuracy@3</th>\n",
       "      <th>cos_sim-Accuracy@5</th>\n",
       "      <th>cos_sim-Accuracy@10</th>\n",
       "      <th>cos_sim-Precision@1</th>\n",
       "      <th>cos_sim-Recall@1</th>\n",
       "      <th>cos_sim-Precision@3</th>\n",
       "      <th>cos_sim-Recall@3</th>\n",
       "      <th>...</th>\n",
       "      <th>dot_score-Recall@1</th>\n",
       "      <th>dot_score-Precision@3</th>\n",
       "      <th>dot_score-Recall@3</th>\n",
       "      <th>dot_score-Precision@5</th>\n",
       "      <th>dot_score-Recall@5</th>\n",
       "      <th>dot_score-Precision@10</th>\n",
       "      <th>dot_score-Recall@10</th>\n",
       "      <th>dot_score-MRR@10</th>\n",
       "      <th>dot_score-NDCG@10</th>\n",
       "      <th>dot_score-MAP@100</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>model</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>bge</th>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.508861</td>\n",
       "      <td>0.686076</td>\n",
       "      <td>0.729114</td>\n",
       "      <td>0.813924</td>\n",
       "      <td>0.508861</td>\n",
       "      <td>0.508861</td>\n",
       "      <td>0.228692</td>\n",
       "      <td>0.686076</td>\n",
       "      <td>...</td>\n",
       "      <td>0.502532</td>\n",
       "      <td>0.227426</td>\n",
       "      <td>0.682278</td>\n",
       "      <td>0.146835</td>\n",
       "      <td>0.734177</td>\n",
       "      <td>0.080886</td>\n",
       "      <td>0.808861</td>\n",
       "      <td>0.605208</td>\n",
       "      <td>0.654535</td>\n",
       "      <td>0.611977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fine_tuned</th>\n",
       "      <td>-1</td>\n",
       "      <td>-1</td>\n",
       "      <td>0.607595</td>\n",
       "      <td>0.786076</td>\n",
       "      <td>0.848101</td>\n",
       "      <td>0.894937</td>\n",
       "      <td>0.607595</td>\n",
       "      <td>0.607595</td>\n",
       "      <td>0.262025</td>\n",
       "      <td>0.786076</td>\n",
       "      <td>...</td>\n",
       "      <td>0.601266</td>\n",
       "      <td>0.262447</td>\n",
       "      <td>0.787342</td>\n",
       "      <td>0.169367</td>\n",
       "      <td>0.846835</td>\n",
       "      <td>0.089367</td>\n",
       "      <td>0.893671</td>\n",
       "      <td>0.704729</td>\n",
       "      <td>0.750924</td>\n",
       "      <td>0.709135</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>2 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            epoch  steps  cos_sim-Accuracy@1  cos_sim-Accuracy@3  \\\n",
       "model                                                              \n",
       "bge            -1     -1            0.508861            0.686076   \n",
       "fine_tuned     -1     -1            0.607595            0.786076   \n",
       "\n",
       "            cos_sim-Accuracy@5  cos_sim-Accuracy@10  cos_sim-Precision@1  \\\n",
       "model                                                                      \n",
       "bge                   0.729114             0.813924             0.508861   \n",
       "fine_tuned            0.848101             0.894937             0.607595   \n",
       "\n",
       "            cos_sim-Recall@1  cos_sim-Precision@3  cos_sim-Recall@3  ...  \\\n",
       "model                                                                ...   \n",
       "bge                 0.508861             0.228692          0.686076  ...   \n",
       "fine_tuned          0.607595             0.262025          0.786076  ...   \n",
       "\n",
       "            dot_score-Recall@1  dot_score-Precision@3  dot_score-Recall@3  \\\n",
       "model                                                                       \n",
       "bge                   0.502532               0.227426            0.682278   \n",
       "fine_tuned            0.601266               0.262447            0.787342   \n",
       "\n",
       "            dot_score-Precision@5  dot_score-Recall@5  dot_score-Precision@10  \\\n",
       "model                                                                           \n",
       "bge                      0.146835            0.734177                0.080886   \n",
       "fine_tuned               0.169367            0.846835                0.089367   \n",
       "\n",
       "            dot_score-Recall@10  dot_score-MRR@10  dot_score-NDCG@10  \\\n",
       "model                                                                  \n",
       "bge                    0.808861          0.605208           0.654535   \n",
       "fine_tuned             0.893671          0.704729           0.750924   \n",
       "\n",
       "            dot_score-MAP@100  \n",
       "model                          \n",
       "bge                  0.611977  \n",
       "fine_tuned           0.709135  \n",
       "\n",
       "[2 rows x 32 columns]"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_st_bge['model'] = 'bge'\n",
    "df_st_finetuned['model'] = 'fine_tuned'\n",
    "df_st_all = pd.concat([df_st_bge, df_st_finetuned])\n",
    "df_st_all = df_st_all.set_index('model')\n",
    "df_st_all"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}