lenskit/lkpy

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Started\n",
    "\n",
    "This notebook gets you started with a brief nDCG evaluation with LensKit for Python.\n",
    "\n",
    "This notebook is also available on [Google Collaboratory](https://colab.research.google.com/drive/1ym040cKkQf85epu80VtIkMXy3LpfYQky?usp=sharing) and [nbviewer](https://nbviewer.jupyter.org/github/lenskit/lkpy/blob/master/doc/GettingStarted.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "We first import the LensKit components we need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from lenskit import batch, topn, util\n",
    "from lenskit import crossfold as xf\n",
    "from lenskit.algorithms import Recommender, als, knn\n",
    "from lenskit.data import from_interactions_df\n",
    "from lenskit.data.movielens import load_movielens_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And Pandas is very useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pyprojroot` package makes it easy to find input data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyprojroot.here import here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "We're going to use the ML-100K data set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user</th>\n",
       "      <th>item</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3.0</td>\n",
       "      <td>881250949</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3.0</td>\n",
       "      <td>891717742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1.0</td>\n",
       "      <td>878887116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2.0</td>\n",
       "      <td>880606923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1.0</td>\n",
       "      <td>886397596</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user  item  rating  timestamp\n",
       "0   196   242     3.0  881250949\n",
       "1   186   302     3.0  891717742\n",
       "2    22   377     1.0  878887116\n",
       "3   244    51     2.0  880606923\n",
       "4   166   346     1.0  886397596"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ml100k = load_movielens_df(here('data/ml-100k.zip'))\n",
    "ml100k.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Defining Algorithms\n",
    "\n",
    "Let's set up two algorithms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "algo_ii = knn.ItemItem(20)\n",
    "algo_als = als.BiasedMF(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running the Evaluation\n",
    "\n",
    "In LensKit, our evaluation proceeds in 2 steps:\n",
    "\n",
    "1. Generate recommendations\n",
    "2. Measure them\n",
    "\n",
    "If memory is a concern, we can measure while generating, but we will not do that for now.\n",
    "\n",
    "We will first define a function to generate recommendations from one algorithm over a single partition of the data set.  It will take an algorithm, a train set, and a test set, and return the recommendations.\n",
    "\n",
    "**Note:** before fitting the algorithm, we clone it.  Some algorithms misbehave when fit multiple times.\n",
    "\n",
    "**Note 2:** our algorithms do not necessarily implement the `Recommender` interface, so we adapt them. This fills in a default candidate selector.\n",
    "\n",
    "The code function looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def eval(aname, algo, train, test):\n",
    "    fittable = util.clone(algo)\n",
    "    fittable = Recommender.adapt(fittable)\n",
    "    fittable.fit(from_interactions_df(train))\n",
    "    users = test.user.unique()\n",
    "    # now we run the recommender\n",
    "    recs = batch.recommend(fittable, users, 100)\n",
    "    # add the algorithm name for analyzability\n",
    "    recs['Algorithm'] = aname\n",
    "    return recs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will loop over the data and the algorithms, and generate recommendations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/mde48/LensKit/lkpy/lenskit/lenskit/data/dataset.py:628: UserWarning: Sparse CSR tensor support is in beta state. If you miss a functionality in the sparse tensor support, please submit a feature request to https://github.com/pytorch/pytorch/issues. (Triggered internally at /Users/runner/miniforge3/conda-bld/libtorch_1719361060788/work/aten/src/ATen/SparseCsrTensorImpl.cpp:55.)\n",
      "  return torch.sparse_csr_tensor(\n"
     ]
    }
   ],
   "source": [
    "all_recs = []\n",
    "test_data = []\n",
    "for train, test in xf.partition_users(ml100k[['user', 'item', 'rating']], 5, xf.SampleFrac(0.2)):\n",
    "    test_data.append(test)\n",
    "    all_recs.append(eval('ItemItem', algo_ii, train, test))\n",
    "    all_recs.append(eval('ALS', algo_als, train, test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the results in place, we can concatenate them into a single data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>item</th>\n",
       "      <th>score</th>\n",
       "      <th>user</th>\n",
       "      <th>rank</th>\n",
       "      <th>Algorithm</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1449</td>\n",
       "      <td>4.994975</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1398</td>\n",
       "      <td>4.866851</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>511</td>\n",
       "      <td>4.845399</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1512</td>\n",
       "      <td>4.805413</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1594</td>\n",
       "      <td>4.788468</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>ItemItem</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   item     score  user  rank Algorithm\n",
       "0  1449  4.994975     2     1  ItemItem\n",
       "1  1398  4.866851     2     2  ItemItem\n",
       "2   511  4.845399     2     3  ItemItem\n",
       "3  1512  4.805413     2     4  ItemItem\n",
       "4  1594  4.788468     2     5  ItemItem"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_recs = pd.concat(all_recs, ignore_index=True)\n",
    "all_recs.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute our analysis, we also need to concatenate the test data into a single frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data = pd.concat(test_data, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We analyze our recommendation lists with a `RecListAnalysis`.  It takes care of the hard work of making sure that the truth data (our test data) and the recoommendations line up properly.\n",
    "\n",
    "We do assume here that each user only appears once per algorithm. Since our crossfold method partitions users, this is fine."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>nrecs</th>\n",
       "      <th>ndcg</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Algorithm</th>\n",
       "      <th>user</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">ItemItem</th>\n",
       "      <th>2</th>\n",
       "      <td>100</td>\n",
       "      <td>0.081186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>100</td>\n",
       "      <td>0.288946</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100</td>\n",
       "      <td>0.082112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>100</td>\n",
       "      <td>0.364167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>100</td>\n",
       "      <td>0.182636</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                nrecs      ndcg\n",
       "Algorithm user                 \n",
       "ItemItem  2       100  0.081186\n",
       "          6       100  0.288946\n",
       "          8       100  0.082112\n",
       "          10      100  0.364167\n",
       "          14      100  0.182636"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rla = topn.RecListAnalysis()\n",
    "rla.add_metric(topn.ndcg)\n",
    "results = rla.compute(all_recs, test_data)\n",
    "results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have nDCG values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Algorithm\n",
       "ALS         0.132649\n",
       "ItemItem    0.096963\n",
       "Name: ndcg, dtype: float64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.groupby('Algorithm').ndcg.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='Algorithm'>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiwAAAHhCAYAAABN6eUeAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAAArW0lEQVR4nO3df1BV953/8dcV+RETwd+gFhHcVnGJpl62Lhgc0xqMZmPcaiS/sBM1XTI2CtRdf6CbxDSSqnGpiUBVSGqmUdpiJ8lKVKzR0cDEStB0Ims2GxXXXMZAEvBHBLyc7x+O97s3F4wXDecDPB8zZ8b7Oe9z7vs4XnnxOT+uw7IsSwAAAAbrYXcDAAAA34bAAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgvJ52N3CrtLS06LPPPlPv3r3lcDjsbgcAANwAy7J0/vx5DRkyRD16tD2P0mUCy2effabIyEi72wAAAO1w5swZfe9732tzfZcJLL1795Z09YBDQ0Nt7gYAANyIhoYGRUZGen6Ot6XLBJZrp4FCQ0MJLAAAdDLfdjkHF90CAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjNfT7gZw84Yv3Wl3C+hAp1683+4WAKDDMcMCAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB47Qosubm5io6OVkhIiJxOpw4ePNhmrcvl0qOPPqqRI0eqR48eSk9P96nZvHmzkpKS1LdvX/Xt21eTJ0/W4cOH29MaAADogvwOLEVFRUpPT1dWVpYqKyuVlJSkqVOnqrq6utX6xsZGDRw4UFlZWRo7dmyrNfv379cjjzyid999V+Xl5Ro2bJiSk5N19uxZf9sDAABdkMOyLMufDcaPH69x48YpLy/PMxYbG6sZM2YoOzv7uttOmjRJd911l3Jycq5b53a71bdvX73yyiuaM2dOqzWNjY1qbGz0vG5oaFBkZKTq6+sVGhp64wfUBQxfutPuFtCBTr14v90tAMAt09DQoLCwsG/9+e3XDEtTU5MqKiqUnJzsNZ6cnKyysrL2ddqKS5cuqbm5Wf369WuzJjs7W2FhYZ4lMjLylr0/AAAwi1+Bpba2Vm63W+Hh4V7j4eHhqqmpuWVNLV26VEOHDtXkyZPbrFm2bJnq6+s9y5kzZ27Z+wMAALP0bM9GDofD67VlWT5j7bVmzRpt27ZN+/fvV0hISJt1wcHBCg4OviXvCQAAzOZXYBkwYIACAgJ8ZlPOnTvnM+vSHuvWrdPq1au1d+9ejRkz5qb3BwAAuga/TgkFBQXJ6XSqtLTUa7y0tFSJiYk31cjatWv1/PPPa9euXYqPj7+pfQEAgK7F71NCmZmZSk1NVXx8vBISErRp0yZVV1crLS1N0tVrS86ePautW7d6tjl69Kgk6cKFC/r888919OhRBQUFafTo0ZKungZauXKl3njjDQ0fPtwzg3PHHXfojjvuuNljBAAAnZzfgSUlJUV1dXVatWqVXC6X4uLiVFJSoqioKElXHxT3zWey/PCHP/T8uaKiQm+88YaioqJ06tQpSVcfRNfU1KRZs2Z5bffMM8/o2Wef9bdFAADQxfj9HBZT3eh93F0Rz2HpXngOC4Cu5Dt5DgsAAIAdCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeO0KLLm5uYqOjlZISIicTqcOHjzYZq3L5dKjjz6qkSNHqkePHkpPT2+1rri4WKNHj1ZwcLBGjx6tP//5z+1pDQAAdEF+B5aioiKlp6crKytLlZWVSkpK0tSpU1VdXd1qfWNjowYOHKisrCyNHTu21Zry8nKlpKQoNTVVx44dU2pqqmbPnq3333/f3/YAAEAX5LAsy/Jng/Hjx2vcuHHKy8vzjMXGxmrGjBnKzs6+7raTJk3SXXfdpZycHK/xlJQUNTQ06J133vGM3Xffferbt6+2bdvW6r4aGxvV2Njoed3Q0KDIyEjV19crNDTUn0Pq9IYv3Wl3C+hAp1683+4WAOCWaWhoUFhY2Lf+/PZrhqWpqUkVFRVKTk72Gk9OTlZZWVn7OtXVGZZv7nPKlCnX3Wd2drbCwsI8S2RkZLvfHwAAmM2vwFJbWyu3263w8HCv8fDwcNXU1LS7iZqaGr/3uWzZMtXX13uWM2fOtPv9AQCA2Xq2ZyOHw+H12rIsn7Hvep/BwcEKDg6+qfcEAACdg18zLAMGDFBAQIDPzMe5c+d8Zkj8ERERccv3CQAAug6/AktQUJCcTqdKS0u9xktLS5WYmNjuJhISEnz2uWfPnpvaJwAA6Dr8PiWUmZmp1NRUxcfHKyEhQZs2bVJ1dbXS0tIkXb225OzZs9q6datnm6NHj0qSLly4oM8//1xHjx5VUFCQRo8eLUlatGiRJk6cqF//+td68MEH9eabb2rv3r06dOjQLThEAADQ2fkdWFJSUlRXV6dVq1bJ5XIpLi5OJSUlioqKknT1QXHffCbLD3/4Q8+fKyoq9MYbbygqKkqnTp2SJCUmJmr79u1asWKFVq5cqREjRqioqEjjx4+/iUMDAABdhd/PYTHVjd7H3RXxHJbuheewAOhKbvTnd7vuEgIAdAx+Iele+IWkbXz5IQAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADBeuwJLbm6uoqOjFRISIqfTqYMHD163/sCBA3I6nQoJCVFMTIzy8/N9anJycjRy5EjddtttioyMVEZGhi5fvtye9gAAQBfjd2ApKipSenq6srKyVFlZqaSkJE2dOlXV1dWt1p88eVLTpk1TUlKSKisrtXz5ci1cuFDFxcWemt///vdaunSpnnnmGVVVVamgoEBFRUVatmxZ+48MAAB0GT393WD9+vWaN2+e5s+fL+nqzMju3buVl5en7Oxsn/r8/HwNGzZMOTk5kqTY2FgdOXJE69at08yZMyVJ5eXlmjBhgh599FFJ0vDhw/XII4/o8OHDbfbR2NioxsZGz+uGhgZ/DwUAAHQSfs2wNDU1qaKiQsnJyV7jycnJKisra3Wb8vJyn/opU6boyJEjam5uliTdfffdqqio8ASUTz/9VCUlJbr//vvb7CU7O1thYWGeJTIy0p9DAQAAnYhfgaW2tlZut1vh4eFe4+Hh4aqpqWl1m5qamlbrr1y5otraWknSww8/rOeff1533323AgMDNWLECN1zzz1aunRpm70sW7ZM9fX1nuXMmTP+HAoAAOhE/D4lJEkOh8PrtWVZPmPfVv9/x/fv368XXnhBubm5Gj9+vD755BMtWrRIgwcP1sqVK1vdZ3BwsIKDg9vTPgAA6GT8CiwDBgxQQECAz2zKuXPnfGZRromIiGi1vmfPnurfv78kaeXKlUpNTfVcF3PnnXfq4sWL+vnPf66srCz16MHd1wAAdGd+JYGgoCA5nU6VlpZ6jZeWlioxMbHVbRISEnzq9+zZo/j4eAUGBkqSLl265BNKAgICZFmWZzYGAAB0X35PXWRmZmrLli0qLCxUVVWVMjIyVF1drbS0NElXry2ZM2eOpz4tLU2nT59WZmamqqqqVFhYqIKCAi1evNhT88ADDygvL0/bt2/XyZMnVVpaqpUrV2r69OkKCAi4BYcJAAA6M7+vYUlJSVFdXZ1WrVoll8uluLg4lZSUKCoqSpLkcrm8nskSHR2tkpISZWRkaOPGjRoyZIg2bNjguaVZklasWCGHw6EVK1bo7NmzGjhwoB544AG98MILt+AQAQBAZ+ewusg5l4aGBoWFham+vl6hoaF2t9Ohhi/daXcL6ECnXmz7dn90PXy+u5fu+Pm+0Z/fXM0KAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxmtXYMnNzVV0dLRCQkLkdDp18ODB69YfOHBATqdTISEhiomJUX5+vk/NV199pQULFmjw4MEKCQlRbGysSkpK2tMeAADoYvwOLEVFRUpPT1dWVpYqKyuVlJSkqVOnqrq6utX6kydPatq0aUpKSlJlZaWWL1+uhQsXqri42FPT1NSke++9V6dOndKf/vQnnThxQps3b9bQoUPbf2QAAKDL6OnvBuvXr9e8efM0f/58SVJOTo52796tvLw8ZWdn+9Tn5+dr2LBhysnJkSTFxsbqyJEjWrdunWbOnClJKiws1BdffKGysjIFBgZKkqKiotp7TAAAoIvxa4alqalJFRUVSk5O9hpPTk5WWVlZq9uUl5f71E+ZMkVHjhxRc3OzJOmtt95SQkKCFixYoPDwcMXFxWn16tVyu91t9tLY2KiGhgavBQAAdE1+BZba2lq53W6Fh4d7jYeHh6umpqbVbWpqalqtv3LlimprayVJn376qf70pz/J7XarpKREK1as0EsvvaQXXnihzV6ys7MVFhbmWSIjI/05FAAA0Im066Jbh8Ph9dqyLJ+xb6v/v+MtLS0aNGiQNm3aJKfTqYcfflhZWVnKy8trc5/Lli1TfX29Zzlz5kx7DgUAAHQCfl3DMmDAAAUEBPjMppw7d85nFuWaiIiIVut79uyp/v37S5IGDx6swMBABQQEeGpiY2NVU1OjpqYmBQUF+ew3ODhYwcHB/rQPAAA6Kb9mWIKCguR0OlVaWuo1XlpaqsTExFa3SUhI8Knfs2eP4uPjPRfYTpgwQZ988olaWlo8NR9//LEGDx7calgBAADdi9+nhDIzM7VlyxYVFhaqqqpKGRkZqq6uVlpamqSrp2rmzJnjqU9LS9Pp06eVmZmpqqoqFRYWqqCgQIsXL/bUPPXUU6qrq9OiRYv08ccfa+fOnVq9erUWLFhwCw4RAAB0dn7f1pySkqK6ujqtWrVKLpdLcXFxKikp8dyG7HK5vJ7JEh0drZKSEmVkZGjjxo0aMmSINmzY4LmlWZIiIyO1Z88eZWRkaMyYMRo6dKgWLVqkJUuW3IJDBAAAnZ3DunYFbCfX0NCgsLAw1dfXKzQ01O52OtTwpTvtbgEd6NSL99vdAjoQn+/upTt+vm/05zffJQQAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGa1dgyc3NVXR0tEJCQuR0OnXw4MHr1h84cEBOp1MhISGKiYlRfn5+m7Xbt2+Xw+HQjBkz2tMaAADogvwOLEVFRUpPT1dWVpYqKyuVlJSkqVOnqrq6utX6kydPatq0aUpKSlJlZaWWL1+uhQsXqri42Kf29OnTWrx4sZKSkvw/EgAA0GX5HVjWr1+vefPmaf78+YqNjVVOTo4iIyOVl5fXan1+fr6GDRumnJwcxcbGav78+Zo7d67WrVvnVed2u/XYY4/pueeeU0xMTPuOBgAAdEl+BZampiZVVFQoOTnZazw5OVllZWWtblNeXu5TP2XKFB05ckTNzc2esVWrVmngwIGaN2/eDfXS2NiohoYGrwUAAHRNfgWW2tpaud1uhYeHe42Hh4erpqam1W1qamparb9y5Ypqa2slSe+9954KCgq0efPmG+4lOztbYWFhniUyMtKfQwEAAJ1Iuy66dTgcXq8ty/IZ+7b6a+Pnz5/X448/rs2bN2vAgAE33MOyZctUX1/vWc6cOePHEQAAgM6kpz/FAwYMUEBAgM9syrlz53xmUa6JiIhotb5nz57q37+/PvroI506dUoPPPCAZ31LS8vV5nr21IkTJzRixAif/QYHBys4ONif9gEAQCfl1wxLUFCQnE6nSktLvcZLS0uVmJjY6jYJCQk+9Xv27FF8fLwCAwM1atQo/e1vf9PRo0c9y/Tp03XPPffo6NGjnOoBAAD+zbBIUmZmplJTUxUfH6+EhARt2rRJ1dXVSktLk3T1VM3Zs2e1detWSVJaWppeeeUVZWZm6sknn1R5ebkKCgq0bds2SVJISIji4uK83qNPnz6S5DMOAAC6J78DS0pKiurq6rRq1Sq5XC7FxcWppKREUVFRkiSXy+X1TJbo6GiVlJQoIyNDGzdu1JAhQ7RhwwbNnDnz1h0FAADo0hzWtStgO7mGhgaFhYWpvr5eoaGhdrfToYYv3Wl3C+hAp1683+4W0IH4fHcv3fHzfaM/v/kuIQAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADBeuwJLbm6uoqOjFRISIqfTqYMHD163/sCBA3I6nQoJCVFMTIzy8/O91m/evFlJSUnq27ev+vbtq8mTJ+vw4cPtaQ0AAHRBfgeWoqIipaenKysrS5WVlUpKStLUqVNVXV3dav3Jkyc1bdo0JSUlqbKyUsuXL9fChQtVXFzsqdm/f78eeeQRvfvuuyovL9ewYcOUnJyss2fPtv/IAABAl+GwLMvyZ4Px48dr3LhxysvL84zFxsZqxowZys7O9qlfsmSJ3nrrLVVVVXnG0tLSdOzYMZWXl7f6Hm63W3379tUrr7yiOXPm3FBfDQ0NCgsLU319vUJDQ/05pE5v+NKddreADnTqxfvtbgEdiM9399IdP983+vPbrxmWpqYmVVRUKDk52Ws8OTlZZWVlrW5TXl7uUz9lyhQdOXJEzc3NrW5z6dIlNTc3q1+/fm320tjYqIaGBq8FAAB0TX4FltraWrndboWHh3uNh4eHq6amptVtampqWq2/cuWKamtrW91m6dKlGjp0qCZPntxmL9nZ2QoLC/MskZGR/hwKAADoRNp10a3D4fB6bVmWz9i31bc2Lklr1qzRtm3btGPHDoWEhLS5z2XLlqm+vt6znDlzxp9DAAAAnUhPf4oHDBiggIAAn9mUc+fO+cyiXBMREdFqfc+ePdW/f3+v8XXr1mn16tXau3evxowZc91egoODFRwc7E/7AACgk/JrhiUoKEhOp1OlpaVe46WlpUpMTGx1m4SEBJ/6PXv2KD4+XoGBgZ6xtWvX6vnnn9euXbsUHx/vT1sAAKCL8/uUUGZmprZs2aLCwkJVVVUpIyND1dXVSktLk3T1VM3/vbMnLS1Np0+fVmZmpqqqqlRYWKiCggItXrzYU7NmzRqtWLFChYWFGj58uGpqalRTU6MLFy7cgkMEAACdnV+nhCQpJSVFdXV1WrVqlVwul+Li4lRSUqKoqChJksvl8nomS3R0tEpKSpSRkaGNGzdqyJAh2rBhg2bOnOmpyc3NVVNTk2bNmuX1Xs8884yeffbZdh4aAADoKvx+DoupeA4Luovu+JyG7ozPd/fSHT/f38lzWAAAAOxAYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAAADjEVgAAIDxCCwAAMB4BBYAAGA8AgsAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWAAAgPEILAAAwHgEFgAAYDwCCwAAMB6BBQAAGI/AAgAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAYj8ACAACMR2ABAADGa1dgyc3NVXR0tEJCQuR0OnXw4MHr1h84cEBOp1MhISGKiYlRfn6+T01xcbFGjx6t4OBgjR49Wn/+85/b0xoAAOiC/A4sRUVFSk9PV1ZWliorK5WUlKSpU6equrq61fqTJ09q2rRpSkpKUmVlpZYvX66FCxequLjYU1NeXq6UlBSlpqbq2LFjSk1N1ezZs/X++++3/8gAAECX4bAsy/Jng/Hjx2vcuHHKy8vzjMXGxmrGjBnKzs72qV+yZIneeustVVVVecbS0tJ07NgxlZeXS5JSUlLU0NCgd955x1Nz3333qW/fvtq2bdsN9dXQ0KCwsDDV19crNDTUn0Pq9IYv3Wl3C+hAp1683+4W0IH4fHcv3fHzfaM/v3v6s9OmpiZVVFRo6dKlXuPJyckqKytrdZvy8nIlJyd7jU2ZMkUFBQVqbm5WYGCgysvLlZGR4VOTk5PTZi+NjY1qbGz0vK6vr5d09cC7m5bGS3a3gA7UHf+Nd2d8vruX7vj5vnbM3zZ/4ldgqa2tldvtVnh4uNd4eHi4ampqWt2mpqam1forV66otrZWgwcPbrOmrX1KUnZ2tp577jmf8cjIyBs9HKBTCsuxuwMA35Xu/Pk+f/68wsLC2lzvV2C5xuFweL22LMtn7Nvqvznu7z6XLVumzMxMz+uWlhZ98cUX6t+//3W3Q9fQ0NCgyMhInTlzptudAgS6Oj7f3YtlWTp//ryGDBly3Tq/AsuAAQMUEBDgM/Nx7tw5nxmSayIiIlqt79mzp/r373/dmrb2KUnBwcEKDg72GuvTp8+NHgq6iNDQUP5DA7ooPt/dx/VmVq7x6y6hoKAgOZ1OlZaWeo2XlpYqMTGx1W0SEhJ86vfs2aP4+HgFBgZet6atfQIAgO7F71NCmZmZSk1NVXx8vBISErRp0yZVV1crLS1N0tVTNWfPntXWrVslXb0j6JVXXlFmZqaefPJJlZeXq6CgwOvun0WLFmnixIn69a9/rQcffFBvvvmm9u7dq0OHDt2iwwQAAJ2Z34ElJSVFdXV1WrVqlVwul+Li4lRSUqKoqChJksvl8nomS3R0tEpKSpSRkaGNGzdqyJAh2rBhg2bOnOmpSUxM1Pbt27VixQqtXLlSI0aMUFFRkcaPH38LDhFdUXBwsJ555hmf04IAOj8+32iN389hAQAA6Gh8lxAAADAegQUAABiPwAIAAIxHYAEAAMYjsAAAAOMRWNApnT59WsePH1dLS4vdrQAAOgCBBUb73e9+5/Ot3T//+c8VExOjO++8U3FxcTpz5ow9zQEAOgyBBUbLz8/3+o6JXbt26dVXX9XWrVv117/+VX369Gn1W7sBdC6XL1/W2rVrNW3aNMXHx2vcuHFeC9Cub2sGOsrHH3+s+Ph4z+s333xT06dP12OPPSZJWr16tZ544gm72gNwi8ydO1elpaWaNWuWfvSjH8nhcNjdEgxDYIHRvv76a69vay0rK9PcuXM9r2NiYny+6RtA57Nz506VlJRowoQJdrcCQ3FKCEaLiopSRUWFJKm2tlYfffSR7r77bs/6mpqaG/pacgBmGzp0qHr37m13GzAYgQVGmzNnjhYsWKDnn39eDz30kEaNGiWn0+lZX1ZWpri4OBs7BHArvPTSS1qyZIlOnz5tdyswFKeEYLQlS5bo0qVL2rFjhyIiIvTHP/7Ra/17772nRx55xKbuANwq8fHxunz5smJiYtSrVy8FBgZ6rf/iiy9s6gym4Nua0ak1NzfL5XJp2LBhdrcC4CZMnjxZ1dXVmjdvnsLDw30uuv3Zz35mU2cwBYEFndqxY8c0btw4ud1uu1sBcBN69eql8vJyjR071u5WYCiuYQEA2G7UqFH6+uuv7W4DBiOwAABs9+KLL+qXv/yl9u/fr7q6OjU0NHgtAKeE0KlxSgjoGnr0uPr78zevXbEsSw6Hg884uEsIZvvwww+vu/7EiRMd1AmA79K7775rdwswHDMsMFqPHj3kcDh0vX+m/PYFAF0fMyww2smTJ7+15ssvv+yATgB81w4ePKjf/va3+vTTT/XHP/5RQ4cO1euvv67o6GivJ1yje+KiWxgtKiqq1aVPnz7auXOnfvrTn3o9+RZA51RcXKwpU6botttu0wcffKDGxkZJ0vnz57V69Wqbu4MJCCzoVPbt26fHH39cgwcP1ssvv6ypU6fqyJEjdrcF4Cb96le/Un5+vjZv3uz1lNvExER98MEHNnYGU3BKCMb73//9X7322msqLCzUxYsXNXv2bDU3N6u4uFijR4+2uz0At8CJEyc0ceJEn/HQ0FB99dVXHd8QjMMMC4w2bdo0jR49WsePH9fLL7+szz77TC+//LLdbQG4xQYPHqxPPvnEZ/zQoUOKiYmxoSOYhhkWGG3Pnj1auHChnnrqKX3/+9+3ux0A35F/+Zd/0aJFi1RYWCiHw6HPPvtM5eXlWrx4sf793//d7vZgAAILjHbw4EEVFhYqPj5eo0aNUmpqqlJSUuxuC8At9m//9m+qr6/XPffco8uXL2vixIkKDg7W4sWL9Ytf/MLu9mAAnsOCTuHSpUvavn27CgsLdfjwYbndbq1fv15z585V79697W4PwC1y6dIlHT9+XC0tLRo9erTuuOMOu1uCIQgs6HROnDihgoICvf766/rqq69077336q233rK7LQA3Ye7cufrNb37j8wvIxYsX9fTTT6uwsNCmzmAKAgs6LbfbrbfffluFhYUEFqCTCwgIkMvl0qBBg7zGa2trFRERoStXrtjUGUzBNSzotAICAjRjxgzNmDHD7lYAtFNDQ4Msy5JlWTp//rxCQkI869xut0pKSnxCDLonAgsAwDZ9+vSRw+GQw+HQD37wA5/1DodDzz33nA2dwTScEgIA2ObAgQOyLEs//vGPVVxcrH79+nnWBQUFKSoqSkOGDLGxQ5iCwAIAsN3p06c1bNgwORwOu1uBoQgsAADbfPjhhzdUN2bMmO+4E5iOwAIAsE2PHj3kcDh0vR9FDodDbre7A7uCibjoFgBgm5MnT9rdAjoJZlgAAIDxmGEBABjh8uXL+vDDD3Xu3Dm1tLR4rZs+fbpNXcEUBBYAgO127dqlOXPmqLa21mcd17BAknrY3QAAAL/4xS/00EMPyeVyqaWlxWshrEDiGhYAgAFCQ0NVWVmpESNG2N0KDMUMCwDAdrNmzdL+/fvtbgMGY4YFAGC7S5cu6aGHHtLAgQN15513KjAw0Gv9woULbeoMpiCwAABst2XLFqWlpem2225T//79vR7R73A49Omnn9rYHUxAYAEA2C4iIkILFy7U0qVL1aMHVyvAF/8qAAC2a2pqUkpKCmEFbeJfBgDAdj/72c9UVFRkdxswGA+OAwDYzu12a82aNdq9e7fGjBnjc9Ht+vXrbeoMpuAaFgCA7e6555421zkcDu3bt68Du4GJCCwAAMB4XMMCADDGJ598ot27d+vrr7+WJPE7Na4hsAAAbFdXV6ef/OQn+sEPfqBp06bJ5XJJkubPn69f/vKXNncHExBYAAC2y8jIUGBgoKqrq9WrVy/PeEpKinbt2mVjZzAFdwkBAGy3Z88e7d69W9/73ve8xr///e/r9OnTNnUFkzDDAgCw3cWLF71mVq6pra1VcHCwDR3BNAQWAIDtJk6cqK1bt3peOxwOtbS0aO3atde95RndB7c1AwBsd/z4cU2aNElOp1P79u3T9OnT9dFHH+mLL77Qe++9pxEjRtjdImxGYAEAGKGmpkZ5eXmqqKhQS0uLxo0bpwULFmjw4MF2twYDEFgAALarrq5WZGSkHA5Hq+uGDRtmQ1cwCYEFAGC7gIAAuVwuDRo0yGu8rq5OgwYNktvttqkzmIKLbgEAtrMsq9XZlQsXLigkJMSGjmAansMCALBNZmampKt3Ba1cudLr1ma32633339fd911l03dwSQEFgCAbSorKyVdnWH529/+pqCgIM+6oKAgjR07VosXL7arPRiEa1gAALZ74okntGHDBvXu3dvuVmAoAgsAwDY//elPb6hux44d33EnMB2nhAAAtgkLC7O7BXQSzLAAAADjcVszAAAwHoEFAAAYj8ACAACMR2ABAADGI7AAuGn79++Xw+HQV199Zcx7DR8+XDk5Od95PwA6BoEFwA0rKytTQECA7rvvPtt6SExMlMvl8twO+9prr6lPnz629QOgYxBYANywwsJCPf300zp06JCqq6s7/P2bm5sVFBSkiIiIVr8oD0DXRWABcEMuXryoP/zhD3rqqaf0T//0T3rttdeuW79582ZFRkaqV69e+ud//metX7/eZyYkLy9PI0aMUFBQkEaOHKnXX3/da73D4VB+fr4efPBB3X777frVr37ldUpo//79euKJJ1RfXy+HwyGHw6Fnn33Ws/2lS5c0d+5c9e7dW8OGDdOmTZs8606dOiWHw6E//OEPSkpK0m233aZ/+Id/0Mcff6y//vWvio+P1x133KH77rtPn3/++c3+9QG4WRYA3ICCggIrPj7esizLevvtt63hw4dbLS0tlmVZ1rvvvmtJsr788kvLsizr0KFDVo8ePay1a9daJ06csDZu3Gj169fPCgsL8+xvx44dVmBgoLVx40brxIkT1ksvvWQFBARY+/bt89RIsgYNGmQVFBRY//M//2OdOnXK670aGxutnJwcKzQ01HK5XJbL5bLOnz9vWZZlRUVFWf369bM2btxo/fd//7eVnZ1t9ejRw6qqqrIsy7JOnjxpSbJGjRpl7dq1yzp+/Lj1j//4j9a4ceOsSZMmWYcOHbI++OAD6+/+7u+stLS0DvgbBnA9BBYANyQxMdHKycmxLMuympubrQEDBlilpaWWZfkGlpSUFOv+++/32v6xxx7zCiyJiYnWk08+6VXz0EMPWdOmTfO8lmSlp6d71XzzvV599VWv/V4TFRVlPf74457XLS0t1qBBg6y8vDzLsv5/YNmyZYunZtu2bZYk6y9/+YtnLDs72xo5cuT1/moAdABOCQH4VidOnNDhw4f18MMPS5J69uyplJQUFRYWtln/ox/9yGvsm6+rqqo0YcIEr7EJEyaoqqrKayw+Pr7dfY8ZM8bzZ4fDoYiICJ07d67NmvDwcEnSnXfe6TX2zW0AdDy+/BDAtyooKNCVK1c0dOhQz5hlWQoMDNSXX37pU29Zls9FsVYrX1vWWs03x26//fZ29x0YGOjzfi0tLW3WXHvvb459cxsAHY8ZFgDXdeXKFW3dulUvvfSSjh496lmOHTumqKgo/f73v/fZZtSoUTp8+LDX2JEjR7xex8bG6tChQ15jZWVlio2N9au/oKAgud1uv7YB0PkwwwLguv7zP/9TX375pebNm+d59sk1s2bNUkFBgf7jP/7Da/zpp5/WxIkTtX79ej3wwAPat2+f3nnnHa/Zk3/913/V7NmzNW7cOP3kJz/R22+/rR07dmjv3r1+9Td8+HBduHBBf/nLXzR27Fj16tVLvXr1av8BAzASMywArqugoECTJ0/2CSuSNHPmTB09elQffPCB1/iECROUn5+v9evXa+zYsdq1a5cyMjIUEhLiqZkxY4Z+85vfaO3atfr7v/97/fa3v9Wrr76qSZMm+dVfYmKi0tLSlJKSooEDB2rNmjXtOk4AZnNYrZ1YBoBb7Mknn9R//dd/6eDBg3a3AqAT4pQQgO/EunXrdO+99+r222/XO++8o9/97nfKzc21uy0AnRQzLAC+E7Nnz9b+/ft1/vx5xcTE6Omnn1ZaWprdbQHopAgsAADAeFx0CwAAjEdgAQAAxiOwAAAA4xFYAACA8QgsAADAeAQWAABgPAILAAAwHoEFAAAY7/8BsXXWBXROp/wAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "results.groupby('Algorithm').ndcg.mean().plot.bar()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}