1.3_movie_recommendation_mf.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5-final"
  },
  "orig_nbformat": 2,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3",
   "language": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# Movie recommendation system\n",
    "\n",
    "It uses **Matrix Factorization (MF) algorithm for Item to Item collaborative Filtering (MF IICBF)**\n",
    "\n",
    "It's a model-based recommender system. It’s a collaborative based filtering but with some additional ideas.  \n",
    "The main idea here - split all items (movies) into groups called Latent Features. For example for different movies we could group them into genre groups, like Action, Horror, and then assign these groups to users - for example Bob likes 20% of Action movies and 80 % of Horrow movies (so all movies he watched and liked could be grouped into 2 features).\n",
    "\n",
    "And the main problem here - splitting all movies into these groups.  \n",
    "There is a PCA technique (Principal Compoonent  Analysis) which  is a statistical procedure which reduces the dimension of our user-item matrix without losing any important information! In simple words it’s a procedure to come up with latent features (find features to group all movies into). \n",
    "\n",
    "\n",
    "At the beginning we do an initial setup and import.\n",
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import pandas as pd\n",
    "import numpy as np  \n",
    "from sklearn.metrics import pairwise_distances\n",
    "from scipy.sparse.linalg import svds # we will do SVD with scipy\n",
    "\n",
    "#initial output setup\n",
    "pd.set_option('display.max_rows', 70)\n",
    "pd.set_option('display.max_columns', 70)\n",
    "pd.set_option('display.width', 500)"
   ]
  },
  {
   "source": [
    "Now we are reading MovieLens datasets with movies and user ratings.  "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reading movies file\n",
    "movies = pd.read_csv('data/movies.csv', sep=',', encoding='latin-1', usecols=['movieId','title','genres'])\n",
    "\n",
    "# Reading ratings file\n",
    "ratings = pd.read_csv('data/ratings.csv', sep=',', encoding='latin-1', usecols=['userId','movieId','rating','timestamp'])"
   ]
  },
  {
   "source": [
    "To find the recommendation for specific user we need to provide movie ratings for this user.  \n",
    "I'm going to provide recommendations for \"Star Wars\"."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "      movieId                                              title                                genres\n224       260          Star Wars: Episode IV - A New Hope (1977)               Action|Adventure|Sci-Fi\n898      1196  Star Wars: Episode V - The Empire Strikes Back...               Action|Adventure|Sci-Fi\n911      1210  Star Wars: Episode VI - Return of the Jedi (1983)               Action|Adventure|Sci-Fi\n1979     2628   Star Wars: Episode I - The Phantom Menace (1999)               Action|Adventure|Sci-Fi\n3832     5378  Star Wars: Episode II - Attack of the Clones (...          Action|Adventure|Sci-Fi|IMAX\n5896    33493  Star Wars: Episode III - Revenge of the Sith (...               Action|Adventure|Sci-Fi\n6823    61160                   Star Wars: The Clone Wars (2008)     Action|Adventure|Animation|Sci-Fi\n7367    79006  Empire of Dreams: The Story of the 'Star Wars'...                           Documentary\n8683   122886  Star Wars: Episode VII - The Force Awakens (2015)  Action|Adventure|Fantasy|Sci-Fi|IMAX\n8908   135216               The Star Wars Holiday Special (1978)      Adventure|Children|Comedy|Sci-Fi\n9433   166528                Rogue One: A Star Wars Story (2016)       Action|Adventure|Fantasy|Sci-Fi\n9645   179819                    Star Wars: The Last Jedi (2017)       Action|Adventure|Fantasy|Sci-Fi\n9710   187595                     Solo: A Star Wars Story (2018)      Action|Adventure|Children|Sci-Fi\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "        userId  movieId  rating     timestamp\n",
       "100834     610   168252     5.0  1.493846e+09\n",
       "100835     610   170875     3.0  1.493846e+09\n",
       "100836     611      260     5.0           NaN\n",
       "100837     611     1196     5.0           NaN\n",
       "100838     611     1210     5.0           NaN"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>userId</th>\n      <th>movieId</th>\n      <th>rating</th>\n      <th>timestamp</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>100834</th>\n      <td>610</td>\n      <td>168252</td>\n      <td>5.0</td>\n      <td>1.493846e+09</td>\n    </tr>\n    <tr>\n      <th>100835</th>\n      <td>610</td>\n      <td>170875</td>\n      <td>3.0</td>\n      <td>1.493846e+09</td>\n    </tr>\n    <tr>\n      <th>100836</th>\n      <td>611</td>\n      <td>260</td>\n      <td>5.0</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>100837</th>\n      <td>611</td>\n      <td>1196</td>\n      <td>5.0</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>100838</th>\n      <td>611</td>\n      <td>1210</td>\n      <td>5.0</td>\n      <td>NaN</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 71
    }
   ],
   "source": [
    "\n",
    "# Find all Star Wars movies\n",
    "print(movies[movies.title.str.contains(\"Star Wars\")])\n",
    "\n",
    "# create a new user and add ratings for Star Wars movies.\n",
    "max_user_id = ratings.userId.max()\n",
    "my_user_id = max_user_id+1\n",
    "my_rating = pd.DataFrame([[my_user_id,260,5],[my_user_id,1196,5],[my_user_id,1210,5]],columns=['userId','movieId',\"rating\"])\n",
    "# append ratings to the existing dataset\n",
    "ratings = ratings.append(my_rating,ignore_index=True)\n",
    "\n",
    "# check last 5 rows to ensure user properly added\n",
    "ratings.tail()"
   ]
  },
  {
   "source": [
    "Now let's perform some data investigation for ratings dataset.  \n",
    "Sparcity is calculated as percent of values in the matrix."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Number of users = 611 | Number of movies = 9724\nThe sparsity level of the dataset is 98.3%\n"
     ]
    }
   ],
   "source": [
    "n_users = ratings.userId.unique().shape[0] # number of unique users\n",
    "n_movies = ratings.movieId.unique().shape[0] # number of unique movies\n",
    "print('Number of users = ' + str(n_users) + ' | Number of movies = ' + str(n_movies))\n",
    "\n",
    "sparsity = round(1.0 - len(ratings) / float(n_users * n_movies), 3)\n",
    "print('The sparsity level of the dataset is ' +  str(sparsity * 100) + '%')"
   ]
  },
  {
   "source": [
    "Now we are going to create sparse user-to-movie pivot table, which contains movie ratings for all movies and every user.  \n",
    "Later we will use this table for MF."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "movieId  1       2       3       4       5       6       7       8       9       10      11      12      13      14      15      16      17      18      19      20      21      22      23      24      25      26      27      28      29      30      31      32      34      36      38      ...  187031  187541  187593  187595  187717  188189  188301  188675  188751  188797  188833  189043  189111  189333  189381  189547  189713  190183  190207  190209  190213  190215  190219  190221  191005  \\\n",
       "userId                                                                                                                                                                                                                                                                                           ...                                                                                                                                                                                                           \n",
       "1           4.0     0.0     4.0     0.0     0.0     4.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "2           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "3           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.5     0.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "4           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     3.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     2.0     0.0     0.0     0.0  ...     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "5           4.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     4.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     4.0     4.0     0.0  ...     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0   \n",
       "\n",
       "movieId  193565  193567  193571  193573  193579  193581  193583  193585  193587  193609  \n",
       "userId                                                                                   \n",
       "1           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  \n",
       "2           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  \n",
       "3           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  \n",
       "4           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  \n",
       "5           0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0     0.0  \n",
       "\n",
       "[5 rows x 9724 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th>movieId</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n      <th>10</th>\n      <th>11</th>\n      <th>12</th>\n      <th>13</th>\n      <th>14</th>\n      <th>15</th>\n      <th>16</th>\n      <th>17</th>\n      <th>18</th>\n      <th>19</th>\n      <th>20</th>\n      <th>21</th>\n      <th>22</th>\n      <th>23</th>\n      <th>24</th>\n      <th>25</th>\n      <th>26</th>\n      <th>27</th>\n      <th>28</th>\n      <th>29</th>\n      <th>30</th>\n      <th>31</th>\n      <th>32</th>\n      <th>34</th>\n      <th>36</th>\n      <th>38</th>\n      <th>...</th>\n      <th>187031</th>\n      <th>187541</th>\n      <th>187593</th>\n      <th>187595</th>\n      <th>187717</th>\n      <th>188189</th>\n      <th>188301</th>\n      <th>188675</th>\n      <th>188751</th>\n      <th>188797</th>\n      <th>188833</th>\n      <th>189043</th>\n      <th>189111</th>\n      <th>189333</th>\n      <th>189381</th>\n      <th>189547</th>\n      <th>189713</th>\n      <th>190183</th>\n      <th>190207</th>\n      <th>190209</th>\n      <th>190213</th>\n      <th>190215</th>\n      <th>190219</th>\n      <th>190221</th>\n      <th>191005</th>\n      <th>193565</th>\n      <th>193567</th>\n      <th>193571</th>\n      <th>193573</th>\n      <th>193579</th>\n      <th>193581</th>\n      <th>193583</th>\n      <th>193585</th>\n      <th>193587</th>\n      <th>193609</th>\n    </tr>\n    <tr>\n      <th>userId</th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n      <th></th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>1</th>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>...</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>...</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.5</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>...</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>3.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>2.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>...</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>4.0</td>\n      <td>4.0</td>\n      <td>0.0</td>\n      <td>...</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n      <td>0.0</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 9724 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 73
    }
   ],
   "source": [
    "\n",
    "ratings_pivot = ratings.pivot(index = 'userId', columns ='movieId', values = 'rating').fillna(0)\n",
    "ratings_pivot.head()"
   ]
  },
  {
   "source": [
    "Now we need to prepare data.  \n",
    "We are going to convert all values as a deviation from the mean value.  \n",
    "\n",
    "So we are calculating mean rating value for every user, and then process the original table and calculate deviation from this value.  \n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "[[4. 0. 4. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]\n ...\n [3. 0. 0. ... 0. 0. 0.]\n [5. 0. 0. ... 0. 0. 0.]\n [0. 0. 0. ... 0. 0. 0.]]\n"
     ]
    }
   ],
   "source": [
    "user_ratings_pivot = ratings_pivot.values # returns NumPy representation of ratings pivot table (dataset's array representation)\n",
    "print(user_ratings_pivot)"
   ]
  },
  {
   "source": [
    "Now we are going to use SVD.  \n",
    "It's an algorightm to do the matrix factorisation.  \n",
    "It processes the input matrix and decompose it into 2 lower rank matrices (one is latent features matrix).  \n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(50, 50)\n(611, 50)\n(50, 9724)\n"
     ]
    }
   ],
   "source": [
    "# apply SVD method to the and get as a result\n",
    "# U = lower-rank matrix with user to features values\n",
    "# sigma - diagonal matrix\n",
    "# Vt - lower-rank matrix with movie to features matrix\n",
    "U, sigma, Vt = svds(user_ratings_pivot, k = 50)\n",
    "\n",
    "sigma = np.diag(sigma) #scipy returns an array instead of the diag matrix, so we need to covnert\n",
    "\n",
    "# check shapes of results\n",
    "print(sigma.shape)\n",
    "print(U.shape)\n",
    "print(Vt.shape)"
   ]
  },
  {
   "source": [
    "Now we can calculate prediction ratings as a dot product of 3 calculated matrices.  \n",
    "Here we need to use correct values instead of mean-deviation.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "(611, 9724)\n"
     ]
    }
   ],
   "source": [
    "all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt)\n",
    "print(all_user_predicted_ratings.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "movieId    1         2         3         4         5         6         7         8         9         10        11        12        13        14        15        16        17        18        19        20        21        22        23        24        25        26        27        28        29        30        31        32        34        36        38      ...    187031    187541    187593    187595    187717    188189    188301    188675    188751    188797    188833    189043    189111  \\\n",
       "0        2.184950  0.391633  0.837100 -0.082701 -0.543542  2.521065 -0.887665 -0.025187  0.196942  1.605342 -0.487325  0.238066  0.340428  0.052730 -0.112107  1.351254 -0.409110 -0.073231  0.288128  0.420735  0.232836  0.357406 -0.145697  0.248317  0.071823 -0.224034 -0.037536  0.072296 -0.161649 -0.076665 -0.335666  0.798281  1.652119 -0.317737  0.138070  ...  0.033877  0.013562  0.155168  0.041159 -0.181594 -0.163434 -0.124946  0.068277  0.038370  0.053101  0.087785  0.009311  0.011174   \n",
       "1        0.209973  0.004889  0.031136  0.017289  0.183989 -0.059980  0.083681  0.023824  0.048244 -0.151797  0.078741  0.063507  0.004551 -0.000828  0.045123  0.095526 -0.025819 -0.004233  0.109314 -0.008865 -0.076552  0.014678  0.008501  0.022756 -0.213004  0.040350 -0.002419 -0.022327 -0.174237  0.013023  0.032537 -0.200054 -0.165025  0.030066  0.000017  ...  0.016133  0.050673  0.067765  0.056652 -0.008167 -0.007350  0.013486 -0.002467  0.037876  0.034995 -0.003172  0.002407  0.002888   \n",
       "2        0.013576  0.034661  0.050505  0.000187 -0.005458  0.114697 -0.007452  0.000736  0.004761 -0.061295 -0.004365  0.032795  0.011505 -0.010543  0.007799  0.039761 -0.012951  0.019850 -0.061142 -0.013329 -0.073133  0.012797 -0.018178  0.046400 -0.041909 -0.004594 -0.012161 -0.005949  0.077093  0.004850  0.020475  0.029609 -0.000791  0.014199  0.003032  ...  0.004665  0.001418 -0.001979  0.009616 -0.008726 -0.007854  0.013464  0.008514 -0.003015  0.007247  0.010947 -0.000003 -0.000004   \n",
       "3        2.012104 -0.395132 -0.290011  0.093849  0.124135  0.259978  0.473116  0.036044  0.011479 -0.023374  0.663969 -0.107960  0.279998  0.265492 -0.029648  0.143392  1.799882 -0.257221 -0.014255  0.120752  1.940472 -0.187845 -0.036481  0.062927  1.637332  0.156245  0.023510  0.546534  0.496478  0.258787 -0.193984  2.078207  1.904503  0.848447 -0.078282  ...  0.000263 -0.013504  0.168943 -0.084685  0.116135  0.104521 -0.004731 -0.019294  0.047298 -0.004414 -0.024807 -0.002637 -0.003164   \n",
       "4        1.336997  0.772816  0.064191  0.113824  0.274960  0.584009  0.250691  0.131507 -0.086427  1.035331  0.963734 -0.071943  0.137122  0.236478  0.207081  0.541823  0.774360 -0.050299  0.333631  0.083555  1.143493  0.329835  0.153495  0.100726  0.863559  0.240538  0.033295  0.120760  0.037955 -0.000212  0.460581  1.260663  1.906905  1.002758 -0.016437  ...  0.000060 -0.006795  0.031475  0.008180 -0.018854 -0.016969  0.000780 -0.004706  0.000969 -0.003761 -0.006051  0.007059  0.008471   \n",
       "\n",
       "movieId    189333    189381    189547    189713    190183    190207    190209    190213    190215    190219    190221    191005    193565    193567    193571    193573    193579    193581    193583    193585    193587    193609  \n",
       "0       -0.019988  0.048769 -0.010174 -0.090797 -0.015214  0.005587  0.014898  0.003725  0.005587  0.003725  0.003725 -0.032011 -0.024897 -0.021340 -0.028454 -0.028454 -0.024897 -0.028454 -0.024897 -0.024897 -0.024897 -0.059140  \n",
       "1        0.018447 -0.001762 -0.001525 -0.004084  0.009439  0.001444  0.003851  0.000963  0.001444  0.000963  0.000963  0.024298  0.018899  0.016199  0.021598  0.021598  0.018899  0.021598  0.018899  0.018899  0.018899  0.031980  \n",
       "2       -0.003638  0.006082 -0.000426 -0.004363 -0.001972 -0.000002 -0.000005 -0.000001 -0.000002 -0.000001 -0.000001 -0.002067 -0.001608 -0.001378 -0.001837 -0.001837 -0.001608 -0.001837 -0.001608 -0.001608 -0.001608 -0.000534  \n",
       "3        0.005605 -0.013782  0.002416  0.058067 -0.022149 -0.001582 -0.004219 -0.001055 -0.001582 -0.001055 -0.001055  0.002527  0.001966  0.001685  0.002246  0.002246  0.001966  0.002246  0.001966  0.001966  0.001966 -0.021530  \n",
       "4       -0.005795 -0.003362  0.000330 -0.009427 -0.014938  0.004236  0.011295  0.002824  0.004236  0.002824  0.002824 -0.005661 -0.004403 -0.003774 -0.005032 -0.005032 -0.004403 -0.005032 -0.004403 -0.004403 -0.004403 -0.006112  \n",
       "\n",
       "[5 rows x 9724 columns]"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th>movieId</th>\n      <th>1</th>\n      <th>2</th>\n      <th>3</th>\n      <th>4</th>\n      <th>5</th>\n      <th>6</th>\n      <th>7</th>\n      <th>8</th>\n      <th>9</th>\n      <th>10</th>\n      <th>11</th>\n      <th>12</th>\n      <th>13</th>\n      <th>14</th>\n      <th>15</th>\n      <th>16</th>\n      <th>17</th>\n      <th>18</th>\n      <th>19</th>\n      <th>20</th>\n      <th>21</th>\n      <th>22</th>\n      <th>23</th>\n      <th>24</th>\n      <th>25</th>\n      <th>26</th>\n      <th>27</th>\n      <th>28</th>\n      <th>29</th>\n      <th>30</th>\n      <th>31</th>\n      <th>32</th>\n      <th>34</th>\n      <th>36</th>\n      <th>38</th>\n      <th>...</th>\n      <th>187031</th>\n      <th>187541</th>\n      <th>187593</th>\n      <th>187595</th>\n      <th>187717</th>\n      <th>188189</th>\n      <th>188301</th>\n      <th>188675</th>\n      <th>188751</th>\n      <th>188797</th>\n      <th>188833</th>\n      <th>189043</th>\n      <th>189111</th>\n      <th>189333</th>\n      <th>189381</th>\n      <th>189547</th>\n      <th>189713</th>\n      <th>190183</th>\n      <th>190207</th>\n      <th>190209</th>\n      <th>190213</th>\n      <th>190215</th>\n      <th>190219</th>\n      <th>190221</th>\n      <th>191005</th>\n      <th>193565</th>\n      <th>193567</th>\n      <th>193571</th>\n      <th>193573</th>\n      <th>193579</th>\n      <th>193581</th>\n      <th>193583</th>\n      <th>193585</th>\n      <th>193587</th>\n      <th>193609</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2.184950</td>\n      <td>0.391633</td>\n      <td>0.837100</td>\n      <td>-0.082701</td>\n      <td>-0.543542</td>\n      <td>2.521065</td>\n      <td>-0.887665</td>\n      <td>-0.025187</td>\n      <td>0.196942</td>\n      <td>1.605342</td>\n      <td>-0.487325</td>\n      <td>0.238066</td>\n      <td>0.340428</td>\n      <td>0.052730</td>\n      <td>-0.112107</td>\n      <td>1.351254</td>\n      <td>-0.409110</td>\n      <td>-0.073231</td>\n      <td>0.288128</td>\n      <td>0.420735</td>\n      <td>0.232836</td>\n      <td>0.357406</td>\n      <td>-0.145697</td>\n      <td>0.248317</td>\n      <td>0.071823</td>\n      <td>-0.224034</td>\n      <td>-0.037536</td>\n      <td>0.072296</td>\n      <td>-0.161649</td>\n      <td>-0.076665</td>\n      <td>-0.335666</td>\n      <td>0.798281</td>\n      <td>1.652119</td>\n      <td>-0.317737</td>\n      <td>0.138070</td>\n      <td>...</td>\n      <td>0.033877</td>\n      <td>0.013562</td>\n      <td>0.155168</td>\n      <td>0.041159</td>\n      <td>-0.181594</td>\n      <td>-0.163434</td>\n      <td>-0.124946</td>\n      <td>0.068277</td>\n      <td>0.038370</td>\n      <td>0.053101</td>\n      <td>0.087785</td>\n      <td>0.009311</td>\n      <td>0.011174</td>\n      <td>-0.019988</td>\n      <td>0.048769</td>\n      <td>-0.010174</td>\n      <td>-0.090797</td>\n      <td>-0.015214</td>\n      <td>0.005587</td>\n      <td>0.014898</td>\n      <td>0.003725</td>\n      <td>0.005587</td>\n      <td>0.003725</td>\n      <td>0.003725</td>\n      <td>-0.032011</td>\n      <td>-0.024897</td>\n      <td>-0.021340</td>\n      <td>-0.028454</td>\n      <td>-0.028454</td>\n      <td>-0.024897</td>\n      <td>-0.028454</td>\n      <td>-0.024897</td>\n      <td>-0.024897</td>\n      <td>-0.024897</td>\n      <td>-0.059140</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>0.209973</td>\n      <td>0.004889</td>\n      <td>0.031136</td>\n      <td>0.017289</td>\n      <td>0.183989</td>\n      <td>-0.059980</td>\n      <td>0.083681</td>\n      <td>0.023824</td>\n      <td>0.048244</td>\n      <td>-0.151797</td>\n      <td>0.078741</td>\n      <td>0.063507</td>\n      <td>0.004551</td>\n      <td>-0.000828</td>\n      <td>0.045123</td>\n      <td>0.095526</td>\n      <td>-0.025819</td>\n      <td>-0.004233</td>\n      <td>0.109314</td>\n      <td>-0.008865</td>\n      <td>-0.076552</td>\n      <td>0.014678</td>\n      <td>0.008501</td>\n      <td>0.022756</td>\n      <td>-0.213004</td>\n      <td>0.040350</td>\n      <td>-0.002419</td>\n      <td>-0.022327</td>\n      <td>-0.174237</td>\n      <td>0.013023</td>\n      <td>0.032537</td>\n      <td>-0.200054</td>\n      <td>-0.165025</td>\n      <td>0.030066</td>\n      <td>0.000017</td>\n      <td>...</td>\n      <td>0.016133</td>\n      <td>0.050673</td>\n      <td>0.067765</td>\n      <td>0.056652</td>\n      <td>-0.008167</td>\n      <td>-0.007350</td>\n      <td>0.013486</td>\n      <td>-0.002467</td>\n      <td>0.037876</td>\n      <td>0.034995</td>\n      <td>-0.003172</td>\n      <td>0.002407</td>\n      <td>0.002888</td>\n      <td>0.018447</td>\n      <td>-0.001762</td>\n      <td>-0.001525</td>\n      <td>-0.004084</td>\n      <td>0.009439</td>\n      <td>0.001444</td>\n      <td>0.003851</td>\n      <td>0.000963</td>\n      <td>0.001444</td>\n      <td>0.000963</td>\n      <td>0.000963</td>\n      <td>0.024298</td>\n      <td>0.018899</td>\n      <td>0.016199</td>\n      <td>0.021598</td>\n      <td>0.021598</td>\n      <td>0.018899</td>\n      <td>0.021598</td>\n      <td>0.018899</td>\n      <td>0.018899</td>\n      <td>0.018899</td>\n      <td>0.031980</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>0.013576</td>\n      <td>0.034661</td>\n      <td>0.050505</td>\n      <td>0.000187</td>\n      <td>-0.005458</td>\n      <td>0.114697</td>\n      <td>-0.007452</td>\n      <td>0.000736</td>\n      <td>0.004761</td>\n      <td>-0.061295</td>\n      <td>-0.004365</td>\n      <td>0.032795</td>\n      <td>0.011505</td>\n      <td>-0.010543</td>\n      <td>0.007799</td>\n      <td>0.039761</td>\n      <td>-0.012951</td>\n      <td>0.019850</td>\n      <td>-0.061142</td>\n      <td>-0.013329</td>\n      <td>-0.073133</td>\n      <td>0.012797</td>\n      <td>-0.018178</td>\n      <td>0.046400</td>\n      <td>-0.041909</td>\n      <td>-0.004594</td>\n      <td>-0.012161</td>\n      <td>-0.005949</td>\n      <td>0.077093</td>\n      <td>0.004850</td>\n      <td>0.020475</td>\n      <td>0.029609</td>\n      <td>-0.000791</td>\n      <td>0.014199</td>\n      <td>0.003032</td>\n      <td>...</td>\n      <td>0.004665</td>\n      <td>0.001418</td>\n      <td>-0.001979</td>\n      <td>0.009616</td>\n      <td>-0.008726</td>\n      <td>-0.007854</td>\n      <td>0.013464</td>\n      <td>0.008514</td>\n      <td>-0.003015</td>\n      <td>0.007247</td>\n      <td>0.010947</td>\n      <td>-0.000003</td>\n      <td>-0.000004</td>\n      <td>-0.003638</td>\n      <td>0.006082</td>\n      <td>-0.000426</td>\n      <td>-0.004363</td>\n      <td>-0.001972</td>\n      <td>-0.000002</td>\n      <td>-0.000005</td>\n      <td>-0.000001</td>\n      <td>-0.000002</td>\n      <td>-0.000001</td>\n      <td>-0.000001</td>\n      <td>-0.002067</td>\n      <td>-0.001608</td>\n      <td>-0.001378</td>\n      <td>-0.001837</td>\n      <td>-0.001837</td>\n      <td>-0.001608</td>\n      <td>-0.001837</td>\n      <td>-0.001608</td>\n      <td>-0.001608</td>\n      <td>-0.001608</td>\n      <td>-0.000534</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2.012104</td>\n      <td>-0.395132</td>\n      <td>-0.290011</td>\n      <td>0.093849</td>\n      <td>0.124135</td>\n      <td>0.259978</td>\n      <td>0.473116</td>\n      <td>0.036044</td>\n      <td>0.011479</td>\n      <td>-0.023374</td>\n      <td>0.663969</td>\n      <td>-0.107960</td>\n      <td>0.279998</td>\n      <td>0.265492</td>\n      <td>-0.029648</td>\n      <td>0.143392</td>\n      <td>1.799882</td>\n      <td>-0.257221</td>\n      <td>-0.014255</td>\n      <td>0.120752</td>\n      <td>1.940472</td>\n      <td>-0.187845</td>\n      <td>-0.036481</td>\n      <td>0.062927</td>\n      <td>1.637332</td>\n      <td>0.156245</td>\n      <td>0.023510</td>\n      <td>0.546534</td>\n      <td>0.496478</td>\n      <td>0.258787</td>\n      <td>-0.193984</td>\n      <td>2.078207</td>\n      <td>1.904503</td>\n      <td>0.848447</td>\n      <td>-0.078282</td>\n      <td>...</td>\n      <td>0.000263</td>\n      <td>-0.013504</td>\n      <td>0.168943</td>\n      <td>-0.084685</td>\n      <td>0.116135</td>\n      <td>0.104521</td>\n      <td>-0.004731</td>\n      <td>-0.019294</td>\n      <td>0.047298</td>\n      <td>-0.004414</td>\n      <td>-0.024807</td>\n      <td>-0.002637</td>\n      <td>-0.003164</td>\n      <td>0.005605</td>\n      <td>-0.013782</td>\n      <td>0.002416</td>\n      <td>0.058067</td>\n      <td>-0.022149</td>\n      <td>-0.001582</td>\n      <td>-0.004219</td>\n      <td>-0.001055</td>\n      <td>-0.001582</td>\n      <td>-0.001055</td>\n      <td>-0.001055</td>\n      <td>0.002527</td>\n      <td>0.001966</td>\n      <td>0.001685</td>\n      <td>0.002246</td>\n      <td>0.002246</td>\n      <td>0.001966</td>\n      <td>0.002246</td>\n      <td>0.001966</td>\n      <td>0.001966</td>\n      <td>0.001966</td>\n      <td>-0.021530</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>1.336997</td>\n      <td>0.772816</td>\n      <td>0.064191</td>\n      <td>0.113824</td>\n      <td>0.274960</td>\n      <td>0.584009</td>\n      <td>0.250691</td>\n      <td>0.131507</td>\n      <td>-0.086427</td>\n      <td>1.035331</td>\n      <td>0.963734</td>\n      <td>-0.071943</td>\n      <td>0.137122</td>\n      <td>0.236478</td>\n      <td>0.207081</td>\n      <td>0.541823</td>\n      <td>0.774360</td>\n      <td>-0.050299</td>\n      <td>0.333631</td>\n      <td>0.083555</td>\n      <td>1.143493</td>\n      <td>0.329835</td>\n      <td>0.153495</td>\n      <td>0.100726</td>\n      <td>0.863559</td>\n      <td>0.240538</td>\n      <td>0.033295</td>\n      <td>0.120760</td>\n      <td>0.037955</td>\n      <td>-0.000212</td>\n      <td>0.460581</td>\n      <td>1.260663</td>\n      <td>1.906905</td>\n      <td>1.002758</td>\n      <td>-0.016437</td>\n      <td>...</td>\n      <td>0.000060</td>\n      <td>-0.006795</td>\n      <td>0.031475</td>\n      <td>0.008180</td>\n      <td>-0.018854</td>\n      <td>-0.016969</td>\n      <td>0.000780</td>\n      <td>-0.004706</td>\n      <td>0.000969</td>\n      <td>-0.003761</td>\n      <td>-0.006051</td>\n      <td>0.007059</td>\n      <td>0.008471</td>\n      <td>-0.005795</td>\n      <td>-0.003362</td>\n      <td>0.000330</td>\n      <td>-0.009427</td>\n      <td>-0.014938</td>\n      <td>0.004236</td>\n      <td>0.011295</td>\n      <td>0.002824</td>\n      <td>0.004236</td>\n      <td>0.002824</td>\n      <td>0.002824</td>\n      <td>-0.005661</td>\n      <td>-0.004403</td>\n      <td>-0.003774</td>\n      <td>-0.005032</td>\n      <td>-0.005032</td>\n      <td>-0.004403</td>\n      <td>-0.005032</td>\n      <td>-0.004403</td>\n      <td>-0.004403</td>\n      <td>-0.004403</td>\n      <td>-0.006112</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 9724 columns</p>\n</div>"
     },
     "metadata": {},
     "execution_count": 77
    }
   ],
   "source": [
    "preds = pd.DataFrame(all_user_predicted_ratings, columns = ratings_pivot.columns)\n",
    "preds.head()"
   ]
  },
  {
   "source": [
    "Now we write the main function  to work with predictions.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_movies(predictions, userID, movies, original_ratings, num_recommendations):\n",
    "    \"\"\"\n",
    "    :param \n",
    "    predictions : The SVD reconstructed matrix, \n",
    "    userID : UserId for which you want to predict the top rated movies, \n",
    "    movies : Matrix with movie data, \n",
    "    original_ratings : Original Rating matrix, \n",
    "    num_recommendations : num of records to be returned\n",
    "    :return: num_recommendations top movies\n",
    "    \"\"\" \n",
    "    # Get and sort the user's predictions\n",
    "    user_row_number = userID - 1 # User ID starts at 1, not 0\n",
    "    # get all the values per movie for specified user\n",
    "    sorted_user_predictions = predictions.iloc[user_row_number].sort_values(ascending=False)\n",
    "        \n",
    "    # Get the user's data and merge in the movie information.\n",
    "    user_data = original_ratings[original_ratings.userId == userID]\n",
    "    user_full = user_data.merge(movies, how = 'left', left_on = 'movieId', right_on = 'movieId').sort_values(['rating'], ascending=False)\n",
    "                 \n",
    "    \n",
    "    print('User {0} has already rated {1} movies.'.format(userID, user_full.shape[0]))\n",
    "    print('Recommending highest {0} predicted ratings movies not already rated.'.format(num_recommendations))\n",
    "    \n",
    "    # Recommend the highest predicted rating movies that the user hasn't seen yet.\n",
    "    recommendations = (movies[~movies['movieId'].isin(user_full['movieId'])].\n",
    "         merge(pd.DataFrame(sorted_user_predictions).reset_index(), how = 'left',\n",
    "               left_on = 'movieId',\n",
    "               right_on = 'movieId').\n",
    "         rename(columns = {user_row_number: 'Predictions'}).\n",
    "         sort_values('Predictions', ascending = False).\n",
    "                       iloc[:num_recommendations, :-1]\n",
    "                      )\n",
    "\n",
    "    return recommendations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "User 611 has already rated 3 movies.\nRecommending highest 20 predicted ratings movies not already rated.\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "      movieId                                              title                                   genres\n",
       "898      1198  Raiders of the Lost Ark (Indiana Jones and the...                         Action|Adventure\n",
       "1936     2571                                 Matrix, The (1999)                   Action|Sci-Fi|Thriller\n",
       "966      1270                          Back to the Future (1985)                  Adventure|Comedy|Sci-Fi\n",
       "506       589                  Terminator 2: Judgment Day (1991)                            Action|Sci-Fi\n",
       "614       780               Independence Day (a.k.a. ID4) (1996)         Action|Adventure|Sci-Fi|Thriller\n",
       "987      1291          Indiana Jones and the Last Crusade (1989)                         Action|Adventure\n",
       "658       858                              Godfather, The (1972)                              Crime|Drama\n",
       "897      1197                         Princess Bride, The (1987)  Action|Adventure|Comedy|Fantasy|Romance\n",
       "900      1200                                      Aliens (1986)           Action|Adventure|Horror|Sci-Fi\n",
       "936      1240                             Terminator, The (1984)                   Action|Sci-Fi|Thriller\n",
       "519       608                                       Fargo (1996)              Comedy|Crime|Drama|Thriller\n",
       "3635     4993  Lord of the Rings: The Fellowship of the Ring,...                        Adventure|Fantasy\n",
       "591       733                                   Rock, The (1996)                Action|Adventure|Thriller\n",
       "1500     2028                         Saving Private Ryan (1998)                         Action|Drama|War\n",
       "792      1036                                    Die Hard (1988)                    Action|Crime|Thriller\n",
       "1041     1356                    Star Trek: First Contact (1996)         Action|Adventure|Sci-Fi|Thriller\n",
       "1976     2628   Star Wars: Episode I - The Phantom Menace (1999)                  Action|Adventure|Sci-Fi\n",
       "5893    33493  Star Wars: Episode III - Revenge of the Sith (...                  Action|Adventure|Sci-Fi\n",
       "814      1073         Willy Wonka & the Chocolate Factory (1971)          Children|Comedy|Fantasy|Musical\n",
       "4134     5952      Lord of the Rings: The Two Towers, The (2002)                        Adventure|Fantasy"
      ],
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>movieId</th>\n      <th>title</th>\n      <th>genres</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>898</th>\n      <td>1198</td>\n      <td>Raiders of the Lost Ark (Indiana Jones and the...</td>\n      <td>Action|Adventure</td>\n    </tr>\n    <tr>\n      <th>1936</th>\n      <td>2571</td>\n      <td>Matrix, The (1999)</td>\n      <td>Action|Sci-Fi|Thriller</td>\n    </tr>\n    <tr>\n      <th>966</th>\n      <td>1270</td>\n      <td>Back to the Future (1985)</td>\n      <td>Adventure|Comedy|Sci-Fi</td>\n    </tr>\n    <tr>\n      <th>506</th>\n      <td>589</td>\n      <td>Terminator 2: Judgment Day (1991)</td>\n      <td>Action|Sci-Fi</td>\n    </tr>\n    <tr>\n      <th>614</th>\n      <td>780</td>\n      <td>Independence Day (a.k.a. ID4) (1996)</td>\n      <td>Action|Adventure|Sci-Fi|Thriller</td>\n    </tr>\n    <tr>\n      <th>987</th>\n      <td>1291</td>\n      <td>Indiana Jones and the Last Crusade (1989)</td>\n      <td>Action|Adventure</td>\n    </tr>\n    <tr>\n      <th>658</th>\n      <td>858</td>\n      <td>Godfather, The (1972)</td>\n      <td>Crime|Drama</td>\n    </tr>\n    <tr>\n      <th>897</th>\n      <td>1197</td>\n      <td>Princess Bride, The (1987)</td>\n      <td>Action|Adventure|Comedy|Fantasy|Romance</td>\n    </tr>\n    <tr>\n      <th>900</th>\n      <td>1200</td>\n      <td>Aliens (1986)</td>\n      <td>Action|Adventure|Horror|Sci-Fi</td>\n    </tr>\n    <tr>\n      <th>936</th>\n      <td>1240</td>\n      <td>Terminator, The (1984)</td>\n      <td>Action|Sci-Fi|Thriller</td>\n    </tr>\n    <tr>\n      <th>519</th>\n      <td>608</td>\n      <td>Fargo (1996)</td>\n      <td>Comedy|Crime|Drama|Thriller</td>\n    </tr>\n    <tr>\n      <th>3635</th>\n      <td>4993</td>\n      <td>Lord of the Rings: The Fellowship of the Ring,...</td>\n      <td>Adventure|Fantasy</td>\n    </tr>\n    <tr>\n      <th>591</th>\n      <td>733</td>\n      <td>Rock, The (1996)</td>\n      <td>Action|Adventure|Thriller</td>\n    </tr>\n    <tr>\n      <th>1500</th>\n      <td>2028</td>\n      <td>Saving Private Ryan (1998)</td>\n      <td>Action|Drama|War</td>\n    </tr>\n    <tr>\n      <th>792</th>\n      <td>1036</td>\n      <td>Die Hard (1988)</td>\n      <td>Action|Crime|Thriller</td>\n    </tr>\n    <tr>\n      <th>1041</th>\n      <td>1356</td>\n      <td>Star Trek: First Contact (1996)</td>\n      <td>Action|Adventure|Sci-Fi|Thriller</td>\n    </tr>\n    <tr>\n      <th>1976</th>\n      <td>2628</td>\n      <td>Star Wars: Episode I - The Phantom Menace (1999)</td>\n      <td>Action|Adventure|Sci-Fi</td>\n    </tr>\n    <tr>\n      <th>5893</th>\n      <td>33493</td>\n      <td>Star Wars: Episode III - Revenge of the Sith (...</td>\n      <td>Action|Adventure|Sci-Fi</td>\n    </tr>\n    <tr>\n      <th>814</th>\n      <td>1073</td>\n      <td>Willy Wonka &amp; the Chocolate Factory (1971)</td>\n      <td>Children|Comedy|Fantasy|Musical</td>\n    </tr>\n    <tr>\n      <th>4134</th>\n      <td>5952</td>\n      <td>Lord of the Rings: The Two Towers, The (2002)</td>\n      <td>Adventure|Fantasy</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "metadata": {},
     "execution_count": 79
    }
   ],
   "source": [
    "# get 20 recommendations\n",
    "recommend_movies(preds, my_user_id, movies, ratings, 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}