wrangle_report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wrangle Report  \n",
    "\n",
    "\n",
    "## Introduction\n",
    "Twitter user `@dog_rates` rates the people's dog with humorous comment about the dog. The dataset we are wrangling in this project is the tweet archive of the discussed user. Twitter user `@dog_rates` is also known as __WeRateDogs__.\n",
    "\n",
    "\n",
    "This report briefly describes my data wrangling efforts. The tasks for project WeRateDogs Twitter are as follows:\n",
    "- Gathering Data\n",
    "- Assessing Data\n",
    "- Cleaning Data\n",
    "- - -\n",
    "## Gathering Data\n",
    "\n",
    "My wrangling efforts for the WeRateDogs Twitter project included gathering data from the\n",
    "following sources:\n",
    "\n",
    "- The WeRateDogs __Twitter archive__. The `twitter_archive_enhanced.csv` file was provided by Udacity to students. This archive contains basic tweet data (tweet ID, timestamp, text, etc.) for all 5000+ of their tweets as they stood on August 1, 2017. \n",
    "- The __tweet image predictions__, what breed of dog (or another object, animal, etc.) is present in each tweet according to a neural network. The `image_predictions.tsv` file was provided by Udacity which required to be downloaded programmactically via request library and special access URL. \n",
    "- __Twitter API__ and __Python's Tweepy library__ to gather all the data for each tweet ID in WeRateDogs Twitter archive. Gathered the data from API and stored it in `tweet_json.txt`. Then read whole `tweet_json.txt` data line by line and stored some attributes `('favorite_count', 'retweet_count', 'followers_count', 'favourites_count', 'friends_count' and 'date_time')` which I like in a DataFrame for further analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- - - \n",
    "## Assessing Data\n",
    "Once all the data gathered successfuly, I started to assessing the data visually and programmatically both for  the *__quality__* and the *__tidiness__* issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quality Issue\n",
    "The quality issues I found in the data are listed below file wise. \n",
    "\n",
    "- __`Twitter Archive Enhanced Data:`__ `Source` format is bad and can't be read easily.\n",
    "- __`Twitter Archive Enhanced Data:`__ These attributes should be `integers/strings` instead of `float`\n",
    "    - in_reply_to_status_id\n",
    "    - in_reply_to_user_id\n",
    "    - retweeted_status_id\n",
    "    - retweeted_status_user_id\n",
    "- __`Twitter Archive Enhanced Data:`__ Columns should be `datetime` instead of `object(string)`.\n",
    "    - retweeted_status_timestamp\n",
    "    - timestamp\n",
    "- __`Twitter Archive Enhanced Data:`__ We may want to change these columns types to `string` because We don't want any operations on them.\n",
    "    - tweet_id\n",
    "    - in_reply_to_status_id   \n",
    "    - in_reply_to_user_id \n",
    "    - retweeted_status_id \n",
    "    - retweeted_status_user_id\n",
    "- __`Twitter Archive Enhanced Data:`__ The columns `numerator` and `denominator` have invalid values.\n",
    "- __`Twitter Archive Enhanced Data:`__ There are invalid names like (`a, an, etc.`).\n",
    "- __`Twitter Archive Enhanced Data:`__ Convert the `null` values to `None` type\n",
    "- __`Image Predictions Data:`__ `tweet_id` should be `object` instead of `integer`, as no calculation is needed.\n",
    "- __`Image Predictions Data:`__ Missing values in _`image_predictions`_ dataset. __2075__ rows of data compared to _`twitter_archive_enhanced`_ dataset which has __2356__ number of records.\n",
    "- __`Image Predictions Data:`__ Some tweets have 2 different `tweet_id`, that are actually retweets.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tidiness Issue\n",
    "The tidiness issues I found in the data are listed below file wise. \n",
    "\n",
    "- __`Twitter Archive Enhanced Data:`__ No need to divide Dog stages in 4 different columns like \n",
    "    - 'doggo'\n",
    "    - 'floofer'\n",
    "    - 'pupper'\n",
    "    - 'puppo'\n",
    "-  __`Image Predictions Data:`__  _`Image Predictions Data`_ should be joined to _`Twitter Archive Enhanced Data`_\n",
    "-  __`Tweets (API) JSON Data:`__ Merge _`Tweets (API) JSON Data`_ with the _`Twitter Archive Enhanced Data`_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- - - \n",
    "## Cleaning Data\n",
    "After the successful assessment, cleaned the data on the basis of *__Define, Code and Test__* for each of the issue addressed in assessing data section. Some data cleaning steps are listed below. \n",
    "\n",
    "- Create a copy of each original dataframe.\n",
    "- Merge the copied version of archive, images, and tweet_json dataframes\n",
    "- Correct the dog types.\n",
    "- Create one column for the various dog types: doggo, floofer, pupper, puppo \n",
    "- Delete retweets.\n",
    "- Remove columns no longer needed: \n",
    "    - 'retweeted_status_id'\n",
    "    - 'retweeted_status_user_id'\n",
    "    - 'retweeted_status_timestamp'\n",
    "    - 'date_time'\n",
    "    - 'friends_count'\n",
    "    - 'img_num'\n",
    "    - 'p1'\n",
    "    - 'p1_conf'\n",
    "    - 'p1_dog'\n",
    "    - 'p2'\n",
    "    - 'p2_conf'\n",
    "    - 'p2_dog'\n",
    "    - 'p3'\n",
    "    - 'p3_conf'\n",
    "    - 'p3_dog'\n",
    "    - 'in_reply_to_status_id'\n",
    "    - 'in_reply_to_user_id'\n",
    "    - 'favourites_count'\n",
    "- Make Source format is good and can read easily.\n",
    "- The columns numerator and denominator have invalid values.\n",
    "- Cleaning decimal values in rating numerators.\n",
    "- Convert the null values to None type\n",
    "- Correcting Data types\n",
    "    - 'tweet_id' as type str\n",
    "    - 'timestamp' as type datetime\n",
    "    - 'source' as type category\n",
    "    - 'rating_numerator' as type float\n",
    "    - 'rating_denominator' as type float\n",
    "    - 'favorite_count' as type int\n",
    "    - 'retweet_count' as type int\n",
    "    - 'followers_count' as type int\n",
    "    - 'dog_stage' as type category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Conclusion\n",
    "\n",
    "Data wrangling is a core skill that whoever handles data should be familiar with. \n",
    "I have used Python programming language and some of its packages. There are several advantages of this tool (as compared to e.g. Excel or Sheets) that is used by many data scientists (including the Google, Amazon and Facebook).\n",
    "\n",
    "- It can deal with a large variety of data either structured data from ERP/SQL databases or unstructured data (like JSON or NoSQL Databases).\n",
    "- It is strong in dealing with big data (much better than Excel, Sheets).\n",
    "- For gathering data there are several packages that help scraping data off the web, either using APIs to collect data (Tweepy for Twitter) or to communicate with SQL databases.\n",
    "- It is easy to document each single step and if needed re-run each single step. Thus, one can leave a perfect audit trail.\n",
    "- One can re-run analysis automatically every period.\n",
    "- Handling, assessing, cleaning and visualizing of data is possible programmatically using code."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}