How_to_Scrape_Quote_Tweets_with_Python.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "How to Scrape Quote Tweets with Python.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "tuwROg2ogm6W"
      ],
      "authorship_tag": "ABX9TyMsRTtbPdtlWqSVNp8BznEh",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/SeyiAgboola/Extract-Tweets-7-days/blob/master/How_to_Scrape_Quote_Tweets_with_Python.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# How to Scrape Quote Tweets with Python\n",
        "\n",
        "This notebook will show you how to track quote tweets of an original tweet of your choice. Quote tweeting is now just as common as tweet replies and hold important information of the sentimental response to original tweets.\n",
        "\n",
        "Main steps within this Notebook are to:\n",
        "\n",
        "* Authenticate your Twitter API Access\n",
        "* Search for the Quote Tweets based on Tweet ID\n",
        "* Store resulting Quote Tweets in a CSV file\n",
        "\n",
        "The main use cases with this notebook are to:\n",
        "\n",
        "* Return tweets from last 7 days based on search term\n",
        "* Identify most popular topic per hashtag\n",
        "* Return Quote tweets of a specific tweet\n",
        "* Sentimental analysis of response to a tweet\n",
        "\n",
        "If you're looking for a new platform to learn platform, you might want to consider DataCamp which has courses for everything to do with Python, SQL and other data related programming languages and software tool. For example:\n",
        "\n",
        "* [Web Scraping in Python](https://datacamp.pxf.io/3PraYy)\n",
        "* [Introduction to Data Science in Python](https://datacamp.pxf.io/NK3JXP)\n",
        "* [Introduction to Deep Learning in Python](https://datacamp.pxf.io/Yg3YLK)\n",
        "* And practice solving real-world problems [with guided and unguided projects](https://datacamp.pxf.io/qngna5)\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "bMecNe5hgpM3"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "gB2OA4XKNxhV"
      },
      "outputs": [],
      "source": [
        "import tweepy\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import re"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Authenticate your Twitter API Access\n",
        "\n",
        "You will need a Developer API Access credentials to scrape tweets from Twitter via Python. I won't go into detail on that process since there are plenty of tutorials that can explain this but I will provide some links to assist with this.\n",
        "\n",
        "* [Developer API site](https://developer.twitter.com/en)\n",
        "* [Tutorial on getting Twitter API access](https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api)\n",
        "\n",
        "Once you have access I recommend you create a file to access your credentials locally on your laptop or PC. I have used a csv file since I am familiar with manipulating pandas dataframes including pulling specific data out.\n",
        "\n",
        "In this case, I created a column for each access detail needed including:\n",
        "* Client ID\n",
        "* Client Secret\n",
        "* API Key ([Also known as Consumer Key](https://developer.twitter.com/en/docs/authentication/oauth-1-0a/api-key-and-secret))\n",
        "* API Key Secret ([Also known as Consumer Secret](https://developer.twitter.com/en/docs/authentication/oauth-1-0a/api-key-and-secret))\n",
        "* Bearer Token\n",
        "* Access Token ([Known as Token credentials to grant access and can be revoked](https://developer.twitter.com/en/docs/authentication/oauth-1-0a/obtaining-user-access-tokens))\n",
        "* Access Token Secret\n",
        "\n",
        "I would be lying to you if I said I knew why we need all of these but these variables are needed for the tweepy functions to work."
      ],
      "metadata": {
        "id": "4O5u8C2QvNDO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Store credentials in a DataFrame to be accessed directly via pandas\n",
        "df = pd.read_csv(\"/content/twicreds - Sheet1.csv\")\n",
        "\n",
        "client_id = df['Client ID'][0]\n",
        "client_secret = df['Client Secret'][0]\n",
        "#The API Key and Secret (also known as Consumer Key and Secret)\n",
        "consumer_key = df['API Key'][0]\n",
        "consumer_secret = df['API Key Secret'][0]\n",
        "bearer_token = df['Bearer Token'][0]\n",
        "access_token = df['Access Token'][0]\n",
        "access_secret = df['Access Token Secret'][0]"
      ],
      "metadata": {
        "id": "Llp6EMwVOQfw"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Create auth object\n",
        "authenticate = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
        "#Set the access token and access token secret\n",
        "authenticate.set_access_token(access_token, access_secret)\n",
        "#Create the API object\n",
        "api = tweepy.API(authenticate, wait_on_rate_limit=True)"
      ],
      "metadata": {
        "id": "LBfEpa0mXG6d"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Things to know when working with Twitter's API\n",
        "\n",
        "There are a few limitations depending on the level of developer access you have. In this case I'm going to assume that you have Essential or Elevated Access which are both free and require minimum criteria to gain. These are the things to be aware of: \n",
        "\n",
        "* Based on your access level, you have a monthly tweet cap. For Essential it's 500K and for Elevated it's [2 million Tweets per month](https://developer.twitter.com/en/docs/twitter-api/tweet-caps)\n",
        "* When searching for tweets you will get more results than you need so you might want to adjust the result_type parameter. Although the tweepy documentation does not list result_type in its parameters, it is an available parameter in the Twitter Dev Docs. Here is the definition of [popular result_type](#https://docs.tweepy.org/en/stable/api.html).\n",
        "* Watch out for guides and tutorials that refer to different versions of the Twitter API. This one for example, [Comprehensive Guide on Using Twitter API V2](https://dev.to/twitterdev/a-comprehensive-guide-for-using-the-twitter-api-v2-using-tweepy-in-python-15d9) - is based on using Tweepy.Client that requires support for Twitter API v2 which is still in development on the master branch.\n",
        "* Beware that the Twitter’s standard search API only “searches against a sampling of recent Tweets [published in the past 7 days.](https://docs.tweepy.org/en/stable/api.html#search-tweets)”. This means you cannot take your search results as gospel. There will be valuable results missing and should be treated as a sample of a bigger population.\n",
        "* Like mentioned above you will only receive Tweets from the last 7 days. This means searching for tweets older than that will not result any accurate results. If you have access to the [Academic Research Product Track](https://developer.twitter.com/en/products/twitter-api/academic-research), you can get Tweets older than 7 days\n",
        "* Direct link to the [Search API tweepy documentation](https://docs.tweepy.org/en/stable/client.html#search-tweets)\n",
        "* Twitter official documentation on [building search queries](https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query)\n",
        "* One common query clause is -is:retweet, which will not match on Retweets, thus matching only on original Tweets, Quote Tweets, and replies\n",
        "\n",
        "Errors to watch out for:\n",
        "\n",
        "* TweepError: [{'message': 'You currently have Essential access which includes access to Twitter API v2 endpoints only. \n",
        "If you need access to this endpoint, you’ll need to apply for Elevated access via the Developer Portal. You can learn more here: https://developer.twitter.com/en/docs/twitter-api/getting-started/about-twitter-api#v2-access-leve', 'code': 453}]\n",
        "* TweepError: Read-only application cannot POST.\n",
        "* AttributeError: module 'tweepy' has no attribute 'Client'\n",
        "* TweepError: Twitter error response: status code = 403 "
      ],
      "metadata": {
        "id": "Vh8GIZUJ6AV2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Search for Tweets with PS5Share hashtag\n",
        "\n",
        "Here we run our search query and store the results into a DataFrame which we can store within a CSV file locally."
      ],
      "metadata": {
        "id": "qbU-VSxaV_0d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Create search term\n",
        "search_term = '#PS5Share -is:retweet' #Refer to https://developer.twitter.com/en/docs/twitter-api/tweets/search/integrate/build-a-query to build advanced queries\n",
        "#Create the cursor object\n",
        "tweets = tweepy.Cursor(api.search, #Refer to https://docs.tweepy.org/en/stable/api.html - API.search_tweets\n",
        "                       q = search_term, #The search query string of 500 characters maximum\n",
        "                       lang='en', #Restricts tweets to the given language\n",
        "                       since='2022-05-01',#Beware of the 7 day limit\n",
        "                       tweet_mode = 'extended' \n",
        "                       ).items(2000) #limit max - important for not going over tweet caps limit\n",
        "#Store the tweets\n",
        "all_tweets = [tweet.full_text for tweet in tweets]\n",
        "#Show all tweets\n",
        "print(len(all_tweets))\n",
        "print(all_tweets[:5])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eRu100U8aRDx",
        "outputId": "28bb4881-950d-4ffc-c530-553068400260"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "2000\n",
            "[\"Matching Michael Schumacher's Winning record🙌\\nIf you know you know 😎\\n#PS5Share, #GranTurismo7 https://t.co/DNwRpkfxyn\", '#PS5Share, #Fortnite First win in the new season https://t.co/RokvAPqvmP', 'RT @kingforever008: SHOOT &amp; RIDE | Day 5 |#GG30SILHOUETTE \\n#Cyberpunk2077 #CyberSunday #NPCSunday  #PS5Share #VirtualPhotography #WIGVP #WV…', 'RT @yesmynameissumo: #PS5Share, #ControlUltimateEdition loving the Alan Wake missions... https://t.co/O0G4IdVkSP', 'RT @ZivLisker: Another repost for: \\n #GG30SILHOUETTE ❤️😇📸\\n#AssassinsCreedOrigins @ubisoft\\n #PS5Share #VGPUnite #VirtualPhotography #ZarnGam…']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Dataframe to store tweets\n",
        "df = pd.DataFrame(all_tweets, columns=['Tweets'])\n",
        "df.head()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "uDke4bMLlQs3",
        "outputId": "f88a5a06-59ce-40b8-b529-4923f5f515f8"
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                              Tweets\n",
              "0  Matching Michael Schumacher's Winning record🙌\\...\n",
              "1  #PS5Share, #Fortnite First win in the new seas...\n",
              "2  RT @kingforever008: SHOOT &amp; RIDE | Day 5 |...\n",
              "3  RT @yesmynameissumo: #PS5Share, #ControlUltima...\n",
              "4  RT @ZivLisker: Another repost for: \\n #GG30SIL..."
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-2fa16ca5-8e12-4247-8da1-31dc7f2e721f\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Tweets</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Matching Michael Schumacher's Winning record🙌\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>#PS5Share, #Fortnite First win in the new seas...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>RT @kingforever008: SHOOT &amp;amp; RIDE | Day 5 |...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>RT @yesmynameissumo: #PS5Share, #ControlUltima...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>RT @ZivLisker: Another repost for: \\n #GG30SIL...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2fa16ca5-8e12-4247-8da1-31dc7f2e721f')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-2fa16ca5-8e12-4247-8da1-31dc7f2e721f button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-2fa16ca5-8e12-4247-8da1-31dc7f2e721f');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Function to clean tweets\n",
        "def cleanTweets(tweet):\n",
        "  tweet = re.sub('RT', '', tweet)\n",
        "  #tweet = re.sub('#[A-Za-z0-9]+', '', tweet) #Remove special characters\n",
        "  tweet = re.sub('\\n', '', tweet) #Remove newlines\n",
        "  #tweet = re.sub('https?\\/\\/\\S+', '', tweet) #Remove hyperlinks\n",
        "  tweet = re.sub('@[\\S]*', '', tweet) #Remove mentions\n",
        "  tweet = re.sub('^[\\s]+|[\\s]+$', '', tweet) #Remove whitespaces\n",
        "  return tweet\n"
      ],
      "metadata": {
        "id": "0hQlPHmCll3D"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df['Clean_tweets'] = df['Tweets'].apply(cleanTweets)"
      ],
      "metadata": {
        "id": "1uByOxOxm79_"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df.head(10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 363
        },
        "id": "MQNhsCgjnedf",
        "outputId": "9f8421fd-732e-4386-d9ed-43a4753a4742"
      },
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                              Tweets  \\\n",
              "0  Matching Michael Schumacher's Winning record🙌\\...   \n",
              "1  #PS5Share, #Fortnite First win in the new seas...   \n",
              "2  RT @kingforever008: SHOOT &amp; RIDE | Day 5 |...   \n",
              "3  RT @yesmynameissumo: #PS5Share, #ControlUltima...   \n",
              "4  RT @ZivLisker: Another repost for: \\n #GG30SIL...   \n",
              "5  RT @Travellingpaddy: The Moon Is A Friend , Fo...   \n",
              "6  Confronting fear is the destiny of a Jedi.\\n#P...   \n",
              "7  RT @NuttyRoyale: Feel like I'm always in this ...   \n",
              "8  RT @hatchiedave: Queue in dramatic walkout! An...   \n",
              "9  #PS5Share, #HorizonForbiddenWest Such a cool g...   \n",
              "\n",
              "                                        Clean_tweets  \n",
              "0  Matching Michael Schumacher's Winning record🙌I...  \n",
              "1  #PS5Share, #Fortnite First win in the new seas...  \n",
              "2  SHOOT &amp; RIDE | Day 5 |#GG30SILHOUETTE #Cyb...  \n",
              "3  #PS5Share, #ControlUltimateEdition loving the ...  \n",
              "4  Another repost for:  #GG30SILHOUETTE ❤️😇📸#Assa...  \n",
              "5  The Moon Is A Friend , For The Lonesome To Tal...  \n",
              "6  Confronting fear is the destiny of a Jedi.#PS5...  \n",
              "7  Feel like I'm always in this domain #PS5Share,...  \n",
              "8  Queue in dramatic walkout! And scene! 🎬Tap for...  \n",
              "9  #PS5Share, #HorizonForbiddenWest Such a cool g...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-12081da0-472c-437c-b689-a7151c4d6774\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Tweets</th>\n",
              "      <th>Clean_tweets</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Matching Michael Schumacher's Winning record🙌\\...</td>\n",
              "      <td>Matching Michael Schumacher's Winning record🙌I...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>#PS5Share, #Fortnite First win in the new seas...</td>\n",
              "      <td>#PS5Share, #Fortnite First win in the new seas...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>RT @kingforever008: SHOOT &amp;amp; RIDE | Day 5 |...</td>\n",
              "      <td>SHOOT &amp;amp; RIDE | Day 5 |#GG30SILHOUETTE #Cyb...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>RT @yesmynameissumo: #PS5Share, #ControlUltima...</td>\n",
              "      <td>#PS5Share, #ControlUltimateEdition loving the ...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>RT @ZivLisker: Another repost for: \\n #GG30SIL...</td>\n",
              "      <td>Another repost for:  #GG30SILHOUETTE ❤️😇📸#Assa...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>RT @Travellingpaddy: The Moon Is A Friend , Fo...</td>\n",
              "      <td>The Moon Is A Friend , For The Lonesome To Tal...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Confronting fear is the destiny of a Jedi.\\n#P...</td>\n",
              "      <td>Confronting fear is the destiny of a Jedi.#PS5...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>RT @NuttyRoyale: Feel like I'm always in this ...</td>\n",
              "      <td>Feel like I'm always in this domain #PS5Share,...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>RT @hatchiedave: Queue in dramatic walkout! An...</td>\n",
              "      <td>Queue in dramatic walkout! And scene! 🎬Tap for...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>#PS5Share, #HorizonForbiddenWest Such a cool g...</td>\n",
              "      <td>#PS5Share, #HorizonForbiddenWest Such a cool g...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-12081da0-472c-437c-b689-a7151c4d6774')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-12081da0-472c-437c-b689-a7151c4d6774 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-12081da0-472c-437c-b689-a7151c4d6774');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#DataFrame rows and columns before\n",
        "print(df.shape)\n",
        "#Drop duplicates inplace\n",
        "df.drop_duplicates(inplace=True)\n",
        "#DataFrame rows and columns after\n",
        "print(df.shape)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GLQuAWZ5n3hr",
        "outputId": "b11f2c06-b061-4f82-917c-8a1cccc416d0"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "(2000, 2)\n",
            "(1597, 2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Create a list of numbers from 0 to total length of dataframe\n",
        "idx = list(range(0, len(df)))"
      ],
      "metadata": {
        "id": "4mWWhxbUoC8P"
      },
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Reset index\n",
        "df = df.set_index(pd.Index(idx))\n",
        "df.head(10)"
      ],
      "metadata": {
        "id": "wYPOa8m6oKpY"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "filename = 'PS5Share_Since_1stMay_2000_MAX.csv'\n",
        "#Save CSV to Files section\n",
        "df.to_csv(filename)"
      ],
      "metadata": {
        "id": "CcXuxLWC35UX"
      },
      "execution_count": 13,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Things to Know Before Scraping Quote Tweets\n",
        "\n",
        "It took me a while to work out how to get quote tweets. So I have wrote down some things to consider if this is something you require.\n",
        "\n",
        "* Quote tweets or better known as [Retweets with comments are just regular tweets](https://github.com/tweepy/tweepy/issues/1291), with a permalink to another tweet at the end. To find Quote Tweets of specific tweets, you can use the Search API to search for the tweet based on it's id which looks something like this: 1193899515901829120\n",
        "* Enterprise Users (paying customers) have it more easy and can get Quote Tweets (by user or of user) much more easily. See [full documentation here](https://developer.twitter.com/en/docs/twitter-api/enterprise/account-activity-api/overview)\n",
        "* If my code for Quote Tweets doesn't work, [here's another article](https://blog.f-secure.com/processing-quote-tweets-with-twitter-api/) that might work. I didn't use this in the end because Status Errors I couldn't unwrangle.\n"
      ],
      "metadata": {
        "id": "THCQu-p6OOiV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Scraping Quote Tweets\n",
        "\n",
        "This piece of code is a combination of:\n",
        "* [Hackernoon's Tweet Replies Guide](https://hackernoon.com/scraping-tweet-replies-with-python-and-tweepy-twitter-api-a-step-by-step-guide-z11x3yr8)\n",
        "* [Processing Quote Tweet Guide](https://blog.f-secure.com/processing-quote-tweets-with-twitter-api/)\n",
        "* [GeekforGeeks Status Object Status Guide](https://www.geeksforgeeks.org/python-status-object-in-tweepy/).\n",
        "\n",
        "The main steps are:\n",
        "\n",
        "* Identify the tweet you want quote tweets for\n",
        "* Build your search query to pull only tweets with that URL\n",
        "* Cross check tweets are quote tweets for that specific tweet\n",
        "* Store resulting tweets into a list\n",
        "* Store resulting list into a CSV file and save locally"
      ],
      "metadata": {
        "id": "_p8kOHUzWWv9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import csv\n",
        "import tweepy\n",
        "# import ssl\n",
        "\n",
        "#Couldn't tell you what this means\n",
        "# ssl._create_default_https_context = ssl._create_unverified_context\n",
        "\n",
        "# Authentication with Twitter\n",
        "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
        "auth.set_access_token(access_token, access_secret)\n",
        "api = tweepy.API(auth)\n",
        "\n",
        "#Insert your Tweet details here including the Account Name + Tweet ID + Tweet URL\n",
        "name = 'PlayStation' #The name of the twitter account that tweeted the original tweet\n",
        "tweet_id = '1532491343116656640' #That long number at the end of the Tweet URL is the Tweet ID\n",
        "tweet_url = \"https://twitter.com/PlayStation/status/1532491343116656640\" #Full Tweet URL\n",
        "\n",
        "replies=[] #replies to original tweet\n",
        "quotes=[] #tweets with true quote status\n",
        "matching_quotes = [] #tweet\n",
        "quote_ids = [] #tweet id of tweet being quoted\n",
        "objects = [] #tweet metadata\n",
        "all_tweets=[] #all tweets returned\n",
        "for tweet in tweepy.Cursor(api.search,q=\"url:\"+tweet_url, result_type='recent', timeout=999999).items(3000):\n",
        "  #Checks if tweet is a tweet reply to the specific tweet\n",
        "    all_tweets.append(tweet)\n",
        "    if hasattr(tweet, 'in_reply_to_status_id_str'):\n",
        "        if (tweet.in_reply_to_status_id_str==tweet_id):\n",
        "            replies.append(tweet)\n",
        "    #if hasattr(tweet, 'quoted_status'):\n",
        "    if tweet.is_quote_status == True:\n",
        "      quotes.append(tweet) #Full Tweet objects with all the meta data\n",
        "      quote_ids.append(tweet.quoted_status_id) #These should all be the exact same\n",
        "      if str(tweet_id) == str(tweet.quoted_status_id):\n",
        "        objects.append(tweet)\n",
        "        matching_quotes.append(tweet.text)\n"
      ],
      "metadata": {
        "id": "nDrLLiJfjoJr"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "print(\"Lengths\")\n",
        "print(\"Total tweets: \" + str(len(all_tweets)))\n",
        "print(\"Total quotes tweets: \" + str(len(quotes)))\n",
        "print(\"Total matching quote texts: \" + str(len(matching_quotes)))\n",
        "print(\"Total quote ids: \" + str(len(quote_ids)))\n",
        "print(\"Total matching quote objects: \" + str(len(objects)))\n",
        "#This is a extra check to make sure we've only pulled quote tweets that are related to the original tweet"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YQEL8WVnWDjN",
        "outputId": "6d95750e-12fa-438e-c7a5-75e7ee4d39b3"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Lengths\n",
            "Total tweets: 100\n",
            "Total quotes tweets: 16\n",
            "Total matching quote texts: 16\n",
            "Total quote ids: 16\n",
            "Total matching quote objects: 16\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "filename = name + \"_\" + tweet_id + \"_\" + 'QTs.csv'\n",
        "#Create a live file\n",
        "with open(filename, 'w') as f:\n",
        "  #Create CSV file with 2 columns\n",
        "    csv_writer = csv.DictWriter(f, fieldnames=('Tweet Text', 'Metadata'))\n",
        "    #Add a header\n",
        "    csv_writer.writeheader()\n",
        "    #For each entity in objects\n",
        "    for tweet in objects:\n",
        "      #Create row of current tweet and insert these values\n",
        "        row = {'Tweet Text': tweet.text, 'Metadata': tweet}\n",
        "      #Add row to current CSV\n",
        "        csv_writer.writerow(row)"
      ],
      "metadata": {
        "id": "nXueemWa8anc"
      },
      "execution_count": 16,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Debugging and Troubleshooting Tweepy"
      ],
      "metadata": {
        "id": "RF1YvMJrWPPx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#Check the first Quote Tweet Text\n",
        "matching_quotes[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "2eyHQIWn2t1w",
        "outputId": "2c1d2700-15ad-48c7-e5dc-b3bbce10d7d2"
      },
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Siii que ganitas dios!!!! pal verano que viene ya sabéis lo que tocará en los directos 😏😏😏😏 https://t.co/hdDht206q6'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#Check first object in list\n",
        "objects[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uz7yG-yC79pE",
        "outputId": "285d1826-472f-4a05-a205-e05e91a16dab"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Status(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'created_at': 'Sat Jun 04 14:56:44 +0000 2022', 'id': 1533100445736087555, 'id_str': '1533100445736087555', 'text': 'Siii que ganitas dios!!!! pal verano que viene ya sabéis lo que tocará en los directos 😏😏😏😏 https://t.co/hdDht206q6', 'truncated': False, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/hdDht206q6', 'expanded_url': 'https://twitter.com/PlayStation/status/1532491343116656640', 'display_url': 'twitter.com/PlayStation/st…', 'indices': [92, 115]}]}, 'metadata': {'iso_language_code': 'es', 'result_type': 'recent'}, 'source': '<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 1678128758, 'id_str': '1678128758', 'name': 'KRN Lexone23', 'screen_name': 'lexone23', 'location': 'Vigo, España', 'description': 'Streamer de todo tipo de videojuegos.  😁 Creador de Contenido  @KirinGamingClub 💜\\nContacto:Lexonetsm@gmail.com', 'url': 'https://t.co/NurUg4RnhG', 'entities': {'url': {'urls': [{'url': 'https://t.co/NurUg4RnhG', 'expanded_url': 'https://www.twitch.tv/lexone23', 'display_url': 'twitch.tv/lexone23', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 324, 'friends_count': 337, 'listed_count': 1, 'created_at': 'Sat Aug 17 12:32:46 +0000 2013', 'favourites_count': 4431, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 5179, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/1678128758/1642800675', 'profile_link_color': 'FAB81E', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': '000000', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': True, 'quoted_status_id': 1532491343116656640, 'quoted_status_id_str': '1532491343116656640', 'quoted_status': {'created_at': 'Thu Jun 02 22:36:23 +0000 2022', 'id': 1532491343116656640, 'id_str': '1532491343116656640', 'text': 'Final Fantasy XVI launches on PlayStation 5 summer 2023. \\n\\nWatch the brand-new “Dominance” trailer now:… https://t.co/VqutMbaSma', 'truncated': True, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/VqutMbaSma', 'expanded_url': 'https://twitter.com/i/web/status/1532491343116656640', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [105, 128]}]}, 'metadata': {'iso_language_code': 'es', 'result_type': 'recent'}, 'source': '<a href=\"https://studio.twitter.com\" rel=\"nofollow\">Twitter Media Studio</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 10671602, 'id_str': '10671602', 'name': 'PlayStation', 'screen_name': 'PlayStation', 'location': 'California', 'description': 'Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', 'url': 'https://t.co/OrydDM2CVy', 'entities': {'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 25281257, 'friends_count': 748, 'listed_count': 34434, 'created_at': 'Tue Nov 27 22:37:20 +0000 2007', 'favourites_count': 1353, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': True, 'statuses_count': 40500, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/10671602/1654293541', 'profile_link_color': '0066CC', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': 'A8A8A8', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': True, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 2214, 'favorite_count': 9285, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'retweet_count': 0, 'favorite_count': 1, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'es'}, created_at=datetime.datetime(2022, 6, 4, 14, 56, 44), id=1533100445736087555, id_str='1533100445736087555', text='Siii que ganitas dios!!!! pal verano que viene ya sabéis lo que tocará en los directos 😏😏😏😏 https://t.co/hdDht206q6', truncated=False, entities={'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/hdDht206q6', 'expanded_url': 'https://twitter.com/PlayStation/status/1532491343116656640', 'display_url': 'twitter.com/PlayStation/st…', 'indices': [92, 115]}]}, metadata={'iso_language_code': 'es', 'result_type': 'recent'}, source='Twitter for Android', source_url='http://twitter.com/download/android', in_reply_to_status_id=None, in_reply_to_status_id_str=None, in_reply_to_user_id=None, in_reply_to_user_id_str=None, in_reply_to_screen_name=None, author=User(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'id': 1678128758, 'id_str': '1678128758', 'name': 'KRN Lexone23', 'screen_name': 'lexone23', 'location': 'Vigo, España', 'description': 'Streamer de todo tipo de videojuegos.  😁 Creador de Contenido  @KirinGamingClub 💜\\nContacto:Lexonetsm@gmail.com', 'url': 'https://t.co/NurUg4RnhG', 'entities': {'url': {'urls': [{'url': 'https://t.co/NurUg4RnhG', 'expanded_url': 'https://www.twitch.tv/lexone23', 'display_url': 'twitch.tv/lexone23', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 324, 'friends_count': 337, 'listed_count': 1, 'created_at': 'Sat Aug 17 12:32:46 +0000 2013', 'favourites_count': 4431, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 5179, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/1678128758/1642800675', 'profile_link_color': 'FAB81E', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': '000000', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, id=1678128758, id_str='1678128758', name='KRN Lexone23', screen_name='lexone23', location='Vigo, España', description='Streamer de todo tipo de videojuegos.  😁 Creador de Contenido  @KirinGamingClub 💜\\nContacto:Lexonetsm@gmail.com', url='https://t.co/NurUg4RnhG', entities={'url': {'urls': [{'url': 'https://t.co/NurUg4RnhG', 'expanded_url': 'https://www.twitch.tv/lexone23', 'display_url': 'twitch.tv/lexone23', 'indices': [0, 23]}]}, 'description': {'urls': []}}, protected=False, followers_count=324, friends_count=337, listed_count=1, created_at=datetime.datetime(2013, 8, 17, 12, 32, 46), favourites_count=4431, utc_offset=None, time_zone=None, geo_enabled=False, verified=False, statuses_count=5179, lang=None, contributors_enabled=False, is_translator=False, is_translation_enabled=False, profile_background_color='000000', profile_background_image_url='http://abs.twimg.com/images/themes/theme1/bg.png', profile_background_image_url_https='https://abs.twimg.com/images/themes/theme1/bg.png', profile_background_tile=False, profile_image_url='http://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', profile_image_url_https='https://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', profile_banner_url='https://pbs.twimg.com/profile_banners/1678128758/1642800675', profile_link_color='FAB81E', profile_sidebar_border_color='000000', profile_sidebar_fill_color='000000', profile_text_color='000000', profile_use_background_image=False, has_extended_profile=True, default_profile=False, default_profile_image=False, following=False, follow_request_sent=False, notifications=False, translator_type='none', withheld_in_countries=[]), user=User(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'id': 1678128758, 'id_str': '1678128758', 'name': 'KRN Lexone23', 'screen_name': 'lexone23', 'location': 'Vigo, España', 'description': 'Streamer de todo tipo de videojuegos.  😁 Creador de Contenido  @KirinGamingClub 💜\\nContacto:Lexonetsm@gmail.com', 'url': 'https://t.co/NurUg4RnhG', 'entities': {'url': {'urls': [{'url': 'https://t.co/NurUg4RnhG', 'expanded_url': 'https://www.twitch.tv/lexone23', 'display_url': 'twitch.tv/lexone23', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 324, 'friends_count': 337, 'listed_count': 1, 'created_at': 'Sat Aug 17 12:32:46 +0000 2013', 'favourites_count': 4431, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 5179, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/1678128758/1642800675', 'profile_link_color': 'FAB81E', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': '000000', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, id=1678128758, id_str='1678128758', name='KRN Lexone23', screen_name='lexone23', location='Vigo, España', description='Streamer de todo tipo de videojuegos.  😁 Creador de Contenido  @KirinGamingClub 💜\\nContacto:Lexonetsm@gmail.com', url='https://t.co/NurUg4RnhG', entities={'url': {'urls': [{'url': 'https://t.co/NurUg4RnhG', 'expanded_url': 'https://www.twitch.tv/lexone23', 'display_url': 'twitch.tv/lexone23', 'indices': [0, 23]}]}, 'description': {'urls': []}}, protected=False, followers_count=324, friends_count=337, listed_count=1, created_at=datetime.datetime(2013, 8, 17, 12, 32, 46), favourites_count=4431, utc_offset=None, time_zone=None, geo_enabled=False, verified=False, statuses_count=5179, lang=None, contributors_enabled=False, is_translator=False, is_translation_enabled=False, profile_background_color='000000', profile_background_image_url='http://abs.twimg.com/images/themes/theme1/bg.png', profile_background_image_url_https='https://abs.twimg.com/images/themes/theme1/bg.png', profile_background_tile=False, profile_image_url='http://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', profile_image_url_https='https://pbs.twimg.com/profile_images/1528737142658281473/3L_DGYRn_normal.jpg', profile_banner_url='https://pbs.twimg.com/profile_banners/1678128758/1642800675', profile_link_color='FAB81E', profile_sidebar_border_color='000000', profile_sidebar_fill_color='000000', profile_text_color='000000', profile_use_background_image=False, has_extended_profile=True, default_profile=False, default_profile_image=False, following=False, follow_request_sent=False, notifications=False, translator_type='none', withheld_in_countries=[]), geo=None, coordinates=None, place=None, contributors=None, is_quote_status=True, quoted_status_id=1532491343116656640, quoted_status_id_str='1532491343116656640', quoted_status=Status(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'created_at': 'Thu Jun 02 22:36:23 +0000 2022', 'id': 1532491343116656640, 'id_str': '1532491343116656640', 'text': 'Final Fantasy XVI launches on PlayStation 5 summer 2023. \\n\\nWatch the brand-new “Dominance” trailer now:… https://t.co/VqutMbaSma', 'truncated': True, 'entities': {'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/VqutMbaSma', 'expanded_url': 'https://twitter.com/i/web/status/1532491343116656640', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [105, 128]}]}, 'metadata': {'iso_language_code': 'es', 'result_type': 'recent'}, 'source': '<a href=\"https://studio.twitter.com\" rel=\"nofollow\">Twitter Media Studio</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 10671602, 'id_str': '10671602', 'name': 'PlayStation', 'screen_name': 'PlayStation', 'location': 'California', 'description': 'Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', 'url': 'https://t.co/OrydDM2CVy', 'entities': {'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 25281257, 'friends_count': 748, 'listed_count': 34434, 'created_at': 'Tue Nov 27 22:37:20 +0000 2007', 'favourites_count': 1353, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': True, 'statuses_count': 40500, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/10671602/1654293541', 'profile_link_color': '0066CC', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': 'A8A8A8', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': True, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 2214, 'favorite_count': 9285, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, created_at=datetime.datetime(2022, 6, 2, 22, 36, 23), id=1532491343116656640, id_str='1532491343116656640', text='Final Fantasy XVI launches on PlayStation 5 summer 2023. \\n\\nWatch the brand-new “Dominance” trailer now:… https://t.co/VqutMbaSma', truncated=True, entities={'hashtags': [], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/VqutMbaSma', 'expanded_url': 'https://twitter.com/i/web/status/1532491343116656640', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [105, 128]}]}, metadata={'iso_language_code': 'es', 'result_type': 'recent'}, source='Twitter Media Studio', source_url='https://studio.twitter.com', in_reply_to_status_id=None, in_reply_to_status_id_str=None, in_reply_to_user_id=None, in_reply_to_user_id_str=None, in_reply_to_screen_name=None, author=User(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'id': 10671602, 'id_str': '10671602', 'name': 'PlayStation', 'screen_name': 'PlayStation', 'location': 'California', 'description': 'Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', 'url': 'https://t.co/OrydDM2CVy', 'entities': {'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 25281257, 'friends_count': 748, 'listed_count': 34434, 'created_at': 'Tue Nov 27 22:37:20 +0000 2007', 'favourites_count': 1353, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': True, 'statuses_count': 40500, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/10671602/1654293541', 'profile_link_color': '0066CC', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': 'A8A8A8', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': True, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, id=10671602, id_str='10671602', name='PlayStation', screen_name='PlayStation', location='California', description='Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', url='https://t.co/OrydDM2CVy', entities={'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, protected=False, followers_count=25281257, friends_count=748, listed_count=34434, created_at=datetime.datetime(2007, 11, 27, 22, 37, 20), favourites_count=1353, utc_offset=None, time_zone=None, geo_enabled=True, verified=True, statuses_count=40500, lang=None, contributors_enabled=False, is_translator=False, is_translation_enabled=False, profile_background_color='000000', profile_background_image_url='http://abs.twimg.com/images/themes/theme1/bg.png', profile_background_image_url_https='https://abs.twimg.com/images/themes/theme1/bg.png', profile_background_tile=False, profile_image_url='http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', profile_image_url_https='https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', profile_banner_url='https://pbs.twimg.com/profile_banners/10671602/1654293541', profile_link_color='0066CC', profile_sidebar_border_color='000000', profile_sidebar_fill_color='A8A8A8', profile_text_color='000000', profile_use_background_image=False, has_extended_profile=True, default_profile=False, default_profile_image=False, following=True, follow_request_sent=False, notifications=False, translator_type='none', withheld_in_countries=[]), user=User(_api=<tweepy.api.API object at 0x7f609ad555d0>, _json={'id': 10671602, 'id_str': '10671602', 'name': 'PlayStation', 'screen_name': 'PlayStation', 'location': 'California', 'description': 'Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', 'url': 'https://t.co/OrydDM2CVy', 'entities': {'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 25281257, 'friends_count': 748, 'listed_count': 34434, 'created_at': 'Tue Nov 27 22:37:20 +0000 2007', 'favourites_count': 1353, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': True, 'statuses_count': 40500, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/10671602/1654293541', 'profile_link_color': '0066CC', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': 'A8A8A8', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': True, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none', 'withheld_in_countries': []}, id=10671602, id_str='10671602', name='PlayStation', screen_name='PlayStation', location='California', description='Official Twitter updates on PlayStation, PS5, PS4, PS VR, PlayStation Plus and more. Support: @AskPlayStation', url='https://t.co/OrydDM2CVy', entities={'url': {'urls': [{'url': 'https://t.co/OrydDM2CVy', 'expanded_url': 'https://www.playstation.com', 'display_url': 'playstation.com', 'indices': [0, 23]}]}, 'description': {'urls': []}}, protected=False, followers_count=25281257, friends_count=748, listed_count=34434, created_at=datetime.datetime(2007, 11, 27, 22, 37, 20), favourites_count=1353, utc_offset=None, time_zone=None, geo_enabled=True, verified=True, statuses_count=40500, lang=None, contributors_enabled=False, is_translator=False, is_translation_enabled=False, profile_background_color='000000', profile_background_image_url='http://abs.twimg.com/images/themes/theme1/bg.png', profile_background_image_url_https='https://abs.twimg.com/images/themes/theme1/bg.png', profile_background_tile=False, profile_image_url='http://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', profile_image_url_https='https://pbs.twimg.com/profile_images/1278183948279922690/ybnDHXn7_normal.jpg', profile_banner_url='https://pbs.twimg.com/profile_banners/10671602/1654293541', profile_link_color='0066CC', profile_sidebar_border_color='000000', profile_sidebar_fill_color='A8A8A8', profile_text_color='000000', profile_use_background_image=False, has_extended_profile=True, default_profile=False, default_profile_image=False, following=True, follow_request_sent=False, notifications=False, translator_type='none', withheld_in_countries=[]), geo=None, coordinates=None, place=None, contributors=None, is_quote_status=False, retweet_count=2214, favorite_count=9285, favorited=False, retweeted=False, possibly_sensitive=False, lang='en'), retweet_count=0, favorite_count=1, favorited=False, retweeted=False, possibly_sensitive=False, lang='es')"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "  # full_tweet = tweet.text\n",
        "  # original_url = parse_qTweet(full_tweet)\n",
        "  # url_splits = original_url.split(\"/\")\n",
        "  # if url_splits[-1] == tweet_id:\n",
        "    # quotes.append(tweet.text)"
      ],
      "metadata": {
        "id": "9g0VRXG68hXB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "* How to use requests to [parse open truncated twitter links](https://stackoverflow.com/questions/8872232/how-can-i-unwrap-t-co-links-with-python) which can be a tweet, video or photo"
      ],
      "metadata": {
        "id": "ZaSJVU0Pi782"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#How to unwrap a t.co link\n",
        "import requests\n",
        "from re import search\n",
        "def parse_qTweet(text):\n",
        "  words = text.split()\n",
        "  print(len(words))\n",
        "  for word in words:\n",
        "    word = str(word)\n",
        "    # if 'https://t.co' in word:\n",
        "    #   print(\"True\")\n",
        "    #   full_url = requests.get(word).url\n",
        "    #   return full_url\n",
        "    if search('https://t.co', word):\n",
        "      full_url = requests.get(word).url\n",
        "      return full_url\n",
        "    if word.find('https://t.co') != -1:\n",
        "      print(\"Found!\")\n",
        "    else:\n",
        "      return \"Invalid\"\n",
        "\n",
        "yo = parse_qTweet(matching_quotes[0])\n",
        "example = matching_quotes[0]\n",
        "splits = example.split()\n",
        "splits\n",
        "for word in splits:\n",
        "    if 'https://t.co' in word:\n",
        "        try: \n",
        "            full_url = requests.get(word).url\n",
        "            print(full_url)\n",
        "        except:\n",
        "          print('Nah')\n",
        "full_url\n",
        "yo"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 0
        },
        "id": "_Roz3lZ0HYuA",
        "outputId": "c8d1119e-9cf3-4003-8255-57e5381769c5"
      },
      "execution_count": 43,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "18\n",
            "https://twitter.com/PlayStation/status/1532491343116656640\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Invalid'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 43
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#See all the attributes in the tweet object\n",
        "dir(all_tweets[0])"
      ],
      "metadata": {
        "id": "l659epROUvJW"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# fetching the status\n",
        "example_id = \"1531228577445728257\"\n",
        "status = api.get_status(example_id)"
      ],
      "metadata": {
        "id": "-FqpLToRFLE8"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#Check quote id for a quote tweet matches original tweet\n",
        "quotes[0].quoted_status_id"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eorzvCbCNtrV",
        "outputId": "e566cc51-e550-4f3d-9e5e-c32907809334"
      },
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1532491343116656640"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# printing the information from a tweet object\n",
        "print(\"The status was created at : \" + str(status.created_at))\n",
        "print(\"The id is : \" + str(status.id))\n",
        "print(\"The id_str is : \" + status.id_str)\n",
        "print(\"The text is : \" + status.text)\n",
        "print(\"The source_url is : \" + status.source_url)\n",
        "  \n",
        "  \n",
        "print(\"The in_reply_to_status_id is : \" + str(status.in_reply_to_status_id))\n",
        "print(\"The in_reply_to_user_id is : \" + str(status.in_reply_to_user_id))\n",
        "print(\"The in_reply_to_screen_name is : \" + str(status.in_reply_to_screen_name))\n",
        "  \n",
        "  \n",
        "print(\"The poster's screen name is : \" + status.user.screen_name)\n",
        "print(\"The is_quote_status is : \" + str(status.is_quote_status))\n",
        "  \n",
        "print(\"Has the authenticated user favourited the status? : \" + str(status.favorited))\n",
        "print(\"Has the authenticated user retweeted the status? \" + str(status.retweeted))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XRsnx6gTFZfT",
        "outputId": "04d0fe79-de48-44c8-b180-4937bb49fa00"
      },
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The status was created at : 2022-05-30 10:58:36\n",
            "The id is : 1531228577445728257\n",
            "The id_str is : 1531228577445728257\n",
            "The text is : Need some more combos https://t.co/Kw2zKAsy5j\n",
            "The source_url is : http://twitter.com/download/android\n",
            "The in_reply_to_status_id is : None\n",
            "The in_reply_to_user_id is : None\n",
            "The in_reply_to_screen_name is : None\n",
            "The poster's screen name is : LiteraryDecay\n",
            "The is_quote_status is : True\n",
            "Has the authenticated user favourited the status? : False\n",
            "Has the authenticated user retweeted the status? False\n"
          ]
        }
      ]
    }
  ]
}