02_Pre-processing_limited_merged.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e1cd8741-e48e-4327-9db6-ac4ebfc5dde8",
   "metadata": {},
   "source": [
    "# Pre-processing - Syrian and Ukrainian inflows\n",
    "**Author**: Andrea Cass"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e5f369f-50d4-4b07-941f-1becb5d9636e",
   "metadata": {},
   "source": [
    "## 1. About this notebook\n",
    "\n",
    "The purpose of this Jupyter notebook is to pre-process the data collected in the 01_Data-collection Notebooks:\n",
    "> *01a_Data-Collection_limited_Syrian_eng.csv*\n",
    "\n",
    "> *01b_Data-Collection_limited_Syrian_de.csv*\n",
    "\n",
    "> *01a_Data-Collection_limited_Ukrainian_eng.csv*\n",
    "\n",
    "> *01b_Data-Collection_limited_Ukrainian_de.csv*\n",
    "\n",
    "If you did not collect this data yourself but instead received the files from me, please ensure these files are moved to a folder called \"CASS_thesis\" after the folder is created in section **3.2. CASS_thesis** of this Notebook. To reiterate, after you finish section **3.2. CASS_thesis**, please move the files I sent you into the folder, \"CASS_thesis\".\n",
    "\n",
    "***\n",
    "\n",
    "Goals:\n",
    "* Format dates\n",
    "* Create new 'language' column\n",
    "* Merge English- and German-language datasets into one\n",
    "* Create new 'inflow' column\n",
    "* Merge Syrian inflow and Ukrainian inflow datasets into one\n",
    "* Dropping unnecessary columns\n",
    "\n",
    "The output will be a single dataset saved as a csv filed titled,\n",
    "> *02_Pre-processed_limited_merged.csv*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4121f57d-cdad-4213-8f1d-b9afb6557461",
   "metadata": {},
   "source": [
    "## 2. Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a7c7a217-c692-4b19-962e-1e4966ef6772",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from textblob import TextBlob\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "from matplotlib import pyplot as plt\n",
    "from matplotlib import dates as mpl_dates\n",
    "from datetime import datetime, timedelta\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from textblob import Word\n",
    "from datetime import datetime as dt\n",
    "import os\n",
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17580792-a3c2-48bb-aeee-f9992a2c867c",
   "metadata": {},
   "source": [
    "## 3. Working directory & file paths\n",
    "\n",
    "Before beginning data pre-processing, the working directory needs to be set up. Additionally, if you did not already use the Notebook titled \"01_Data-Collection_Syrian\" to create a folder called \"CASS_thesis\", code is provided here to do so. \n",
    "\n",
    "Two objects will be named:\n",
    "* **cwd**: the current working directory (e.g., your Desktop)\n",
    "* **CASS_thesis**: the folder where all data from my Notebooks will be saved"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7727b0b-14eb-452b-9a79-bc1bd42d70d6",
   "metadata": {},
   "source": [
    "### 3.1. Current working directory\n",
    "Use the code below to find out what your current working directory is set to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "836400d9-379b-4d26-8b4e-93db0e8bcaf8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/andycass/Jupyterlab_main-folder/THESIS'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# find current working directory\n",
    "\n",
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e35b8554-5ca6-409c-945e-4c48f1436752",
   "metadata": {},
   "source": [
    "If your current working directory is not your desired directory, follow the subsequent steps to change the working diectory by:\n",
    "1. deciding where you would like your working directory to be (e.g., your Desktop)\n",
    "2. entering the file path of your desired working directory into the code below\n",
    "\n",
    "**NOTE**: If you are satisfied with your working directory and do NOT wish to change it, skip the block of code underneath **3.1.1. Changing current working directory** and, instead, proceed from the block of code underneath **3.1.2. Naming current working directory**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b41c5d8b-ba69-495c-bb22-2bdcb869b4e5",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 3.1.1. Changing current working directory\n",
    "**NOTE**: The code below contains the path to **my** desired working directory to serve as an example. You must alter it to the path of **your** desired working directory. Keep in mind that my example is formatted according to Macbook standards, and Windows formatting differs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "2fd17b1c-5eca-4533-a53a-30f00a881159",
   "metadata": {},
   "outputs": [],
   "source": [
    "# changing current working directory\n",
    "\n",
    "os.chdir('/Users/andycass/Desktop/Thesis_data-and-code')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9fe35f3-7598-44d6-bac6-7719d3797d60",
   "metadata": {},
   "source": [
    "#### 3.1.2. Naming current working directory\n",
    "Now that your current working directory is established, use the code below to name it \"cwd\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b385e217-2a51-4c30-9948-94620eb3af2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the current working directory\n",
    "\n",
    "cwd = Path.cwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a390114b-d06c-45bd-8ea0-6fd89e8caab3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the current working directory location\n",
    "\n",
    "cwd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "642698cd-bd3c-47c8-9b75-43362fcb8a08",
   "metadata": {},
   "source": [
    "### 3.2 CASS_thesis\n",
    "You may or may not have already created a folder named \"CASS_thesis\" depending on whether you ran the code from the first Data Collection Notebook. Nevertheless, the code below will work in either case. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "304754bb-a0a2-4752-972a-7fb72e348bb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# naming the CASS_thesis folder\n",
    "\n",
    "CASS_thesis = cwd / 'CASS_thesis'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dffca52-01bf-4517-b07b-9968bae87b8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# creating the CASS_thesis folder\n",
    "\n",
    "CASS_thesis.mkdir(exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "de782030-0c42-4d27-8b28-3754afff747a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/Users/andycass/Desktop/Thesis_data-and-code/CASS_thesis')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking the CASS_thesis location\n",
    "\n",
    "CASS_thesis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31451f62-e311-411e-b54f-8c36031d7e77",
   "metadata": {},
   "source": [
    "## 4. Syrian inflow datasets\n",
    "**NOTE**: Before proceeding, ensure that the folder called \"CASS_thesis\" contains the following files:\n",
    "> *01a_Data-Collection_limited_Syrian_eng.csv*\n",
    "\n",
    "> *01b_Data-Collection_limited_Syrian_de.csv*\n",
    "\n",
    "> *01a_Data-Collection_limited_Ukrainian_eng.csv*\n",
    "\n",
    "> *01b_Data-Collection_limited_Ukrainian_de.csv*\n",
    "\n",
    "### 4.1. English-language dataset\n",
    "#### 4.1.1 Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "0e54f998-b52a-4327-b418-b325d6fb79b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_eng = pd.read_csv(CASS_thesis / \"01a_Data-Collection_limited_Syrian-eng.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "be8ac3f6-77a3-4fa5-8cfb-09a692811f03",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>lang</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>created_at</th>\n",
       "      <th>author_id</th>\n",
       "      <th>id</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>edit_controls.edits_remaining</th>\n",
       "      <th>...</th>\n",
       "      <th>attachments.media_keys</th>\n",
       "      <th>geo.coordinates.type</th>\n",
       "      <th>geo.coordinates.coordinates</th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>referenced_tweets</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>['722876040876580864']</td>\n",
       "      <td>en</td>\n",
       "      <td>UNHCR - Survivors report massive loss of life ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20T19:54:13.000Z</td>\n",
       "      <td>339833759</td>\n",
       "      <td>722876040876580864</td>\n",
       "      <td>722876040876580864</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>['722802163114733572']</td>\n",
       "      <td>en</td>\n",
       "      <td>Syrian artists are painting bright murals in t...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20T15:00:39.000Z</td>\n",
       "      <td>412624794</td>\n",
       "      <td>722802163114733572</td>\n",
       "      <td>722802163114733572</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>['722712144354144256']</td>\n",
       "      <td>en</td>\n",
       "      <td>Paul Guest is an excellent moderator at the co...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20T09:02:57.000Z</td>\n",
       "      <td>2863752725</td>\n",
       "      <td>722712144354144256</td>\n",
       "      <td>722712144354144256</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>['3_722697412549193728']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>['722592172663394304']</td>\n",
       "      <td>en</td>\n",
       "      <td>DEN : According to a Jerusalem Post review of ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20T01:06:13.000Z</td>\n",
       "      <td>186899860</td>\n",
       "      <td>722592172663394304</td>\n",
       "      <td>722592172663394304</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.46757813, 52.5913198]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>['722520855218008065']</td>\n",
       "      <td>en</td>\n",
       "      <td>Why #europe are sending #syrian #refugees to #...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-19T20:22:50.000Z</td>\n",
       "      <td>69418092</td>\n",
       "      <td>722520855218008065</td>\n",
       "      <td>722520855218008065</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.0667, 52.4]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5165</th>\n",
       "      <td>False</td>\n",
       "      <td>['547694737776320513']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : Syrian President Bashar Assad, shown...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-24T10:06:15.000Z</td>\n",
       "      <td>186899860</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.46757813, 52.5913198]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5166</th>\n",
       "      <td>False</td>\n",
       "      <td>['547609017623654400']</td>\n",
       "      <td>en</td>\n",
       "      <td>#CelioGermanyDesk German President Gauck calls...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-24T04:25:38.000Z</td>\n",
       "      <td>271776179</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[6.75600452, 51.22732903]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5167</th>\n",
       "      <td>False</td>\n",
       "      <td>['547102939484286976']</td>\n",
       "      <td>en</td>\n",
       "      <td>#munich is colourful! welcome #refugees! no #p...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-22T18:54:40.000Z</td>\n",
       "      <td>15105772</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>['3_547102922203353088']</td>\n",
       "      <td>Point</td>\n",
       "      <td>[11.54845144, 48.12402972]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5168</th>\n",
       "      <td>False</td>\n",
       "      <td>['546699388316184577']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : There is little to break the monoton...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-21T16:11:06.000Z</td>\n",
       "      <td>186899860</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.46757813, 52.5913198]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5169</th>\n",
       "      <td>False</td>\n",
       "      <td>['546544368169918464']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : View of a vacant lot earmarked for r...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-21T05:55:06.000Z</td>\n",
       "      <td>186899860</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.46757813, 52.5913198]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5170 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      possibly_sensitive  edit_history_tweet_ids lang  \\\n",
       "0                  False  ['722876040876580864']   en   \n",
       "1                  False  ['722802163114733572']   en   \n",
       "2                  False  ['722712144354144256']   en   \n",
       "3                  False  ['722592172663394304']   en   \n",
       "4                  False  ['722520855218008065']   en   \n",
       "...                  ...                     ...  ...   \n",
       "5165               False  ['547694737776320513']   en   \n",
       "5166               False  ['547609017623654400']   en   \n",
       "5167               False  ['547102939484286976']   en   \n",
       "5168               False  ['546699388316184577']   en   \n",
       "5169               False  ['546544368169918464']   en   \n",
       "\n",
       "                                                   text reply_settings  \\\n",
       "0     UNHCR - Survivors report massive loss of life ...       everyone   \n",
       "1     Syrian artists are painting bright murals in t...       everyone   \n",
       "2     Paul Guest is an excellent moderator at the co...       everyone   \n",
       "3     DEN : According to a Jerusalem Post review of ...       everyone   \n",
       "4     Why #europe are sending #syrian #refugees to #...       everyone   \n",
       "...                                                 ...            ...   \n",
       "5165  DE-News : Syrian President Bashar Assad, shown...       everyone   \n",
       "5166  #CelioGermanyDesk German President Gauck calls...       everyone   \n",
       "5167  #munich is colourful! welcome #refugees! no #p...       everyone   \n",
       "5168  DE-News : There is little to break the monoton...       everyone   \n",
       "5169  DE-News : View of a vacant lot earmarked for r...       everyone   \n",
       "\n",
       "                    created_at   author_id                  id  \\\n",
       "0     2016-04-20T19:54:13.000Z   339833759  722876040876580864   \n",
       "1     2016-04-20T15:00:39.000Z   412624794  722802163114733572   \n",
       "2     2016-04-20T09:02:57.000Z  2863752725  722712144354144256   \n",
       "3     2016-04-20T01:06:13.000Z   186899860  722592172663394304   \n",
       "4     2016-04-19T20:22:50.000Z    69418092  722520855218008065   \n",
       "...                        ...         ...                 ...   \n",
       "5165  2014-12-24T10:06:15.000Z   186899860  547694737776320513   \n",
       "5166  2014-12-24T04:25:38.000Z   271776179  547609017623654400   \n",
       "5167  2014-12-22T18:54:40.000Z    15105772  547102939484286976   \n",
       "5168  2014-12-21T16:11:06.000Z   186899860  546699388316184577   \n",
       "5169  2014-12-21T05:55:06.000Z   186899860  546544368169918464   \n",
       "\n",
       "         conversation_id  edit_controls.edits_remaining  ...  \\\n",
       "0     722876040876580864                              5  ...   \n",
       "1     722802163114733572                              5  ...   \n",
       "2     722712144354144256                              5  ...   \n",
       "3     722592172663394304                              5  ...   \n",
       "4     722520855218008065                              5  ...   \n",
       "...                  ...                            ...  ...   \n",
       "5165  547694737776320513                              5  ...   \n",
       "5166  547609017623654400                              5  ...   \n",
       "5167  547102939484286976                              5  ...   \n",
       "5168  546699388316184577                              5  ...   \n",
       "5169  546544368169918464                              5  ...   \n",
       "\n",
       "        attachments.media_keys geo.coordinates.type  \\\n",
       "0                          NaN                  NaN   \n",
       "1                          NaN                  NaN   \n",
       "2     ['3_722697412549193728']                  NaN   \n",
       "3                          NaN                Point   \n",
       "4                          NaN                Point   \n",
       "...                        ...                  ...   \n",
       "5165                       NaN                Point   \n",
       "5166                       NaN                Point   \n",
       "5167  ['3_547102922203353088']                Point   \n",
       "5168                       NaN                Point   \n",
       "5169                       NaN                Point   \n",
       "\n",
       "     geo.coordinates.coordinates context_annotations referenced_tweets  \\\n",
       "0                            NaN                 NaN               NaN   \n",
       "1                            NaN                 NaN               NaN   \n",
       "2                            NaN                 NaN               NaN   \n",
       "3      [13.46757813, 52.5913198]                 NaN               NaN   \n",
       "4                [13.0667, 52.4]                 NaN               NaN   \n",
       "...                          ...                 ...               ...   \n",
       "5165   [13.46757813, 52.5913198]                 NaN               NaN   \n",
       "5166   [6.75600452, 51.22732903]                 NaN               NaN   \n",
       "5167  [11.54845144, 48.12402972]                 NaN               NaN   \n",
       "5168   [13.46757813, 52.5913198]                 NaN               NaN   \n",
       "5169   [13.46757813, 52.5913198]                 NaN               NaN   \n",
       "\n",
       "     in_reply_to_user_id  attachments.poll_ids  withheld.copyright  \\\n",
       "0                    NaN                   NaN                 NaN   \n",
       "1                    NaN                   NaN                 NaN   \n",
       "2                    NaN                   NaN                 NaN   \n",
       "3                    NaN                   NaN                 NaN   \n",
       "4                    NaN                   NaN                 NaN   \n",
       "...                  ...                   ...                 ...   \n",
       "5165                 NaN                   NaN                 NaN   \n",
       "5166                 NaN                   NaN                 NaN   \n",
       "5167                 NaN                   NaN                 NaN   \n",
       "5168                 NaN                   NaN                 NaN   \n",
       "5169                 NaN                   NaN                 NaN   \n",
       "\n",
       "      withheld.country_codes  withheld.scope  \n",
       "0                        NaN             NaN  \n",
       "1                        NaN             NaN  \n",
       "2                        NaN             NaN  \n",
       "3                        NaN             NaN  \n",
       "4                        NaN             NaN  \n",
       "...                      ...             ...  \n",
       "5165                     NaN             NaN  \n",
       "5166                     NaN             NaN  \n",
       "5167                     NaN             NaN  \n",
       "5168                     NaN             NaN  \n",
       "5169                     NaN             NaN  \n",
       "\n",
       "[5170 rows x 32 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_eng"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc4e9bc-b9e0-46a1-a311-e3d4f1bec1b4",
   "metadata": {},
   "source": [
    "#### 4.1.3 Formatting dates \n",
    "The created_at column, containing information about when the tweet was posted, will be converted to datetime format and normalized so that new columns (e.g., 'year-month') can be derived from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "43dacd2b-54f6-489b-b043-517279856adc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting created_at to datetime format\n",
    "\n",
    "df_eng[\"created_at\"] = pd.to_datetime(df_eng[\"created_at\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0f31e671-272a-4c07-9a23-e0b2be0cd02d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting it to date and creating a new column called \"date\"\n",
    "\n",
    "df_eng['date'] = df_eng['created_at'].dt.normalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4738e57b-6ac3-49bc-a8a0-cea781a14612",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_77946/3519539084.py:3: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.\n",
      "  df_eng['week'] = df_eng['created_at'].dt.week\n"
     ]
    }
   ],
   "source": [
    "# creating week, month, year, year-week, and year-month columns\n",
    "\n",
    "df_eng['week'] = df_eng['created_at'].dt.week\n",
    "df_eng['month'] = df_eng['created_at'].dt.month\n",
    "df_eng['year'] = df_eng['created_at'].dt.year\n",
    "df_eng['year-week'] = df_eng['created_at'].dt.strftime('%Y-%U')\n",
    "df_eng['year-month'] = df_eng['created_at'].dt.strftime('%Y-%m')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "9981e064-e08c-4320-8e32-202a4a024c32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>lang</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>created_at</th>\n",
       "      <th>author_id</th>\n",
       "      <th>id</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>edit_controls.edits_remaining</th>\n",
       "      <th>...</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "      <th>date</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>['722876040876580864']</td>\n",
       "      <td>en</td>\n",
       "      <td>UNHCR - Survivors report massive loss of life ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20 19:54:13+00:00</td>\n",
       "      <td>339833759</td>\n",
       "      <td>722876040876580864</td>\n",
       "      <td>722876040876580864</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>['722802163114733572']</td>\n",
       "      <td>en</td>\n",
       "      <td>Syrian artists are painting bright murals in t...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20 15:00:39+00:00</td>\n",
       "      <td>412624794</td>\n",
       "      <td>722802163114733572</td>\n",
       "      <td>722802163114733572</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>['722712144354144256']</td>\n",
       "      <td>en</td>\n",
       "      <td>Paul Guest is an excellent moderator at the co...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20 09:02:57+00:00</td>\n",
       "      <td>2863752725</td>\n",
       "      <td>722712144354144256</td>\n",
       "      <td>722712144354144256</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>['722592172663394304']</td>\n",
       "      <td>en</td>\n",
       "      <td>DEN : According to a Jerusalem Post review of ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-20 01:06:13+00:00</td>\n",
       "      <td>186899860</td>\n",
       "      <td>722592172663394304</td>\n",
       "      <td>722592172663394304</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>['722520855218008065']</td>\n",
       "      <td>en</td>\n",
       "      <td>Why #europe are sending #syrian #refugees to #...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2016-04-19 20:22:50+00:00</td>\n",
       "      <td>69418092</td>\n",
       "      <td>722520855218008065</td>\n",
       "      <td>722520855218008065</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2016-04-19 00:00:00+00:00</td>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>2016</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5165</th>\n",
       "      <td>False</td>\n",
       "      <td>['547694737776320513']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : Syrian President Bashar Assad, shown...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-24 10:06:15+00:00</td>\n",
       "      <td>186899860</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>52</td>\n",
       "      <td>12</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5166</th>\n",
       "      <td>False</td>\n",
       "      <td>['547609017623654400']</td>\n",
       "      <td>en</td>\n",
       "      <td>#CelioGermanyDesk German President Gauck calls...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-24 04:25:38+00:00</td>\n",
       "      <td>271776179</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>52</td>\n",
       "      <td>12</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5167</th>\n",
       "      <td>False</td>\n",
       "      <td>['547102939484286976']</td>\n",
       "      <td>en</td>\n",
       "      <td>#munich is colourful! welcome #refugees! no #p...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-22 18:54:40+00:00</td>\n",
       "      <td>15105772</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014-12-22 00:00:00+00:00</td>\n",
       "      <td>52</td>\n",
       "      <td>12</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5168</th>\n",
       "      <td>False</td>\n",
       "      <td>['546699388316184577']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : There is little to break the monoton...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-21 16:11:06+00:00</td>\n",
       "      <td>186899860</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>51</td>\n",
       "      <td>12</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5169</th>\n",
       "      <td>False</td>\n",
       "      <td>['546544368169918464']</td>\n",
       "      <td>en</td>\n",
       "      <td>DE-News : View of a vacant lot earmarked for r...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>2014-12-21 05:55:06+00:00</td>\n",
       "      <td>186899860</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>5</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>51</td>\n",
       "      <td>12</td>\n",
       "      <td>2014</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5170 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      possibly_sensitive  edit_history_tweet_ids lang  \\\n",
       "0                  False  ['722876040876580864']   en   \n",
       "1                  False  ['722802163114733572']   en   \n",
       "2                  False  ['722712144354144256']   en   \n",
       "3                  False  ['722592172663394304']   en   \n",
       "4                  False  ['722520855218008065']   en   \n",
       "...                  ...                     ...  ...   \n",
       "5165               False  ['547694737776320513']   en   \n",
       "5166               False  ['547609017623654400']   en   \n",
       "5167               False  ['547102939484286976']   en   \n",
       "5168               False  ['546699388316184577']   en   \n",
       "5169               False  ['546544368169918464']   en   \n",
       "\n",
       "                                                   text reply_settings  \\\n",
       "0     UNHCR - Survivors report massive loss of life ...       everyone   \n",
       "1     Syrian artists are painting bright murals in t...       everyone   \n",
       "2     Paul Guest is an excellent moderator at the co...       everyone   \n",
       "3     DEN : According to a Jerusalem Post review of ...       everyone   \n",
       "4     Why #europe are sending #syrian #refugees to #...       everyone   \n",
       "...                                                 ...            ...   \n",
       "5165  DE-News : Syrian President Bashar Assad, shown...       everyone   \n",
       "5166  #CelioGermanyDesk German President Gauck calls...       everyone   \n",
       "5167  #munich is colourful! welcome #refugees! no #p...       everyone   \n",
       "5168  DE-News : There is little to break the monoton...       everyone   \n",
       "5169  DE-News : View of a vacant lot earmarked for r...       everyone   \n",
       "\n",
       "                    created_at   author_id                  id  \\\n",
       "0    2016-04-20 19:54:13+00:00   339833759  722876040876580864   \n",
       "1    2016-04-20 15:00:39+00:00   412624794  722802163114733572   \n",
       "2    2016-04-20 09:02:57+00:00  2863752725  722712144354144256   \n",
       "3    2016-04-20 01:06:13+00:00   186899860  722592172663394304   \n",
       "4    2016-04-19 20:22:50+00:00    69418092  722520855218008065   \n",
       "...                        ...         ...                 ...   \n",
       "5165 2014-12-24 10:06:15+00:00   186899860  547694737776320513   \n",
       "5166 2014-12-24 04:25:38+00:00   271776179  547609017623654400   \n",
       "5167 2014-12-22 18:54:40+00:00    15105772  547102939484286976   \n",
       "5168 2014-12-21 16:11:06+00:00   186899860  546699388316184577   \n",
       "5169 2014-12-21 05:55:06+00:00   186899860  546544368169918464   \n",
       "\n",
       "         conversation_id  edit_controls.edits_remaining  ...  \\\n",
       "0     722876040876580864                              5  ...   \n",
       "1     722802163114733572                              5  ...   \n",
       "2     722712144354144256                              5  ...   \n",
       "3     722592172663394304                              5  ...   \n",
       "4     722520855218008065                              5  ...   \n",
       "...                  ...                            ...  ...   \n",
       "5165  547694737776320513                              5  ...   \n",
       "5166  547609017623654400                              5  ...   \n",
       "5167  547102939484286976                              5  ...   \n",
       "5168  546699388316184577                              5  ...   \n",
       "5169  546544368169918464                              5  ...   \n",
       "\n",
       "      attachments.poll_ids withheld.copyright withheld.country_codes  \\\n",
       "0                      NaN                NaN                    NaN   \n",
       "1                      NaN                NaN                    NaN   \n",
       "2                      NaN                NaN                    NaN   \n",
       "3                      NaN                NaN                    NaN   \n",
       "4                      NaN                NaN                    NaN   \n",
       "...                    ...                ...                    ...   \n",
       "5165                   NaN                NaN                    NaN   \n",
       "5166                   NaN                NaN                    NaN   \n",
       "5167                   NaN                NaN                    NaN   \n",
       "5168                   NaN                NaN                    NaN   \n",
       "5169                   NaN                NaN                    NaN   \n",
       "\n",
       "     withheld.scope                      date week  month  year  year-week  \\\n",
       "0               NaN 2016-04-20 00:00:00+00:00   16      4  2016    2016-16   \n",
       "1               NaN 2016-04-20 00:00:00+00:00   16      4  2016    2016-16   \n",
       "2               NaN 2016-04-20 00:00:00+00:00   16      4  2016    2016-16   \n",
       "3               NaN 2016-04-20 00:00:00+00:00   16      4  2016    2016-16   \n",
       "4               NaN 2016-04-19 00:00:00+00:00   16      4  2016    2016-16   \n",
       "...             ...                       ...  ...    ...   ...        ...   \n",
       "5165            NaN 2014-12-24 00:00:00+00:00   52     12  2014    2014-51   \n",
       "5166            NaN 2014-12-24 00:00:00+00:00   52     12  2014    2014-51   \n",
       "5167            NaN 2014-12-22 00:00:00+00:00   52     12  2014    2014-51   \n",
       "5168            NaN 2014-12-21 00:00:00+00:00   51     12  2014    2014-51   \n",
       "5169            NaN 2014-12-21 00:00:00+00:00   51     12  2014    2014-51   \n",
       "\n",
       "      year-month  \n",
       "0        2016-04  \n",
       "1        2016-04  \n",
       "2        2016-04  \n",
       "3        2016-04  \n",
       "4        2016-04  \n",
       "...          ...  \n",
       "5165     2014-12  \n",
       "5166     2014-12  \n",
       "5167     2014-12  \n",
       "5168     2014-12  \n",
       "5169     2014-12  \n",
       "\n",
       "[5170 rows x 38 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_eng"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "162e5648-15b8-427f-8396-e67eb03456ad",
   "metadata": {},
   "source": [
    "### 4.2. German-language dataset\n",
    "#### 4.2.1 Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bfe07636-9eb6-49c6-ad58-1b1d0f8470f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de = pd.read_csv(CASS_thesis / \"01b_Data-Collection_limited_Syrian-de.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b01ca32e-4917-4f4d-a730-448eb46a2fe6",
   "metadata": {},
   "source": [
    "#### 4.2.2 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "622ca5c6-93cb-4b87-901a-a42d68deb1e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>...</th>\n",
       "      <th>geo.coordinates.type</th>\n",
       "      <th>geo.coordinates.coordinates</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>referenced_tweets</th>\n",
       "      <th>entities.hashtags</th>\n",
       "      <th>attachments.media_keys</th>\n",
       "      <th>entities.mentions</th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>entities.cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.229216e+17</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>722921572810366977</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>['722921572810366977']</td>\n",
       "      <td>2016-04-20T22:55:08.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228995e+17</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>722899547039473665</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>['722899547039473665']</td>\n",
       "      <td>2016-04-20T21:27:37.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>Point</td>\n",
       "      <td>[7.1468836, 50.7306348]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228974e+17</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>722897370313195521</td>\n",
       "      <td>224607633</td>\n",
       "      <td>['722897370313195521']</td>\n",
       "      <td>2016-04-20T21:18:58.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228536e+17</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>722847691101880320</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>['722853635751809025']</td>\n",
       "      <td>2016-04-20T18:25:11.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 83, 'end': 106, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '722847691101880...</td>\n",
       "      <td>[{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]</td>\n",
       "      <td>['7_722852972233912321']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228240e+17</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>722824011063799809</td>\n",
       "      <td>606265303</td>\n",
       "      <td>['722824011063799809']</td>\n",
       "      <td>2016-04-20T16:27:28.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...</td>\n",
       "      <td>['3_722824009939886080']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19458</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.467140e+17</td>\n",
       "      <td>Stehe am Bahnhof und eine Frau erklärt ihren(?...</td>\n",
       "      <td>546714009261858816</td>\n",
       "      <td>19926396</td>\n",
       "      <td>['546714009261858816']</td>\n",
       "      <td>2014-12-21T17:09:11.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 107, 'end': 122, 'tag': 'ichkönntko...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19459</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.467096e+17</td>\n",
       "      <td>Fast 500 Flüchtlinge, Obdachlose und Heimkinde...</td>\n",
       "      <td>546709582018781184</td>\n",
       "      <td>12358112</td>\n",
       "      <td>['546709582018781184']</td>\n",
       "      <td>2014-12-21T16:51:36.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 91, 'end': 114, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 83, 'end': 89, 'tag': 'Hansa'}]</td>\n",
       "      <td>['3_546709581229879297']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19460</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466898e+17</td>\n",
       "      <td>@OomenBerlin @Rex_Cramer Ostsachsen noPerspekt...</td>\n",
       "      <td>546399944803090432</td>\n",
       "      <td>224651746</td>\n",
       "      <td>['546689796337573888']</td>\n",
       "      <td>2014-12-21T15:32:59.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>75067369</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '546399944803090...</td>\n",
       "      <td>[{'start': 129, 'end': 136, 'tag': 'Pegida'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 0, 'end': 12, 'username': 'OomenBer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19461</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466522e+17</td>\n",
       "      <td>'Nichts gegen Flüchtlinge, aber ein Gefängnis ...</td>\n",
       "      <td>546652160600317952</td>\n",
       "      <td>40870544</td>\n",
       "      <td>['546652160600317952']</td>\n",
       "      <td>2014-12-21T13:03:26.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 75, 'end': 97, 'url': 'http://t.co/...</td>\n",
       "      <td>...</td>\n",
       "      <td>Point</td>\n",
       "      <td>[13.4140765, 52.4883914]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19462</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466389e+17</td>\n",
       "      <td>@MartinSoechting denn diese asoziale Bagage ha...</td>\n",
       "      <td>546637023936016384</td>\n",
       "      <td>1426103292</td>\n",
       "      <td>['546638863075405824']</td>\n",
       "      <td>2014-12-21T12:10:35.000Z</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1960792351</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '546637849320505...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 0, 'end': 16, 'username': 'MartinSo...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19463 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      reply_settings possibly_sensitive            id  \\\n",
       "0           everyone              False  7.229216e+17   \n",
       "1           everyone              False  7.228995e+17   \n",
       "2           everyone              False  7.228974e+17   \n",
       "3           everyone              False  7.228536e+17   \n",
       "4           everyone              False  7.228240e+17   \n",
       "...              ...                ...           ...   \n",
       "19458       everyone              False  5.467140e+17   \n",
       "19459       everyone              False  5.467096e+17   \n",
       "19460       everyone              False  5.466898e+17   \n",
       "19461       everyone              False  5.466522e+17   \n",
       "19462       everyone              False  5.466389e+17   \n",
       "\n",
       "                                                    text     conversation_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  722921572810366977   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  722899547039473665   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  722897370313195521   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  722847691101880320   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  722824011063799809   \n",
       "...                                                  ...                 ...   \n",
       "19458  Stehe am Bahnhof und eine Frau erklärt ihren(?...  546714009261858816   \n",
       "19459  Fast 500 Flüchtlinge, Obdachlose und Heimkinde...  546709582018781184   \n",
       "19460  @OomenBerlin @Rex_Cramer Ostsachsen noPerspekt...  546399944803090432   \n",
       "19461  'Nichts gegen Flüchtlinge, aber ein Gefängnis ...  546652160600317952   \n",
       "19462  @MartinSoechting denn diese asoziale Bagage ha...  546637023936016384   \n",
       "\n",
       "        author_id  edit_history_tweet_ids                created_at lang  \\\n",
       "0      4122038069  ['722921572810366977']  2016-04-20T22:55:08.000Z   de   \n",
       "1      1179543852  ['722899547039473665']  2016-04-20T21:27:37.000Z   de   \n",
       "2       224607633  ['722897370313195521']  2016-04-20T21:18:58.000Z   de   \n",
       "3      2480764313  ['722853635751809025']  2016-04-20T18:25:11.000Z   de   \n",
       "4       606265303  ['722824011063799809']  2016-04-20T16:27:28.000Z   de   \n",
       "...           ...                     ...                       ...  ...   \n",
       "19458    19926396  ['546714009261858816']  2014-12-21T17:09:11.000Z   de   \n",
       "19459    12358112  ['546709582018781184']  2014-12-21T16:51:36.000Z   de   \n",
       "19460   224651746  ['546689796337573888']  2014-12-21T15:32:59.000Z   de   \n",
       "19461    40870544  ['546652160600317952']  2014-12-21T13:03:26.000Z   de   \n",
       "19462  1426103292  ['546638863075405824']  2014-12-21T12:10:35.000Z   de   \n",
       "\n",
       "                                           entities.urls  ...  \\\n",
       "0      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   \n",
       "1                                                    NaN  ...   \n",
       "2                                                    NaN  ...   \n",
       "3      [{'start': 83, 'end': 106, 'url': 'https://t.c...  ...   \n",
       "4      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   \n",
       "...                                                  ...  ...   \n",
       "19458                                                NaN  ...   \n",
       "19459  [{'start': 91, 'end': 114, 'url': 'https://t.c...  ...   \n",
       "19460                                                NaN  ...   \n",
       "19461  [{'start': 75, 'end': 97, 'url': 'http://t.co/...  ...   \n",
       "19462                                                NaN  ...   \n",
       "\n",
       "      geo.coordinates.type  geo.coordinates.coordinates in_reply_to_user_id  \\\n",
       "0                      NaN                          NaN                 NaN   \n",
       "1                    Point      [7.1468836, 50.7306348]                 NaN   \n",
       "2                      NaN                          NaN                 NaN   \n",
       "3                      NaN                          NaN          2480764313   \n",
       "4                      NaN                          NaN                 NaN   \n",
       "...                    ...                          ...                 ...   \n",
       "19458                  NaN                          NaN                 NaN   \n",
       "19459                  NaN                          NaN                 NaN   \n",
       "19460                  NaN                          NaN            75067369   \n",
       "19461                Point     [13.4140765, 52.4883914]                 NaN   \n",
       "19462                  NaN                          NaN          1960792351   \n",
       "\n",
       "                                       referenced_tweets  \\\n",
       "0                                                    NaN   \n",
       "1                                                    NaN   \n",
       "2                                                    NaN   \n",
       "3      [{'type': 'replied_to', 'id': '722847691101880...   \n",
       "4                                                    NaN   \n",
       "...                                                  ...   \n",
       "19458                                                NaN   \n",
       "19459                                                NaN   \n",
       "19460  [{'type': 'replied_to', 'id': '546399944803090...   \n",
       "19461                                                NaN   \n",
       "19462  [{'type': 'replied_to', 'id': '546637849320505...   \n",
       "\n",
       "                                       entities.hashtags  \\\n",
       "0                                                    NaN   \n",
       "1                                                    NaN   \n",
       "2                                                    NaN   \n",
       "3       [{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]   \n",
       "4      [{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...   \n",
       "...                                                  ...   \n",
       "19458  [{'start': 107, 'end': 122, 'tag': 'ichkönntko...   \n",
       "19459         [{'start': 83, 'end': 89, 'tag': 'Hansa'}]   \n",
       "19460      [{'start': 129, 'end': 136, 'tag': 'Pegida'}]   \n",
       "19461                                                NaN   \n",
       "19462                                                NaN   \n",
       "\n",
       "         attachments.media_keys  \\\n",
       "0                           NaN   \n",
       "1                           NaN   \n",
       "2                           NaN   \n",
       "3      ['7_722852972233912321']   \n",
       "4      ['3_722824009939886080']   \n",
       "...                         ...   \n",
       "19458                       NaN   \n",
       "19459  ['3_546709581229879297']   \n",
       "19460                       NaN   \n",
       "19461                       NaN   \n",
       "19462                       NaN   \n",
       "\n",
       "                                       entities.mentions context_annotations  \\\n",
       "0                                                    NaN                 NaN   \n",
       "1                                                    NaN                 NaN   \n",
       "2                                                    NaN                 NaN   \n",
       "3                                                    NaN                 NaN   \n",
       "4                                                    NaN                 NaN   \n",
       "...                                                  ...                 ...   \n",
       "19458                                                NaN                 NaN   \n",
       "19459                                                NaN                 NaN   \n",
       "19460  [{'start': 0, 'end': 12, 'username': 'OomenBer...                 NaN   \n",
       "19461                                                NaN                 NaN   \n",
       "19462  [{'start': 0, 'end': 16, 'username': 'MartinSo...                 NaN   \n",
       "\n",
       "       attachments.poll_ids entities.cashtags  \n",
       "0                       NaN               NaN  \n",
       "1                       NaN               NaN  \n",
       "2                       NaN               NaN  \n",
       "3                       NaN               NaN  \n",
       "4                       NaN               NaN  \n",
       "...                     ...               ...  \n",
       "19458                   NaN               NaN  \n",
       "19459                   NaN               NaN  \n",
       "19460                   NaN               NaN  \n",
       "19461                   NaN               NaN  \n",
       "19462                   NaN               NaN  \n",
       "\n",
       "[19463 rows x 29 columns]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10e5ca73-3fb3-4f4d-8c65-9ff8df5cf856",
   "metadata": {},
   "source": [
    "#### 4.2.3 Formatting dates "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ee235c4-90e2-4e6f-916c-cd19368c98e0",
   "metadata": {},
   "source": [
    "**NOTE**: When trying to convert created at to datetime format, it was discovered that one or more observations had a value of \"5bcd72da50f0ee77\" for created_at, likely due to an error during collection. The following code locates, views, and drops the observation(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0a063f3b-7a8e-4077-a8a7-f96afe5e50d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8643     5bcd72da50f0ee77\n",
       "13733    5bcd72da50f0ee77\n",
       "Name: created_at, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# locating the error\n",
    "\n",
    "df_de.loc[df_de['created_at'] == '5bcd72da50f0ee77', 'created_at']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6fa4628-660c-4ff8-82a7-601e1f7ed895",
   "metadata": {},
   "source": [
    "The output from the code above indicates that the error is located at two indeces:\n",
    "* 8643\n",
    "* 13733"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "6eb4821b-b2d8-4b6d-a512-f0cb6bbdf4a9",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "reply_settings                     Das tut mir sehr leid. Mein Beileid an die Ang...\n",
       "possibly_sensitive                                                662278575119273984\n",
       "id                                                                      2982735723.0\n",
       "text                                                          ['662281747464327169']\n",
       "conversation_id                                             2015-11-05T14:54:07.000Z\n",
       "author_id                                                                         de\n",
       "edit_history_tweet_ids                                                           NaN\n",
       "created_at                                                          5bcd72da50f0ee77\n",
       "lang                                                                               5\n",
       "entities.urls                                                                   True\n",
       "geo.place_id                                                2015-11-05T15:24:07.000Z\n",
       "edit_controls.edits_remaining                                                    0.0\n",
       "edit_controls.is_edit_eligible                                                     0\n",
       "edit_controls.editable_until                                                       1\n",
       "public_metrics.retweet_count                                                     0.0\n",
       "public_metrics.reply_count                                                       0.0\n",
       "public_metrics.like_count                                                      Point\n",
       "public_metrics.quote_count                                [10.04900676, 53.57986559]\n",
       "public_metrics.impression_count                                           15738602.0\n",
       "geo.coordinates.type               [{'type': 'replied_to', 'id': '662278575119273...\n",
       "geo.coordinates.coordinates                                                      NaN\n",
       "in_reply_to_user_id                                                              NaN\n",
       "referenced_tweets                                                                NaN\n",
       "entities.hashtags                                                                NaN\n",
       "attachments.media_keys                                                           NaN\n",
       "entities.mentions                                                                NaN\n",
       "context_annotations                                                              NaN\n",
       "attachments.poll_ids                                                             NaN\n",
       "entities.cashtags                                                                NaN\n",
       "Name: 8643, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the first index\n",
    "\n",
    "df_de.iloc[8643]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9058cb32-a69a-4edc-91fb-455b0ba0f72e",
   "metadata": {},
   "source": [
    "Upon viewing 8643, it is apparant that it indeed needs to be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6f3795d2-e8c7-4e91-9db3-60ba54bc1172",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de = df_de.drop(df_de.index[8643])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "f3c65a3d-b2e2-477e-bb57-f38272761a7b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "reply_settings                     Am ehemaligen Zaun vom Freihafen so gesehen. h...\n",
       "possibly_sensitive                                                641895980024102912\n",
       "id                                                                       317067353.0\n",
       "text                                                          ['641895980024102912']\n",
       "conversation_id                                             2015-09-10T08:48:22.000Z\n",
       "author_id                                                                         de\n",
       "edit_history_tweet_ids             [{'start': 105, 'end': 127, 'url': 'http://t.c...\n",
       "created_at                                                          5bcd72da50f0ee77\n",
       "lang                                                                               5\n",
       "entities.urls                                                                   True\n",
       "geo.place_id                                                2015-09-10T09:18:22.000Z\n",
       "edit_controls.edits_remaining                                                   11.0\n",
       "edit_controls.is_edit_eligible                                                     0\n",
       "edit_controls.editable_until                                                      11\n",
       "public_metrics.retweet_count                                                     0.0\n",
       "public_metrics.reply_count                                                       0.0\n",
       "public_metrics.like_count                                                      Point\n",
       "public_metrics.quote_count                                 [9.98315295, 53.51938516]\n",
       "public_metrics.impression_count                                                  NaN\n",
       "geo.coordinates.type                                                             NaN\n",
       "geo.coordinates.coordinates        [{'start': 0, 'end': 16, 'tag': 'refugeeswelco...\n",
       "in_reply_to_user_id                                         ['3_641895979885662208']\n",
       "referenced_tweets                                                                NaN\n",
       "entities.hashtags                                                                NaN\n",
       "attachments.media_keys                                                           NaN\n",
       "entities.mentions                                                                NaN\n",
       "context_annotations                                                              NaN\n",
       "attachments.poll_ids                                                             NaN\n",
       "entities.cashtags                                                                NaN\n",
       "Name: 13733, dtype: object"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing th second index\n",
    "\n",
    "df_de.iloc[13732]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c2a2598-b002-424c-b90d-c4d3f8317137",
   "metadata": {},
   "source": [
    "Upon viewing 13733 (written as 13732 in the code due to the change in position after dropping 8643), it is apparant that it indeed needs to be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "51e9f767-9e6f-4d63-9071-24eeb66bc37b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de = df_de.drop(df_de.index[13732])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab68f0b-9930-447c-82cd-269ec0b151cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# double-checking that the error is gone\n",
    "\n",
    "df_de.loc[df_de['created_at'] == '5bcd72da50f0ee77', 'created_at']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14584e24-f1f6-40f6-bc29-6ce53af88f95",
   "metadata": {},
   "source": [
    "Now that the errors have been removed, dates can be formatted as usual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "2f965f52-7cf5-4ea2-8600-539da92f4d93",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting created_at to datetime format\n",
    "\n",
    "df_de[\"created_at\"] = pd.to_datetime(df_de[\"created_at\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "1f8439c4-83be-4c7b-8da6-0b9e0047f020",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting it to date and creating a new column called \"date\"\n",
    "\n",
    "df_de['date'] = df_de['created_at'].dt.normalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "a835f2f3-7551-4e67-a21b-75285cb58601",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_77946/4248642937.py:3: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.\n",
      "  df_de['week'] = df_de['created_at'].dt.week\n"
     ]
    }
   ],
   "source": [
    "# creating week, month, and year columns\n",
    "\n",
    "df_de['week'] = df_de['created_at'].dt.week\n",
    "df_de['month'] = df_de['created_at'].dt.month\n",
    "df_de['year'] = df_de['created_at'].dt.year\n",
    "df_de['year-week'] = df_de['created_at'].dt.strftime('%Y-%U')\n",
    "df_de['year-month'] = df_de['created_at'].dt.strftime('%Y-%m')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "555df434-a687-4e39-a633-eb0e2ac264f8",
   "metadata": {},
   "source": [
    "#### 4.2.4 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1ed0ce0d-a485-4a58-8ea0-fbec0086bc97",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>...</th>\n",
       "      <th>attachments.media_keys</th>\n",
       "      <th>entities.mentions</th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>entities.cashtags</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.229216e+17</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>722921572810366977</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>['722921572810366977']</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228995e+17</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>722899547039473665</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>['722899547039473665']</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228974e+17</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>722897370313195521</td>\n",
       "      <td>224607633</td>\n",
       "      <td>['722897370313195521']</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228536e+17</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>722847691101880320</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>['722853635751809025']</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 83, 'end': 106, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>['7_722852972233912321']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228240e+17</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>722824011063799809</td>\n",
       "      <td>606265303</td>\n",
       "      <td>['722824011063799809']</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>['3_722824009939886080']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19458</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.467140e+17</td>\n",
       "      <td>Stehe am Bahnhof und eine Frau erklärt ihren(?...</td>\n",
       "      <td>546714009261858816</td>\n",
       "      <td>19926396</td>\n",
       "      <td>['546714009261858816']</td>\n",
       "      <td>2014-12-21 17:09:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19459</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.467096e+17</td>\n",
       "      <td>Fast 500 Flüchtlinge, Obdachlose und Heimkinde...</td>\n",
       "      <td>546709582018781184</td>\n",
       "      <td>12358112</td>\n",
       "      <td>['546709582018781184']</td>\n",
       "      <td>2014-12-21 16:51:36+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 91, 'end': 114, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>['3_546709581229879297']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19460</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466898e+17</td>\n",
       "      <td>@OomenBerlin @Rex_Cramer Ostsachsen noPerspekt...</td>\n",
       "      <td>546399944803090432</td>\n",
       "      <td>224651746</td>\n",
       "      <td>['546689796337573888']</td>\n",
       "      <td>2014-12-21 15:32:59+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 0, 'end': 12, 'username': 'OomenBer...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19461</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466522e+17</td>\n",
       "      <td>'Nichts gegen Flüchtlinge, aber ein Gefängnis ...</td>\n",
       "      <td>546652160600317952</td>\n",
       "      <td>40870544</td>\n",
       "      <td>['546652160600317952']</td>\n",
       "      <td>2014-12-21 13:03:26+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 75, 'end': 97, 'url': 'http://t.co/...</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19462</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466389e+17</td>\n",
       "      <td>@MartinSoechting denn diese asoziale Bagage ha...</td>\n",
       "      <td>546637023936016384</td>\n",
       "      <td>1426103292</td>\n",
       "      <td>['546638863075405824']</td>\n",
       "      <td>2014-12-21 12:10:35+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 0, 'end': 16, 'username': 'MartinSo...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>51.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>19461 rows × 34 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      reply_settings possibly_sensitive            id  \\\n",
       "0           everyone              False  7.229216e+17   \n",
       "1           everyone              False  7.228995e+17   \n",
       "2           everyone              False  7.228974e+17   \n",
       "3           everyone              False  7.228536e+17   \n",
       "4           everyone              False  7.228240e+17   \n",
       "...              ...                ...           ...   \n",
       "19458       everyone              False  5.467140e+17   \n",
       "19459       everyone              False  5.467096e+17   \n",
       "19460       everyone              False  5.466898e+17   \n",
       "19461       everyone              False  5.466522e+17   \n",
       "19462       everyone              False  5.466389e+17   \n",
       "\n",
       "                                                    text     conversation_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  722921572810366977   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  722899547039473665   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  722897370313195521   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  722847691101880320   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  722824011063799809   \n",
       "...                                                  ...                 ...   \n",
       "19458  Stehe am Bahnhof und eine Frau erklärt ihren(?...  546714009261858816   \n",
       "19459  Fast 500 Flüchtlinge, Obdachlose und Heimkinde...  546709582018781184   \n",
       "19460  @OomenBerlin @Rex_Cramer Ostsachsen noPerspekt...  546399944803090432   \n",
       "19461  'Nichts gegen Flüchtlinge, aber ein Gefängnis ...  546652160600317952   \n",
       "19462  @MartinSoechting denn diese asoziale Bagage ha...  546637023936016384   \n",
       "\n",
       "        author_id  edit_history_tweet_ids                created_at lang  \\\n",
       "0      4122038069  ['722921572810366977'] 2016-04-20 22:55:08+00:00   de   \n",
       "1      1179543852  ['722899547039473665'] 2016-04-20 21:27:37+00:00   de   \n",
       "2       224607633  ['722897370313195521'] 2016-04-20 21:18:58+00:00   de   \n",
       "3      2480764313  ['722853635751809025'] 2016-04-20 18:25:11+00:00   de   \n",
       "4       606265303  ['722824011063799809'] 2016-04-20 16:27:28+00:00   de   \n",
       "...           ...                     ...                       ...  ...   \n",
       "19458    19926396  ['546714009261858816'] 2014-12-21 17:09:11+00:00   de   \n",
       "19459    12358112  ['546709582018781184'] 2014-12-21 16:51:36+00:00   de   \n",
       "19460   224651746  ['546689796337573888'] 2014-12-21 15:32:59+00:00   de   \n",
       "19461    40870544  ['546652160600317952'] 2014-12-21 13:03:26+00:00   de   \n",
       "19462  1426103292  ['546638863075405824'] 2014-12-21 12:10:35+00:00   de   \n",
       "\n",
       "                                           entities.urls  ...  \\\n",
       "0      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   \n",
       "1                                                    NaN  ...   \n",
       "2                                                    NaN  ...   \n",
       "3      [{'start': 83, 'end': 106, 'url': 'https://t.c...  ...   \n",
       "4      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   \n",
       "...                                                  ...  ...   \n",
       "19458                                                NaN  ...   \n",
       "19459  [{'start': 91, 'end': 114, 'url': 'https://t.c...  ...   \n",
       "19460                                                NaN  ...   \n",
       "19461  [{'start': 75, 'end': 97, 'url': 'http://t.co/...  ...   \n",
       "19462                                                NaN  ...   \n",
       "\n",
       "         attachments.media_keys  \\\n",
       "0                           NaN   \n",
       "1                           NaN   \n",
       "2                           NaN   \n",
       "3      ['7_722852972233912321']   \n",
       "4      ['3_722824009939886080']   \n",
       "...                         ...   \n",
       "19458                       NaN   \n",
       "19459  ['3_546709581229879297']   \n",
       "19460                       NaN   \n",
       "19461                       NaN   \n",
       "19462                       NaN   \n",
       "\n",
       "                                       entities.mentions context_annotations  \\\n",
       "0                                                    NaN                 NaN   \n",
       "1                                                    NaN                 NaN   \n",
       "2                                                    NaN                 NaN   \n",
       "3                                                    NaN                 NaN   \n",
       "4                                                    NaN                 NaN   \n",
       "...                                                  ...                 ...   \n",
       "19458                                                NaN                 NaN   \n",
       "19459                                                NaN                 NaN   \n",
       "19460  [{'start': 0, 'end': 12, 'username': 'OomenBer...                 NaN   \n",
       "19461                                                NaN                 NaN   \n",
       "19462  [{'start': 0, 'end': 16, 'username': 'MartinSo...                 NaN   \n",
       "\n",
       "      attachments.poll_ids  entities.cashtags  week month    year  year-week  \\\n",
       "0                      NaN                NaN  16.0   4.0  2016.0    2016-16   \n",
       "1                      NaN                NaN  16.0   4.0  2016.0    2016-16   \n",
       "2                      NaN                NaN  16.0   4.0  2016.0    2016-16   \n",
       "3                      NaN                NaN  16.0   4.0  2016.0    2016-16   \n",
       "4                      NaN                NaN  16.0   4.0  2016.0    2016-16   \n",
       "...                    ...                ...   ...   ...     ...        ...   \n",
       "19458                  NaN                NaN  51.0  12.0  2014.0    2014-51   \n",
       "19459                  NaN                NaN  51.0  12.0  2014.0    2014-51   \n",
       "19460                  NaN                NaN  51.0  12.0  2014.0    2014-51   \n",
       "19461                  NaN                NaN  51.0  12.0  2014.0    2014-51   \n",
       "19462                  NaN                NaN  51.0  12.0  2014.0    2014-51   \n",
       "\n",
       "      year-month  \n",
       "0        2016-04  \n",
       "1        2016-04  \n",
       "2        2016-04  \n",
       "3        2016-04  \n",
       "4        2016-04  \n",
       "...          ...  \n",
       "19458    2014-12  \n",
       "19459    2014-12  \n",
       "19460    2014-12  \n",
       "19461    2014-12  \n",
       "19462    2014-12  \n",
       "\n",
       "[19461 rows x 34 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50e03cd9-304a-46a4-a40e-c8e2ea9581b7",
   "metadata": {},
   "source": [
    "### 4.3 Merging the English-language and German-language dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bcc1e14-58b8-45aa-b72d-d88c359d1cb3",
   "metadata": {},
   "source": [
    "#### 4.3.1 Creating language column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "abbc9141-2d48-4ad2-9b8e-30a698dfe8bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de['Language'] = 'German'\n",
    "df_eng['Language'] = 'English'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "367c1431-4779-4cb1-a77f-d433c0e92926",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a joint data frame\n",
    "\n",
    "df_syr = pd.concat([df_de, df_eng], ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6756401a-3314-413a-a227-bdc4d646cb22",
   "metadata": {},
   "source": [
    "#### 4.3.3 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "cc23c770-c175-4113-8033-1bdb110e295e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>...</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>entities.annotations</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.229216e+17</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>722921572810366977</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>['722921572810366977']</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228995e+17</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>722899547039473665</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>['722899547039473665']</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228974e+17</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>722897370313195521</td>\n",
       "      <td>224607633</td>\n",
       "      <td>['722897370313195521']</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228536e+17</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>722847691101880320</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>['722853635751809025']</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 83, 'end': 106, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228240e+17</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>722824011063799809</td>\n",
       "      <td>606265303</td>\n",
       "      <td>['722824011063799809']</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24626</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.476947e+17</td>\n",
       "      <td>DE-News : Syrian President Bashar Assad, shown...</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['547694737776320513']</td>\n",
       "      <td>2014-12-24 10:06:15+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 117, 'end': 139, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>[{'start': 27, 'end': 38, 'probability': 0.949...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24627</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.476090e+17</td>\n",
       "      <td>#CelioGermanyDesk German President Gauck calls...</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>271776179</td>\n",
       "      <td>['547609017623654400']</td>\n",
       "      <td>2014-12-24 04:25:38+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 96, 'end': 118, 'url': 'http://t.co...</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>[{'start': 1, 'end': 16, 'probability': 0.2996...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24628</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.471029e+17</td>\n",
       "      <td>#munich is colourful! welcome #refugees! no #p...</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>15105772</td>\n",
       "      <td>['547102939484286976']</td>\n",
       "      <td>2014-12-22 18:54:40+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 65, 'end': 87, 'url': 'http://t.co/...</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-22 00:00:00+00:00</td>\n",
       "      <td>[{'start': 1, 'end': 6, 'probability': 0.3992,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24629</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466994e+17</td>\n",
       "      <td>DE-News : There is little to break the monoton...</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['546699388316184577']</td>\n",
       "      <td>2014-12-21 16:11:06+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 101, 'end': 123, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>[{'start': 3, 'end': 6, 'probability': 0.4604,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24630</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.465444e+17</td>\n",
       "      <td>DE-News : View of a vacant lot earmarked for r...</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['546544368169918464']</td>\n",
       "      <td>2014-12-21 05:55:06+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 109, 'end': 131, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>12.0</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>[{'start': 57, 'end': 62, 'probability': 0.848...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>24631 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      reply_settings possibly_sensitive            id  \\\n",
       "0           everyone              False  7.229216e+17   \n",
       "1           everyone              False  7.228995e+17   \n",
       "2           everyone              False  7.228974e+17   \n",
       "3           everyone              False  7.228536e+17   \n",
       "4           everyone              False  7.228240e+17   \n",
       "...              ...                ...           ...   \n",
       "24626       everyone              False  5.476947e+17   \n",
       "24627       everyone              False  5.476090e+17   \n",
       "24628       everyone              False  5.471029e+17   \n",
       "24629       everyone              False  5.466994e+17   \n",
       "24630       everyone              False  5.465444e+17   \n",
       "\n",
       "                                                    text     conversation_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  722921572810366977   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  722899547039473665   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  722897370313195521   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  722847691101880320   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  722824011063799809   \n",
       "...                                                  ...                 ...   \n",
       "24626  DE-News : Syrian President Bashar Assad, shown...  547694737776320513   \n",
       "24627  #CelioGermanyDesk German President Gauck calls...  547609017623654400   \n",
       "24628  #munich is colourful! welcome #refugees! no #p...  547102939484286976   \n",
       "24629  DE-News : There is little to break the monoton...  546699388316184577   \n",
       "24630  DE-News : View of a vacant lot earmarked for r...  546544368169918464   \n",
       "\n",
       "        author_id  edit_history_tweet_ids                created_at lang  \\\n",
       "0      4122038069  ['722921572810366977'] 2016-04-20 22:55:08+00:00   de   \n",
       "1      1179543852  ['722899547039473665'] 2016-04-20 21:27:37+00:00   de   \n",
       "2       224607633  ['722897370313195521'] 2016-04-20 21:18:58+00:00   de   \n",
       "3      2480764313  ['722853635751809025'] 2016-04-20 18:25:11+00:00   de   \n",
       "4       606265303  ['722824011063799809'] 2016-04-20 16:27:28+00:00   de   \n",
       "...           ...                     ...                       ...  ...   \n",
       "24626   186899860  ['547694737776320513'] 2014-12-24 10:06:15+00:00   en   \n",
       "24627   271776179  ['547609017623654400'] 2014-12-24 04:25:38+00:00   en   \n",
       "24628    15105772  ['547102939484286976'] 2014-12-22 18:54:40+00:00   en   \n",
       "24629   186899860  ['546699388316184577'] 2014-12-21 16:11:06+00:00   en   \n",
       "24630   186899860  ['546544368169918464'] 2014-12-21 05:55:06+00:00   en   \n",
       "\n",
       "                                           entities.urls  ... month    year  \\\n",
       "0      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   4.0  2016.0   \n",
       "1                                                    NaN  ...   4.0  2016.0   \n",
       "2                                                    NaN  ...   4.0  2016.0   \n",
       "3      [{'start': 83, 'end': 106, 'url': 'https://t.c...  ...   4.0  2016.0   \n",
       "4      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...   4.0  2016.0   \n",
       "...                                                  ...  ...   ...     ...   \n",
       "24626  [{'start': 117, 'end': 139, 'url': 'http://t.c...  ...  12.0  2014.0   \n",
       "24627  [{'start': 96, 'end': 118, 'url': 'http://t.co...  ...  12.0  2014.0   \n",
       "24628  [{'start': 65, 'end': 87, 'url': 'http://t.co/...  ...  12.0  2014.0   \n",
       "24629  [{'start': 101, 'end': 123, 'url': 'http://t.c...  ...  12.0  2014.0   \n",
       "24630  [{'start': 109, 'end': 131, 'url': 'http://t.c...  ...  12.0  2014.0   \n",
       "\n",
       "      year-week year-month  Language                      date  \\\n",
       "0       2016-16    2016-04    German 2016-04-20 00:00:00+00:00   \n",
       "1       2016-16    2016-04    German 2016-04-20 00:00:00+00:00   \n",
       "2       2016-16    2016-04    German 2016-04-20 00:00:00+00:00   \n",
       "3       2016-16    2016-04    German 2016-04-20 00:00:00+00:00   \n",
       "4       2016-16    2016-04    German 2016-04-20 00:00:00+00:00   \n",
       "...         ...        ...       ...                       ...   \n",
       "24626   2014-51    2014-12   English 2014-12-24 00:00:00+00:00   \n",
       "24627   2014-51    2014-12   English 2014-12-24 00:00:00+00:00   \n",
       "24628   2014-51    2014-12   English 2014-12-22 00:00:00+00:00   \n",
       "24629   2014-51    2014-12   English 2014-12-21 00:00:00+00:00   \n",
       "24630   2014-51    2014-12   English 2014-12-21 00:00:00+00:00   \n",
       "\n",
       "                                    entities.annotations withheld.copyright  \\\n",
       "0                                                    NaN                NaN   \n",
       "1                                                    NaN                NaN   \n",
       "2                                                    NaN                NaN   \n",
       "3                                                    NaN                NaN   \n",
       "4                                                    NaN                NaN   \n",
       "...                                                  ...                ...   \n",
       "24626  [{'start': 27, 'end': 38, 'probability': 0.949...                NaN   \n",
       "24627  [{'start': 1, 'end': 16, 'probability': 0.2996...                NaN   \n",
       "24628  [{'start': 1, 'end': 6, 'probability': 0.3992,...                NaN   \n",
       "24629  [{'start': 3, 'end': 6, 'probability': 0.4604,...                NaN   \n",
       "24630  [{'start': 57, 'end': 62, 'probability': 0.848...                NaN   \n",
       "\n",
       "       withheld.country_codes withheld.scope  \n",
       "0                         NaN            NaN  \n",
       "1                         NaN            NaN  \n",
       "2                         NaN            NaN  \n",
       "3                         NaN            NaN  \n",
       "4                         NaN            NaN  \n",
       "...                       ...            ...  \n",
       "24626                     NaN            NaN  \n",
       "24627                     NaN            NaN  \n",
       "24628                     NaN            NaN  \n",
       "24629                     NaN            NaN  \n",
       "24630                     NaN            NaN  \n",
       "\n",
       "[24631 rows x 40 columns]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_syr"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02be679d-f8c6-40c9-883f-87d47689cb61",
   "metadata": {},
   "source": [
    "## 5. Ukrainian inflow datasets\n",
    "### 5.1. English-language dataset\n",
    "#### 5.1.1 Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "5a7829ed-ffe0-4c48-9695-491c831a5fe6",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_eng = pd.read_csv(CASS_thesis / \"01a_Data-Collection_limited_Ukrainian-eng.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0107e985-3dc7-4d91-b800-5e04e0167eca",
   "metadata": {},
   "source": [
    "#### 5.1.2 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6ca37127-3c06-4cb9-ba95-237e133e7978",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>id</th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>...</th>\n",
       "      <th>entities.hashtags</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>attachments.media_keys</th>\n",
       "      <th>referenced_tweets</th>\n",
       "      <th>entities.cashtags</th>\n",
       "      <th>geo.coordinates.type</th>\n",
       "      <th>geo.coordinates.coordinates</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23T21:26:35.000Z</td>\n",
       "      <td>@elonmusk Elon Musk: it’s.       🆎out time you...</td>\n",
       "      <td>['1584295241993310209']</td>\n",
       "      <td>1584295241993310209</td>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>en</td>\n",
       "      <td>229784080</td>\n",
       "      <td>False</td>\n",
       "      <td>4.419640e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23T16:48:33.000Z</td>\n",
       "      <td>That was a nice visit @DOK_Leipzig the last co...</td>\n",
       "      <td>['1584225275914915842']</td>\n",
       "      <td>1584225275914915842</td>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>en</td>\n",
       "      <td>930852774</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 230, 'end': 245, 'tag': 'festivalse...</td>\n",
       "      <td>[{'start': 288, 'end': 311, 'url': 'https://t....</td>\n",
       "      <td>['3_1584225201956454401', '3_15842252332167086...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23T16:36:15.000Z</td>\n",
       "      <td>The Evil of the Ukrainian Forces has No bounds...</td>\n",
       "      <td>['1584222176730750976']</td>\n",
       "      <td>1584222176730750976</td>\n",
       "      <td>[{'domain': {'id': '123', 'name': 'Ongoing New...</td>\n",
       "      <td>en</td>\n",
       "      <td>1454177533931540482</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 77, 'end': 100, 'url': 'https://t.c...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23T16:26:46.000Z</td>\n",
       "      <td>@thesiriusreport The Ukrainians now boast of b...</td>\n",
       "      <td>['1584219793670176769']</td>\n",
       "      <td>1584219793670176769</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>False</td>\n",
       "      <td>7.017749e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '158414894324339...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23T14:36:11.000Z</td>\n",
       "      <td>@tom_username_ DPR/LNR militia did a huge part...</td>\n",
       "      <td>['1584191961053159425']</td>\n",
       "      <td>1584191961053159425</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>1121807798826930177</td>\n",
       "      <td>False</td>\n",
       "      <td>8.725516e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '158419130384313...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8006</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-28T14:43:31.000Z</td>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>['1409522855814041606']</td>\n",
       "      <td>1409522855814041606</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>110402493</td>\n",
       "      <td>False</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 95, 'end': 108, 'tag': 'SocialRight...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '140843450365639...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8007</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-27T13:03:29.000Z</td>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>['1409135295895855104']</td>\n",
       "      <td>1409135295895855104</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>False</td>\n",
       "      <td>1.586954e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '140913435503797...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8008</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-27T08:37:21.000Z</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>['1409068320947593220']</td>\n",
       "      <td>1409068320947593220</td>\n",
       "      <td>[{'domain': {'id': '130', 'name': 'Multimedia ...</td>\n",
       "      <td>en</td>\n",
       "      <td>563381751</td>\n",
       "      <td>False</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 272, 'end': 295, 'url': 'https://t....</td>\n",
       "      <td>['3_1409068306196205568', '3_14090683170636595...</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '140906829783282...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8009</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-26T08:03:22.000Z</td>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>['1408697379063271425']</td>\n",
       "      <td>1408697379063271425</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>False</td>\n",
       "      <td>3.591885e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'type': 'replied_to', 'id': '140845747170700...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8010</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-25T21:02:17.000Z</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>['1408531015518326784']</td>\n",
       "      <td>1408531015518326784</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>134401670</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]</td>\n",
       "      <td>[{'start': 239, 'end': 262, 'url': 'https://t....</td>\n",
       "      <td>['7_1408530965446733835']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8011 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     reply_settings                created_at  \\\n",
       "0          everyone  2022-10-23T21:26:35.000Z   \n",
       "1          everyone  2022-10-23T16:48:33.000Z   \n",
       "2          everyone  2022-10-23T16:36:15.000Z   \n",
       "3          everyone  2022-10-23T16:26:46.000Z   \n",
       "4          everyone  2022-10-23T14:36:11.000Z   \n",
       "...             ...                       ...   \n",
       "8006       everyone  2021-06-28T14:43:31.000Z   \n",
       "8007       everyone  2021-06-27T13:03:29.000Z   \n",
       "8008       everyone  2021-06-27T08:37:21.000Z   \n",
       "8009       everyone  2021-06-26T08:03:22.000Z   \n",
       "8010       everyone  2021-06-25T21:02:17.000Z   \n",
       "\n",
       "                                                   text  \\\n",
       "0     @elonmusk Elon Musk: it’s.       🆎out time you...   \n",
       "1     That was a nice visit @DOK_Leipzig the last co...   \n",
       "2     The Evil of the Ukrainian Forces has No bounds...   \n",
       "3     @thesiriusreport The Ukrainians now boast of b...   \n",
       "4     @tom_username_ DPR/LNR militia did a huge part...   \n",
       "...                                                 ...   \n",
       "8006  For day 1 of week 2, @AnnaMariaKonsta discusse...   \n",
       "8007  @ariadneconill Europe is racist, but in a diff...   \n",
       "8008  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "8009  @simongerman600 I must have missed the great f...   \n",
       "8010  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "       edit_history_tweet_ids                   id  \\\n",
       "0     ['1584295241993310209']  1584295241993310209   \n",
       "1     ['1584225275914915842']  1584225275914915842   \n",
       "2     ['1584222176730750976']  1584222176730750976   \n",
       "3     ['1584219793670176769']  1584219793670176769   \n",
       "4     ['1584191961053159425']  1584191961053159425   \n",
       "...                       ...                  ...   \n",
       "8006  ['1409522855814041606']  1409522855814041606   \n",
       "8007  ['1409135295895855104']  1409135295895855104   \n",
       "8008  ['1409068320947593220']  1409068320947593220   \n",
       "8009  ['1408697379063271425']  1408697379063271425   \n",
       "8010  ['1408531015518326784']  1408531015518326784   \n",
       "\n",
       "                                    context_annotations lang  \\\n",
       "0     [{'domain': {'id': '46', 'name': 'Business Tax...   en   \n",
       "1     [{'domain': {'id': '46', 'name': 'Business Tax...   en   \n",
       "2     [{'domain': {'id': '123', 'name': 'Ongoing New...   en   \n",
       "3                                                   NaN   en   \n",
       "4                                                   NaN   en   \n",
       "...                                                 ...  ...   \n",
       "8006                                                NaN   en   \n",
       "8007                                                NaN   en   \n",
       "8008  [{'domain': {'id': '130', 'name': 'Multimedia ...   en   \n",
       "8009                                                NaN   en   \n",
       "8010                                                NaN   en   \n",
       "\n",
       "                author_id  possibly_sensitive  in_reply_to_user_id  ...  \\\n",
       "0               229784080               False         4.419640e+07  ...   \n",
       "1               930852774               False                  NaN  ...   \n",
       "2     1454177533931540482               False                  NaN  ...   \n",
       "3              1272454735               False         7.017749e+17  ...   \n",
       "4     1121807798826930177               False         8.725516e+17  ...   \n",
       "...                   ...                 ...                  ...  ...   \n",
       "8006            110402493               False         1.104025e+08  ...   \n",
       "8007           2521808908               False         1.586954e+07  ...   \n",
       "8008            563381751               False         5.633818e+08  ...   \n",
       "8009           2591892350               False         3.591885e+08  ...   \n",
       "8010            134401670                True                  NaN  ...   \n",
       "\n",
       "                                      entities.hashtags  \\\n",
       "0                                                   NaN   \n",
       "1     [{'start': 230, 'end': 245, 'tag': 'festivalse...   \n",
       "2                                                   NaN   \n",
       "3                                                   NaN   \n",
       "4                                                   NaN   \n",
       "...                                                 ...   \n",
       "8006  [{'start': 95, 'end': 108, 'tag': 'SocialRight...   \n",
       "8007                                                NaN   \n",
       "8008                                                NaN   \n",
       "8009                                                NaN   \n",
       "8010      [{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]   \n",
       "\n",
       "                                          entities.urls  \\\n",
       "0                                                   NaN   \n",
       "1     [{'start': 288, 'end': 311, 'url': 'https://t....   \n",
       "2     [{'start': 77, 'end': 100, 'url': 'https://t.c...   \n",
       "3                                                   NaN   \n",
       "4                                                   NaN   \n",
       "...                                                 ...   \n",
       "8006                                                NaN   \n",
       "8007                                                NaN   \n",
       "8008  [{'start': 272, 'end': 295, 'url': 'https://t....   \n",
       "8009                                                NaN   \n",
       "8010  [{'start': 239, 'end': 262, 'url': 'https://t....   \n",
       "\n",
       "                                 attachments.media_keys  \\\n",
       "0                                                   NaN   \n",
       "1     ['3_1584225201956454401', '3_15842252332167086...   \n",
       "2                                                   NaN   \n",
       "3                                                   NaN   \n",
       "4                                                   NaN   \n",
       "...                                                 ...   \n",
       "8006                                                NaN   \n",
       "8007                                                NaN   \n",
       "8008  ['3_1409068306196205568', '3_14090683170636595...   \n",
       "8009                                                NaN   \n",
       "8010                          ['7_1408530965446733835']   \n",
       "\n",
       "                                      referenced_tweets  entities.cashtags  \\\n",
       "0                                                   NaN                NaN   \n",
       "1                                                   NaN                NaN   \n",
       "2                                                   NaN                NaN   \n",
       "3     [{'type': 'replied_to', 'id': '158414894324339...                NaN   \n",
       "4     [{'type': 'replied_to', 'id': '158419130384313...                NaN   \n",
       "...                                                 ...                ...   \n",
       "8006  [{'type': 'replied_to', 'id': '140843450365639...                NaN   \n",
       "8007  [{'type': 'replied_to', 'id': '140913435503797...                NaN   \n",
       "8008  [{'type': 'replied_to', 'id': '140906829783282...                NaN   \n",
       "8009  [{'type': 'replied_to', 'id': '140845747170700...                NaN   \n",
       "8010                                                NaN                NaN   \n",
       "\n",
       "     geo.coordinates.type geo.coordinates.coordinates  withheld.copyright  \\\n",
       "0                     NaN                         NaN                 NaN   \n",
       "1                     NaN                         NaN                 NaN   \n",
       "2                     NaN                         NaN                 NaN   \n",
       "3                     NaN                         NaN                 NaN   \n",
       "4                     NaN                         NaN                 NaN   \n",
       "...                   ...                         ...                 ...   \n",
       "8006                  NaN                         NaN                 NaN   \n",
       "8007                  NaN                         NaN                 NaN   \n",
       "8008                  NaN                         NaN                 NaN   \n",
       "8009                  NaN                         NaN                 NaN   \n",
       "8010                  NaN                         NaN                 NaN   \n",
       "\n",
       "      withheld.country_codes  attachments.poll_ids  \n",
       "0                        NaN                   NaN  \n",
       "1                        NaN                   NaN  \n",
       "2                        NaN                   NaN  \n",
       "3                        NaN                   NaN  \n",
       "4                        NaN                   NaN  \n",
       "...                      ...                   ...  \n",
       "8006                     NaN                   NaN  \n",
       "8007                     NaN                   NaN  \n",
       "8008                     NaN                   NaN  \n",
       "8009                     NaN                   NaN  \n",
       "8010                     NaN                   NaN  \n",
       "\n",
       "[8011 rows x 32 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_eng"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99c6dc9f-c508-498b-90b5-716adf7e13da",
   "metadata": {},
   "source": [
    "#### 5.1.3 Formatting dates "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "1750f31a-b064-4075-889e-0fe07845537b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting created_at to datetime format\n",
    "\n",
    "df_eng[\"created_at\"] = pd.to_datetime(df_eng[\"created_at\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "5c71a6a0-3bf8-4d7c-8549-466887d4f91c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting it to date and creating a new column called \"date\"\n",
    "\n",
    "df_eng['date'] = df_eng['created_at'].dt.normalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c101eb3e-8147-453b-b5ed-82db4cf939eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_77946/3519539084.py:3: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.\n",
      "  df_eng['week'] = df_eng['created_at'].dt.week\n"
     ]
    }
   ],
   "source": [
    "# creating week, month, year, year-week, and year-month columns\n",
    "\n",
    "df_eng['week'] = df_eng['created_at'].dt.week\n",
    "df_eng['month'] = df_eng['created_at'].dt.month\n",
    "df_eng['year'] = df_eng['created_at'].dt.year\n",
    "df_eng['year-week'] = df_eng['created_at'].dt.strftime('%Y-%U')\n",
    "df_eng['year-month'] = df_eng['created_at'].dt.strftime('%Y-%m')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c121ff95-385f-42e0-b080-e80623558935",
   "metadata": {},
   "source": [
    "#### 5.1.4 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "55999080-f752-4b24-bf9e-330d8bcd42e9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>created_at</th>\n",
       "      <th>text</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>id</th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>...</th>\n",
       "      <th>geo.coordinates.coordinates</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>date</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23 21:26:35+00:00</td>\n",
       "      <td>@elonmusk Elon Musk: it’s.       🆎out time you...</td>\n",
       "      <td>['1584295241993310209']</td>\n",
       "      <td>1584295241993310209</td>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>en</td>\n",
       "      <td>229784080</td>\n",
       "      <td>False</td>\n",
       "      <td>4.419640e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42</td>\n",
       "      <td>10</td>\n",
       "      <td>2022</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23 16:48:33+00:00</td>\n",
       "      <td>That was a nice visit @DOK_Leipzig the last co...</td>\n",
       "      <td>['1584225275914915842']</td>\n",
       "      <td>1584225275914915842</td>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>en</td>\n",
       "      <td>930852774</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42</td>\n",
       "      <td>10</td>\n",
       "      <td>2022</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23 16:36:15+00:00</td>\n",
       "      <td>The Evil of the Ukrainian Forces has No bounds...</td>\n",
       "      <td>['1584222176730750976']</td>\n",
       "      <td>1584222176730750976</td>\n",
       "      <td>[{'domain': {'id': '123', 'name': 'Ongoing New...</td>\n",
       "      <td>en</td>\n",
       "      <td>1454177533931540482</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42</td>\n",
       "      <td>10</td>\n",
       "      <td>2022</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23 16:26:46+00:00</td>\n",
       "      <td>@thesiriusreport The Ukrainians now boast of b...</td>\n",
       "      <td>['1584219793670176769']</td>\n",
       "      <td>1584219793670176769</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>False</td>\n",
       "      <td>7.017749e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42</td>\n",
       "      <td>10</td>\n",
       "      <td>2022</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2022-10-23 14:36:11+00:00</td>\n",
       "      <td>@tom_username_ DPR/LNR militia did a huge part...</td>\n",
       "      <td>['1584191961053159425']</td>\n",
       "      <td>1584191961053159425</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>1121807798826930177</td>\n",
       "      <td>False</td>\n",
       "      <td>8.725516e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42</td>\n",
       "      <td>10</td>\n",
       "      <td>2022</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8006</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>['1409522855814041606']</td>\n",
       "      <td>1409522855814041606</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>110402493</td>\n",
       "      <td>False</td>\n",
       "      <td>1.104025e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>26</td>\n",
       "      <td>6</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8007</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>['1409135295895855104']</td>\n",
       "      <td>1409135295895855104</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>False</td>\n",
       "      <td>1.586954e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8008</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>['1409068320947593220']</td>\n",
       "      <td>1409068320947593220</td>\n",
       "      <td>[{'domain': {'id': '130', 'name': 'Multimedia ...</td>\n",
       "      <td>en</td>\n",
       "      <td>563381751</td>\n",
       "      <td>False</td>\n",
       "      <td>5.633818e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8009</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>['1408697379063271425']</td>\n",
       "      <td>1408697379063271425</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>False</td>\n",
       "      <td>3.591885e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8010</th>\n",
       "      <td>everyone</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>['1408531015518326784']</td>\n",
       "      <td>1408531015518326784</td>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>134401670</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25</td>\n",
       "      <td>6</td>\n",
       "      <td>2021</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8011 rows × 38 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     reply_settings                created_at  \\\n",
       "0          everyone 2022-10-23 21:26:35+00:00   \n",
       "1          everyone 2022-10-23 16:48:33+00:00   \n",
       "2          everyone 2022-10-23 16:36:15+00:00   \n",
       "3          everyone 2022-10-23 16:26:46+00:00   \n",
       "4          everyone 2022-10-23 14:36:11+00:00   \n",
       "...             ...                       ...   \n",
       "8006       everyone 2021-06-28 14:43:31+00:00   \n",
       "8007       everyone 2021-06-27 13:03:29+00:00   \n",
       "8008       everyone 2021-06-27 08:37:21+00:00   \n",
       "8009       everyone 2021-06-26 08:03:22+00:00   \n",
       "8010       everyone 2021-06-25 21:02:17+00:00   \n",
       "\n",
       "                                                   text  \\\n",
       "0     @elonmusk Elon Musk: it’s.       🆎out time you...   \n",
       "1     That was a nice visit @DOK_Leipzig the last co...   \n",
       "2     The Evil of the Ukrainian Forces has No bounds...   \n",
       "3     @thesiriusreport The Ukrainians now boast of b...   \n",
       "4     @tom_username_ DPR/LNR militia did a huge part...   \n",
       "...                                                 ...   \n",
       "8006  For day 1 of week 2, @AnnaMariaKonsta discusse...   \n",
       "8007  @ariadneconill Europe is racist, but in a diff...   \n",
       "8008  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "8009  @simongerman600 I must have missed the great f...   \n",
       "8010  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "       edit_history_tweet_ids                   id  \\\n",
       "0     ['1584295241993310209']  1584295241993310209   \n",
       "1     ['1584225275914915842']  1584225275914915842   \n",
       "2     ['1584222176730750976']  1584222176730750976   \n",
       "3     ['1584219793670176769']  1584219793670176769   \n",
       "4     ['1584191961053159425']  1584191961053159425   \n",
       "...                       ...                  ...   \n",
       "8006  ['1409522855814041606']  1409522855814041606   \n",
       "8007  ['1409135295895855104']  1409135295895855104   \n",
       "8008  ['1409068320947593220']  1409068320947593220   \n",
       "8009  ['1408697379063271425']  1408697379063271425   \n",
       "8010  ['1408531015518326784']  1408531015518326784   \n",
       "\n",
       "                                    context_annotations lang  \\\n",
       "0     [{'domain': {'id': '46', 'name': 'Business Tax...   en   \n",
       "1     [{'domain': {'id': '46', 'name': 'Business Tax...   en   \n",
       "2     [{'domain': {'id': '123', 'name': 'Ongoing New...   en   \n",
       "3                                                   NaN   en   \n",
       "4                                                   NaN   en   \n",
       "...                                                 ...  ...   \n",
       "8006                                                NaN   en   \n",
       "8007                                                NaN   en   \n",
       "8008  [{'domain': {'id': '130', 'name': 'Multimedia ...   en   \n",
       "8009                                                NaN   en   \n",
       "8010                                                NaN   en   \n",
       "\n",
       "                author_id  possibly_sensitive  in_reply_to_user_id  ...  \\\n",
       "0               229784080               False         4.419640e+07  ...   \n",
       "1               930852774               False                  NaN  ...   \n",
       "2     1454177533931540482               False                  NaN  ...   \n",
       "3              1272454735               False         7.017749e+17  ...   \n",
       "4     1121807798826930177               False         8.725516e+17  ...   \n",
       "...                   ...                 ...                  ...  ...   \n",
       "8006            110402493               False         1.104025e+08  ...   \n",
       "8007           2521808908               False         1.586954e+07  ...   \n",
       "8008            563381751               False         5.633818e+08  ...   \n",
       "8009           2591892350               False         3.591885e+08  ...   \n",
       "8010            134401670                True                  NaN  ...   \n",
       "\n",
       "      geo.coordinates.coordinates withheld.copyright withheld.country_codes  \\\n",
       "0                             NaN                NaN                    NaN   \n",
       "1                             NaN                NaN                    NaN   \n",
       "2                             NaN                NaN                    NaN   \n",
       "3                             NaN                NaN                    NaN   \n",
       "4                             NaN                NaN                    NaN   \n",
       "...                           ...                ...                    ...   \n",
       "8006                          NaN                NaN                    NaN   \n",
       "8007                          NaN                NaN                    NaN   \n",
       "8008                          NaN                NaN                    NaN   \n",
       "8009                          NaN                NaN                    NaN   \n",
       "8010                          NaN                NaN                    NaN   \n",
       "\n",
       "      attachments.poll_ids                      date week month  year  \\\n",
       "0                      NaN 2022-10-23 00:00:00+00:00   42    10  2022   \n",
       "1                      NaN 2022-10-23 00:00:00+00:00   42    10  2022   \n",
       "2                      NaN 2022-10-23 00:00:00+00:00   42    10  2022   \n",
       "3                      NaN 2022-10-23 00:00:00+00:00   42    10  2022   \n",
       "4                      NaN 2022-10-23 00:00:00+00:00   42    10  2022   \n",
       "...                    ...                       ...  ...   ...   ...   \n",
       "8006                   NaN 2021-06-28 00:00:00+00:00   26     6  2021   \n",
       "8007                   NaN 2021-06-27 00:00:00+00:00   25     6  2021   \n",
       "8008                   NaN 2021-06-27 00:00:00+00:00   25     6  2021   \n",
       "8009                   NaN 2021-06-26 00:00:00+00:00   25     6  2021   \n",
       "8010                   NaN 2021-06-25 00:00:00+00:00   25     6  2021   \n",
       "\n",
       "      year-week  year-month  \n",
       "0       2022-43     2022-10  \n",
       "1       2022-43     2022-10  \n",
       "2       2022-43     2022-10  \n",
       "3       2022-43     2022-10  \n",
       "4       2022-43     2022-10  \n",
       "...         ...         ...  \n",
       "8006    2021-26     2021-06  \n",
       "8007    2021-26     2021-06  \n",
       "8008    2021-26     2021-06  \n",
       "8009    2021-25     2021-06  \n",
       "8010    2021-25     2021-06  \n",
       "\n",
       "[8011 rows x 38 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_eng"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9575f6a3-7663-4905-ba84-eeeae240ae07",
   "metadata": {},
   "source": [
    "### 5.2. German-language dataset\n",
    "#### 5.2.1 Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "2bdfca3b-3c00-4e3d-a936-ed3f17d115f3",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_77946/3020554363.py:1: DtypeWarning: Columns (2,9,10,18,19,27,28,29,30) have mixed types. Specify dtype option on import or set low_memory=False.\n",
      "  df_de = pd.read_csv(CASS_thesis / \"01b_Data-Collection_limited_Ukrainian-de.csv\")\n"
     ]
    }
   ],
   "source": [
    "df_de = pd.read_csv(CASS_thesis / \"01b_Data-Collection_limited_Ukrainian-de.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bff3f48-de9d-4e8d-a378-e2d466dc10a7",
   "metadata": {},
   "source": [
    "#### 5.2.2 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "aae22e02-7044-4059-be56-42faf9774675",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>...</th>\n",
       "      <th>entities.mentions</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>entities.hashtags</th>\n",
       "      <th>attachments.media_keys</th>\n",
       "      <th>geo.coordinates.type</th>\n",
       "      <th>geo.coordinates.coordinates</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>de</td>\n",
       "      <td>1508097355458961410</td>\n",
       "      <td>Die Ukraine plant eine „False Flag“ Operation ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313222764511232']</td>\n",
       "      <td>1.584313e+18</td>\n",
       "      <td>2022-10-23T22:38:02.000Z</td>\n",
       "      <td>1584313222764511232.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1498603032640167936</td>\n",
       "      <td>@MalteKaufmann Oh wow..Ja, schlimme Zustände w...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313179298885633']</td>\n",
       "      <td>1.584091e+18</td>\n",
       "      <td>2022-10-23T22:37:51.000Z</td>\n",
       "      <td>1584313179298885632.0</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 0, 'end': 14, 'username': 'MalteKau...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1310099812474331147</td>\n",
       "      <td>@HasnainKazim Ach, es ist ja so einfach mit de...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584258196608192512']</td>\n",
       "      <td>1.583888e+18</td>\n",
       "      <td>2022-10-23T18:59:22.000Z</td>\n",
       "      <td>1584258196608192512.0</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 0, 'end': 13, 'username': 'HasnainK...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>2218012226</td>\n",
       "      <td>Hätte er auch gewonnen, wenn er kein Ukrainer ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584253429714944000']</td>\n",
       "      <td>1.584253e+18</td>\n",
       "      <td>2022-10-23T18:40:26.000Z</td>\n",
       "      <td>1584253429714944000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'domain': {'id': '10', 'name': 'Person', 'de...</td>\n",
       "      <td>de</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>Unsere Politiker sind allmählich komplett verr...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584212627927941120']</td>\n",
       "      <td>1.584208e+18</td>\n",
       "      <td>2022-10-23T15:58:18.000Z</td>\n",
       "      <td>1584212627927941120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33619</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1081214456099684352</td>\n",
       "      <td>@polenz_r Sie ist nicht nur wunderbar, sondern...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408456160915689479']</td>\n",
       "      <td>1.337706e+18</td>\n",
       "      <td>2021-06-25T16:04:51.000Z</td>\n",
       "      <td>1408456160915689479</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 0, 'end': 9, 'username': 'polenz_r'...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33620</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1322552611241971714</td>\n",
       "      <td>Und morgen sind die Verantwortlichen als Flüch...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408386043800399873']</td>\n",
       "      <td>1.408386e+18</td>\n",
       "      <td>2021-06-25T11:26:14.000Z</td>\n",
       "      <td>1408386043800399873</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33621</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>3021093443</td>\n",
       "      <td>4.700.000.000,- Euro\\nfür Syrer. \\nEU unterstü...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408289579115978758']</td>\n",
       "      <td>1.408290e+18</td>\n",
       "      <td>2021-06-25T05:02:55.000Z</td>\n",
       "      <td>1408289579115978758</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33622</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>10456882</td>\n",
       "      <td>Menschen hetzen gegen #LGBTQI und Flüchtlinge ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408140630149173255']</td>\n",
       "      <td>1.408141e+18</td>\n",
       "      <td>2021-06-24T19:11:02.000Z</td>\n",
       "      <td>1408140630149173255</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 22, 'end': 29, 'tag': 'LGBTQI'}]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33623</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>3197513446</td>\n",
       "      <td>Der Fall der Familie Imerlishvilli aus Pirna b...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408093903631360002']</td>\n",
       "      <td>1.408094e+18</td>\n",
       "      <td>2021-06-24T16:05:22.000Z</td>\n",
       "      <td>1408093903631360002</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'start': 128, 'end': 144, 'username': 'AG_As...</td>\n",
       "      <td>[{'start': 237, 'end': 260, 'url': 'https://t....</td>\n",
       "      <td>[{'start': 211, 'end': 236, 'tag': 'bring_back...</td>\n",
       "      <td>['3_1408093900959592454']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33624 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     context_annotations lang  \\\n",
       "0      [{'domain': {'id': '46', 'name': 'Business Tax...   de   \n",
       "1                                                    NaN   de   \n",
       "2                                                    NaN   de   \n",
       "3                                                    NaN   de   \n",
       "4      [{'domain': {'id': '10', 'name': 'Person', 'de...   de   \n",
       "...                                                  ...  ...   \n",
       "33619                                                NaN   de   \n",
       "33620                                                NaN   de   \n",
       "33621                                                NaN   de   \n",
       "33622                                                NaN   de   \n",
       "33623                                                NaN   de   \n",
       "\n",
       "                 author_id                                               text  \\\n",
       "0      1508097355458961410  Die Ukraine plant eine „False Flag“ Operation ...   \n",
       "1      1498603032640167936  @MalteKaufmann Oh wow..Ja, schlimme Zustände w...   \n",
       "2      1310099812474331147  @HasnainKazim Ach, es ist ja so einfach mit de...   \n",
       "3               2218012226  Hätte er auch gewonnen, wenn er kein Ukrainer ...   \n",
       "4               1272454735  Unsere Politiker sind allmählich komplett verr...   \n",
       "...                    ...                                                ...   \n",
       "33619  1081214456099684352  @polenz_r Sie ist nicht nur wunderbar, sondern...   \n",
       "33620  1322552611241971714  Und morgen sind die Verantwortlichen als Flüch...   \n",
       "33621           3021093443  4.700.000.000,- Euro\\nfür Syrer. \\nEU unterstü...   \n",
       "33622             10456882  Menschen hetzen gegen #LGBTQI und Flüchtlinge ...   \n",
       "33623           3197513446  Der Fall der Familie Imerlishvilli aus Pirna b...   \n",
       "\n",
       "      reply_settings possibly_sensitive   edit_history_tweet_ids  \\\n",
       "0           everyone              False  ['1584313222764511232']   \n",
       "1           everyone              False  ['1584313179298885633']   \n",
       "2           everyone              False  ['1584258196608192512']   \n",
       "3           everyone              False  ['1584253429714944000']   \n",
       "4           everyone              False  ['1584212627927941120']   \n",
       "...              ...                ...                      ...   \n",
       "33619       everyone              False  ['1408456160915689479']   \n",
       "33620       everyone              False  ['1408386043800399873']   \n",
       "33621       everyone              False  ['1408289579115978758']   \n",
       "33622       everyone              False  ['1408140630149173255']   \n",
       "33623       everyone              False  ['1408093903631360002']   \n",
       "\n",
       "       conversation_id                created_at                     id  ...  \\\n",
       "0         1.584313e+18  2022-10-23T22:38:02.000Z  1584313222764511232.0  ...   \n",
       "1         1.584091e+18  2022-10-23T22:37:51.000Z  1584313179298885632.0  ...   \n",
       "2         1.583888e+18  2022-10-23T18:59:22.000Z  1584258196608192512.0  ...   \n",
       "3         1.584253e+18  2022-10-23T18:40:26.000Z  1584253429714944000.0  ...   \n",
       "4         1.584208e+18  2022-10-23T15:58:18.000Z  1584212627927941120.0  ...   \n",
       "...                ...                       ...                    ...  ...   \n",
       "33619     1.337706e+18  2021-06-25T16:04:51.000Z    1408456160915689479  ...   \n",
       "33620     1.408386e+18  2021-06-25T11:26:14.000Z    1408386043800399873  ...   \n",
       "33621     1.408290e+18  2021-06-25T05:02:55.000Z    1408289579115978758  ...   \n",
       "33622     1.408141e+18  2021-06-24T19:11:02.000Z    1408140630149173255  ...   \n",
       "33623     1.408094e+18  2021-06-24T16:05:22.000Z    1408093903631360002  ...   \n",
       "\n",
       "                                       entities.mentions  \\\n",
       "0                                                    NaN   \n",
       "1      [{'start': 0, 'end': 14, 'username': 'MalteKau...   \n",
       "2      [{'start': 0, 'end': 13, 'username': 'HasnainK...   \n",
       "3                                                    NaN   \n",
       "4                                                    NaN   \n",
       "...                                                  ...   \n",
       "33619  [{'start': 0, 'end': 9, 'username': 'polenz_r'...   \n",
       "33620                                                NaN   \n",
       "33621                                                NaN   \n",
       "33622                                                NaN   \n",
       "33623  [{'start': 128, 'end': 144, 'username': 'AG_As...   \n",
       "\n",
       "                                           entities.urls  \\\n",
       "0                                                    NaN   \n",
       "1                                                    NaN   \n",
       "2                                                    NaN   \n",
       "3                                                    NaN   \n",
       "4                                                    NaN   \n",
       "...                                                  ...   \n",
       "33619                                                NaN   \n",
       "33620                                                NaN   \n",
       "33621                                                NaN   \n",
       "33622                                                NaN   \n",
       "33623  [{'start': 237, 'end': 260, 'url': 'https://t....   \n",
       "\n",
       "                                       entities.hashtags  \\\n",
       "0                                                    NaN   \n",
       "1                                                    NaN   \n",
       "2                                                    NaN   \n",
       "3                                                    NaN   \n",
       "4                                                    NaN   \n",
       "...                                                  ...   \n",
       "33619                                                NaN   \n",
       "33620                                                NaN   \n",
       "33621                                                NaN   \n",
       "33622        [{'start': 22, 'end': 29, 'tag': 'LGBTQI'}]   \n",
       "33623  [{'start': 211, 'end': 236, 'tag': 'bring_back...   \n",
       "\n",
       "          attachments.media_keys  geo.coordinates.type  \\\n",
       "0                            NaN                   NaN   \n",
       "1                            NaN                   NaN   \n",
       "2                            NaN                   NaN   \n",
       "3                            NaN                   NaN   \n",
       "4                            NaN                   NaN   \n",
       "...                          ...                   ...   \n",
       "33619                        NaN                   NaN   \n",
       "33620                        NaN                   NaN   \n",
       "33621                        NaN                   NaN   \n",
       "33622                        NaN                   NaN   \n",
       "33623  ['3_1408093900959592454']                   NaN   \n",
       "\n",
       "       geo.coordinates.coordinates  attachments.poll_ids  withheld.copyright  \\\n",
       "0                              NaN                   NaN                 NaN   \n",
       "1                              NaN                   NaN                 NaN   \n",
       "2                              NaN                   NaN                 NaN   \n",
       "3                              NaN                   NaN                 NaN   \n",
       "4                              NaN                   NaN                 NaN   \n",
       "...                            ...                   ...                 ...   \n",
       "33619                          NaN                   NaN                 NaN   \n",
       "33620                          NaN                   NaN                 NaN   \n",
       "33621                          NaN                   NaN                 NaN   \n",
       "33622                          NaN                   NaN                 NaN   \n",
       "33623                          NaN                   NaN                 NaN   \n",
       "\n",
       "      withheld.country_codes withheld.scope  \n",
       "0                        NaN            NaN  \n",
       "1                        NaN            NaN  \n",
       "2                        NaN            NaN  \n",
       "3                        NaN            NaN  \n",
       "4                        NaN            NaN  \n",
       "...                      ...            ...  \n",
       "33619                    NaN            NaN  \n",
       "33620                    NaN            NaN  \n",
       "33621                    NaN            NaN  \n",
       "33622                    NaN            NaN  \n",
       "33623                    NaN            NaN  \n",
       "\n",
       "[33624 rows x 31 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e08eb8c-35a2-44d2-a2c3-706945b32740",
   "metadata": {},
   "source": [
    "#### 5.2.3 Formatting dates \n",
    "**NOTE**: When trying to convert created_at to datetime format, it was discovered that one or more observations had a value of \"True\" for created_at, likely due to an error during collection. The following code locates, views, and drops the observation(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1ce90bbb-a30c-406e-be52-a62b059759dd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32768    True\n",
       "Name: created_at, dtype: object"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# locating the error\n",
    "\n",
    "df_de.loc[df_de['created_at'] == 'True', 'created_at']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b81cdc6c-d863-4827-8607-64a11bc2fec5",
   "metadata": {},
   "source": [
    "The error is located at index 32768"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "71e3230f-2841-43c0-8150-ada7f5f5f85f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "context_annotations                                          https://t.co/mY57RUt49f\n",
       "lang                                                                        everyone\n",
       "author_id                                                                      False\n",
       "text                                                         ['1465568558868480001']\n",
       "reply_settings                                                   1465568558868480001\n",
       "possibly_sensitive                                          2021-11-30T06:29:08.000Z\n",
       "edit_history_tweet_ids                                           1465568558868480001\n",
       "conversation_id                                                                  5.0\n",
       "created_at                                                                      True\n",
       "id                                                          2021-11-30T06:59:08.000Z\n",
       "edit_controls.edits_remaining                                       6e100b0c8dc4fa7e\n",
       "edit_controls.is_edit_eligible                                                     0\n",
       "edit_controls.editable_until                                                       0\n",
       "geo.place_id                                                                       0\n",
       "public_metrics.retweet_count                                                     0.0\n",
       "public_metrics.reply_count                                                       0.0\n",
       "public_metrics.like_count                                                        NaN\n",
       "public_metrics.quote_count                                                       NaN\n",
       "public_metrics.impression_count    [{'start': 45, 'end': 52, 'username': 'RND_de'...\n",
       "in_reply_to_user_id                [{'start': 54, 'end': 77, 'url': 'https://t.co...\n",
       "referenced_tweets                                                                NaN\n",
       "entities.mentions                                                                NaN\n",
       "entities.urls                                                                    NaN\n",
       "entities.hashtags                                                                NaN\n",
       "attachments.media_keys                                                           NaN\n",
       "geo.coordinates.type                                                             NaN\n",
       "geo.coordinates.coordinates                                                      NaN\n",
       "attachments.poll_ids                                                             NaN\n",
       "withheld.copyright                                                               NaN\n",
       "withheld.country_codes                                                           NaN\n",
       "withheld.scope                                                                   NaN\n",
       "Name: 32768, dtype: object"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the error\n",
    "\n",
    "df_de.iloc[32768]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a4f5dfb-1b09-44a9-a76d-8778a41b2706",
   "metadata": {},
   "source": [
    "Upon viewing index 32768, it is apparant that it indeed needs to be dropped."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "ad3f661b-ebf2-4e4e-bd57-ec234d41f266",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de = df_de.drop(df_de.index[32768])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "6b109871-0c06-4f45-ada6-55f30eb8c79f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Series([], Name: created_at, dtype: object)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# double-checking that the error is gone\n",
    "\n",
    "df_de.loc[df_de['created_at'] == 'True', 'created_at']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cfb9e02-6e5f-45b0-a291-4740103d27b2",
   "metadata": {},
   "source": [
    "Now that problematic entry has been removed, formatting dates can continue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "6653b287-80c4-4e86-819c-a71561051687",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting created_at to datetime format\n",
    "\n",
    "df_de[\"created_at\"] = pd.to_datetime(df_de[\"created_at\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "57f0132f-bf5a-4d60-aa4d-17e2b8a2fc9e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# converting it to date and creating a new column called \"date\"\n",
    "\n",
    "df_de['date'] = df_de['created_at'].dt.normalize()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "42d99314-e5b7-485c-9a67-ec8ed3d4abd6",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/by/bypfcz9933j8kf0kc0k6ch500000gn/T/ipykernel_77946/783683637.py:3: FutureWarning: Series.dt.weekofyear and Series.dt.week have been deprecated. Please use Series.dt.isocalendar().week instead.\n",
      "  df_de['week'] = df_de['created_at'].dt.week\n"
     ]
    }
   ],
   "source": [
    "# creating week, month, year, year-week, and year-month columns\n",
    "\n",
    "df_de['week'] = df_de['created_at'].dt.week\n",
    "df_de['month'] = df_de['created_at'].dt.month\n",
    "df_de['year'] = df_de['created_at'].dt.year\n",
    "df_de['year-week'] = df_de['created_at'].dt.strftime('%Y-%U')\n",
    "df_de['year-month'] = df_de['created_at'].dt.strftime('%Y-%m')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3767518c-1d69-4a66-b5f3-1ef9ee5f181d",
   "metadata": {},
   "source": [
    "#### 5.2.4 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "517a2fbb-d52b-439a-b66b-1d79dd620e6f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>...</th>\n",
       "      <th>attachments.poll_ids</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "      <th>date</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>de</td>\n",
       "      <td>1508097355458961410</td>\n",
       "      <td>Die Ukraine plant eine „False Flag“ Operation ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313222764511232']</td>\n",
       "      <td>1.584313e+18</td>\n",
       "      <td>2022-10-23 22:38:02+00:00</td>\n",
       "      <td>1584313222764511232.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1498603032640167936</td>\n",
       "      <td>@MalteKaufmann Oh wow..Ja, schlimme Zustände w...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313179298885633']</td>\n",
       "      <td>1.584091e+18</td>\n",
       "      <td>2022-10-23 22:37:51+00:00</td>\n",
       "      <td>1584313179298885632.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1310099812474331147</td>\n",
       "      <td>@HasnainKazim Ach, es ist ja so einfach mit de...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584258196608192512']</td>\n",
       "      <td>1.583888e+18</td>\n",
       "      <td>2022-10-23 18:59:22+00:00</td>\n",
       "      <td>1584258196608192512.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>2218012226</td>\n",
       "      <td>Hätte er auch gewonnen, wenn er kein Ukrainer ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584253429714944000']</td>\n",
       "      <td>1.584253e+18</td>\n",
       "      <td>2022-10-23 18:40:26+00:00</td>\n",
       "      <td>1584253429714944000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'domain': {'id': '10', 'name': 'Person', 'de...</td>\n",
       "      <td>de</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>Unsere Politiker sind allmählich komplett verr...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584212627927941120']</td>\n",
       "      <td>1.584208e+18</td>\n",
       "      <td>2022-10-23 15:58:18+00:00</td>\n",
       "      <td>1584212627927941120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33619</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1081214456099684352</td>\n",
       "      <td>@polenz_r Sie ist nicht nur wunderbar, sondern...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408456160915689479']</td>\n",
       "      <td>1.337706e+18</td>\n",
       "      <td>2021-06-25 16:04:51+00:00</td>\n",
       "      <td>1408456160915689479</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33620</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1322552611241971714</td>\n",
       "      <td>Und morgen sind die Verantwortlichen als Flüch...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408386043800399873']</td>\n",
       "      <td>1.408386e+18</td>\n",
       "      <td>2021-06-25 11:26:14+00:00</td>\n",
       "      <td>1408386043800399873</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33621</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>3021093443</td>\n",
       "      <td>4.700.000.000,- Euro\\nfür Syrer. \\nEU unterstü...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408289579115978758']</td>\n",
       "      <td>1.408290e+18</td>\n",
       "      <td>2021-06-25 05:02:55+00:00</td>\n",
       "      <td>1408289579115978758</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33622</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>10456882</td>\n",
       "      <td>Menschen hetzen gegen #LGBTQI und Flüchtlinge ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408140630149173255']</td>\n",
       "      <td>1.408141e+18</td>\n",
       "      <td>2021-06-24 19:11:02+00:00</td>\n",
       "      <td>1408140630149173255</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-24 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33623</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>3197513446</td>\n",
       "      <td>Der Fall der Familie Imerlishvilli aus Pirna b...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408093903631360002']</td>\n",
       "      <td>1.408094e+18</td>\n",
       "      <td>2021-06-24 16:05:22+00:00</td>\n",
       "      <td>1408093903631360002</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-24 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>33623 rows × 37 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     context_annotations lang  \\\n",
       "0      [{'domain': {'id': '46', 'name': 'Business Tax...   de   \n",
       "1                                                    NaN   de   \n",
       "2                                                    NaN   de   \n",
       "3                                                    NaN   de   \n",
       "4      [{'domain': {'id': '10', 'name': 'Person', 'de...   de   \n",
       "...                                                  ...  ...   \n",
       "33619                                                NaN   de   \n",
       "33620                                                NaN   de   \n",
       "33621                                                NaN   de   \n",
       "33622                                                NaN   de   \n",
       "33623                                                NaN   de   \n",
       "\n",
       "                 author_id                                               text  \\\n",
       "0      1508097355458961410  Die Ukraine plant eine „False Flag“ Operation ...   \n",
       "1      1498603032640167936  @MalteKaufmann Oh wow..Ja, schlimme Zustände w...   \n",
       "2      1310099812474331147  @HasnainKazim Ach, es ist ja so einfach mit de...   \n",
       "3               2218012226  Hätte er auch gewonnen, wenn er kein Ukrainer ...   \n",
       "4               1272454735  Unsere Politiker sind allmählich komplett verr...   \n",
       "...                    ...                                                ...   \n",
       "33619  1081214456099684352  @polenz_r Sie ist nicht nur wunderbar, sondern...   \n",
       "33620  1322552611241971714  Und morgen sind die Verantwortlichen als Flüch...   \n",
       "33621           3021093443  4.700.000.000,- Euro\\nfür Syrer. \\nEU unterstü...   \n",
       "33622             10456882  Menschen hetzen gegen #LGBTQI und Flüchtlinge ...   \n",
       "33623           3197513446  Der Fall der Familie Imerlishvilli aus Pirna b...   \n",
       "\n",
       "      reply_settings possibly_sensitive   edit_history_tweet_ids  \\\n",
       "0           everyone              False  ['1584313222764511232']   \n",
       "1           everyone              False  ['1584313179298885633']   \n",
       "2           everyone              False  ['1584258196608192512']   \n",
       "3           everyone              False  ['1584253429714944000']   \n",
       "4           everyone              False  ['1584212627927941120']   \n",
       "...              ...                ...                      ...   \n",
       "33619       everyone              False  ['1408456160915689479']   \n",
       "33620       everyone              False  ['1408386043800399873']   \n",
       "33621       everyone              False  ['1408289579115978758']   \n",
       "33622       everyone              False  ['1408140630149173255']   \n",
       "33623       everyone              False  ['1408093903631360002']   \n",
       "\n",
       "       conversation_id                created_at                     id  ...  \\\n",
       "0         1.584313e+18 2022-10-23 22:38:02+00:00  1584313222764511232.0  ...   \n",
       "1         1.584091e+18 2022-10-23 22:37:51+00:00  1584313179298885632.0  ...   \n",
       "2         1.583888e+18 2022-10-23 18:59:22+00:00  1584258196608192512.0  ...   \n",
       "3         1.584253e+18 2022-10-23 18:40:26+00:00  1584253429714944000.0  ...   \n",
       "4         1.584208e+18 2022-10-23 15:58:18+00:00  1584212627927941120.0  ...   \n",
       "...                ...                       ...                    ...  ...   \n",
       "33619     1.337706e+18 2021-06-25 16:04:51+00:00    1408456160915689479  ...   \n",
       "33620     1.408386e+18 2021-06-25 11:26:14+00:00    1408386043800399873  ...   \n",
       "33621     1.408290e+18 2021-06-25 05:02:55+00:00    1408289579115978758  ...   \n",
       "33622     1.408141e+18 2021-06-24 19:11:02+00:00    1408140630149173255  ...   \n",
       "33623     1.408094e+18 2021-06-24 16:05:22+00:00    1408093903631360002  ...   \n",
       "\n",
       "      attachments.poll_ids withheld.copyright withheld.country_codes  \\\n",
       "0                      NaN                NaN                    NaN   \n",
       "1                      NaN                NaN                    NaN   \n",
       "2                      NaN                NaN                    NaN   \n",
       "3                      NaN                NaN                    NaN   \n",
       "4                      NaN                NaN                    NaN   \n",
       "...                    ...                ...                    ...   \n",
       "33619                  NaN                NaN                    NaN   \n",
       "33620                  NaN                NaN                    NaN   \n",
       "33621                  NaN                NaN                    NaN   \n",
       "33622                  NaN                NaN                    NaN   \n",
       "33623                  NaN                NaN                    NaN   \n",
       "\n",
       "      withheld.scope                      date  week  month    year year-week  \\\n",
       "0                NaN 2022-10-23 00:00:00+00:00  42.0   10.0  2022.0   2022-43   \n",
       "1                NaN 2022-10-23 00:00:00+00:00  42.0   10.0  2022.0   2022-43   \n",
       "2                NaN 2022-10-23 00:00:00+00:00  42.0   10.0  2022.0   2022-43   \n",
       "3                NaN 2022-10-23 00:00:00+00:00  42.0   10.0  2022.0   2022-43   \n",
       "4                NaN 2022-10-23 00:00:00+00:00  42.0   10.0  2022.0   2022-43   \n",
       "...              ...                       ...   ...    ...     ...       ...   \n",
       "33619            NaN 2021-06-25 00:00:00+00:00  25.0    6.0  2021.0   2021-25   \n",
       "33620            NaN 2021-06-25 00:00:00+00:00  25.0    6.0  2021.0   2021-25   \n",
       "33621            NaN 2021-06-25 00:00:00+00:00  25.0    6.0  2021.0   2021-25   \n",
       "33622            NaN 2021-06-24 00:00:00+00:00  25.0    6.0  2021.0   2021-25   \n",
       "33623            NaN 2021-06-24 00:00:00+00:00  25.0    6.0  2021.0   2021-25   \n",
       "\n",
       "      year-month  \n",
       "0        2022-10  \n",
       "1        2022-10  \n",
       "2        2022-10  \n",
       "3        2022-10  \n",
       "4        2022-10  \n",
       "...          ...  \n",
       "33619    2021-06  \n",
       "33620    2021-06  \n",
       "33621    2021-06  \n",
       "33622    2021-06  \n",
       "33623    2021-06  \n",
       "\n",
       "[33623 rows x 37 columns]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_de"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5020ce4d-0d9a-4b47-a7c7-272394b55aca",
   "metadata": {},
   "source": [
    "### 5.3 Merging the English-language and German-language dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "583baadb-58c2-4bef-ac4a-c83cedb516bf",
   "metadata": {},
   "source": [
    "#### 5.3.1 Creating language column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "0fa1db84-d658-43e3-9769-1e1b0d77bdb4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_de['Language'] = 'German'\n",
    "df_eng['Language'] = 'English'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9994fd7f-9de5-49a7-a3c2-edef1fa7c82b",
   "metadata": {},
   "source": [
    "#### 5.3.2 Merging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "9b552b97-2f49-4a61-b826-057488320ff5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a joint data frame\n",
    "\n",
    "df_uk = pd.concat([df_de, df_eng], ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bc6d3ee-3d7a-42ab-b4bf-71e1905da0d0",
   "metadata": {},
   "source": [
    "#### 5.3.3 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "5acd4de6-4e23-4ec2-975c-2bccf55a6b0b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>...</th>\n",
       "      <th>withheld.scope</th>\n",
       "      <th>date</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>entities.annotations</th>\n",
       "      <th>entities.cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>de</td>\n",
       "      <td>1508097355458961410</td>\n",
       "      <td>Die Ukraine plant eine „False Flag“ Operation ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313222764511232']</td>\n",
       "      <td>1.584313e+18</td>\n",
       "      <td>2022-10-23 22:38:02+00:00</td>\n",
       "      <td>1584313222764511232.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1498603032640167936</td>\n",
       "      <td>@MalteKaufmann Oh wow..Ja, schlimme Zustände w...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313179298885633']</td>\n",
       "      <td>1.584091e+18</td>\n",
       "      <td>2022-10-23 22:37:51+00:00</td>\n",
       "      <td>1584313179298885632.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1310099812474331147</td>\n",
       "      <td>@HasnainKazim Ach, es ist ja so einfach mit de...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584258196608192512']</td>\n",
       "      <td>1.583888e+18</td>\n",
       "      <td>2022-10-23 18:59:22+00:00</td>\n",
       "      <td>1584258196608192512.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>2218012226</td>\n",
       "      <td>Hätte er auch gewonnen, wenn er kein Ukrainer ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584253429714944000']</td>\n",
       "      <td>1.584253e+18</td>\n",
       "      <td>2022-10-23 18:40:26+00:00</td>\n",
       "      <td>1584253429714944000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'domain': {'id': '10', 'name': 'Person', 'de...</td>\n",
       "      <td>de</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>Unsere Politiker sind allmählich komplett verr...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584212627927941120']</td>\n",
       "      <td>1.584208e+18</td>\n",
       "      <td>2022-10-23 15:58:18+00:00</td>\n",
       "      <td>1584212627927941120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41629</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>110402493</td>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409522855814041606']</td>\n",
       "      <td>1.407592e+18</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>1409522855814041606</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41630</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409135295895855104']</td>\n",
       "      <td>1.409134e+18</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>1409135295895855104</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 15, 'end': 20, 'probability': 0.963...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41631</th>\n",
       "      <td>[{'domain': {'id': '130', 'name': 'Multimedia ...</td>\n",
       "      <td>en</td>\n",
       "      <td>563381751</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409068320947593220']</td>\n",
       "      <td>1.409068e+18</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>1409068320947593220</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 59, 'end': 63, 'probability': 0.416...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41632</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408697379063271425']</td>\n",
       "      <td>1.408457e+18</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>1408697379063271425</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 59, 'end': 68, 'probability': 0.670...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41633</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>134401670</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>True</td>\n",
       "      <td>['1408531015518326784']</td>\n",
       "      <td>1.408531e+18</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>1408531015518326784</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 1, 'end': 9, 'probability': 0.9824,...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>41634 rows × 40 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     context_annotations lang  \\\n",
       "0      [{'domain': {'id': '46', 'name': 'Business Tax...   de   \n",
       "1                                                    NaN   de   \n",
       "2                                                    NaN   de   \n",
       "3                                                    NaN   de   \n",
       "4      [{'domain': {'id': '10', 'name': 'Person', 'de...   de   \n",
       "...                                                  ...  ...   \n",
       "41629                                                NaN   en   \n",
       "41630                                                NaN   en   \n",
       "41631  [{'domain': {'id': '130', 'name': 'Multimedia ...   en   \n",
       "41632                                                NaN   en   \n",
       "41633                                                NaN   en   \n",
       "\n",
       "                 author_id                                               text  \\\n",
       "0      1508097355458961410  Die Ukraine plant eine „False Flag“ Operation ...   \n",
       "1      1498603032640167936  @MalteKaufmann Oh wow..Ja, schlimme Zustände w...   \n",
       "2      1310099812474331147  @HasnainKazim Ach, es ist ja so einfach mit de...   \n",
       "3               2218012226  Hätte er auch gewonnen, wenn er kein Ukrainer ...   \n",
       "4               1272454735  Unsere Politiker sind allmählich komplett verr...   \n",
       "...                    ...                                                ...   \n",
       "41629            110402493  For day 1 of week 2, @AnnaMariaKonsta discusse...   \n",
       "41630           2521808908  @ariadneconill Europe is racist, but in a diff...   \n",
       "41631            563381751  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "41632           2591892350  @simongerman600 I must have missed the great f...   \n",
       "41633            134401670  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "      reply_settings possibly_sensitive   edit_history_tweet_ids  \\\n",
       "0           everyone              False  ['1584313222764511232']   \n",
       "1           everyone              False  ['1584313179298885633']   \n",
       "2           everyone              False  ['1584258196608192512']   \n",
       "3           everyone              False  ['1584253429714944000']   \n",
       "4           everyone              False  ['1584212627927941120']   \n",
       "...              ...                ...                      ...   \n",
       "41629       everyone              False  ['1409522855814041606']   \n",
       "41630       everyone              False  ['1409135295895855104']   \n",
       "41631       everyone              False  ['1409068320947593220']   \n",
       "41632       everyone              False  ['1408697379063271425']   \n",
       "41633       everyone               True  ['1408531015518326784']   \n",
       "\n",
       "       conversation_id                created_at                     id  ...  \\\n",
       "0         1.584313e+18 2022-10-23 22:38:02+00:00  1584313222764511232.0  ...   \n",
       "1         1.584091e+18 2022-10-23 22:37:51+00:00  1584313179298885632.0  ...   \n",
       "2         1.583888e+18 2022-10-23 18:59:22+00:00  1584258196608192512.0  ...   \n",
       "3         1.584253e+18 2022-10-23 18:40:26+00:00  1584253429714944000.0  ...   \n",
       "4         1.584208e+18 2022-10-23 15:58:18+00:00  1584212627927941120.0  ...   \n",
       "...                ...                       ...                    ...  ...   \n",
       "41629     1.407592e+18 2021-06-28 14:43:31+00:00    1409522855814041606  ...   \n",
       "41630     1.409134e+18 2021-06-27 13:03:29+00:00    1409135295895855104  ...   \n",
       "41631     1.409068e+18 2021-06-27 08:37:21+00:00    1409068320947593220  ...   \n",
       "41632     1.408457e+18 2021-06-26 08:03:22+00:00    1408697379063271425  ...   \n",
       "41633     1.408531e+18 2021-06-25 21:02:17+00:00    1408531015518326784  ...   \n",
       "\n",
       "      withheld.scope                      date  week month    year  year-week  \\\n",
       "0                NaN 2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43   \n",
       "1                NaN 2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43   \n",
       "2                NaN 2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43   \n",
       "3                NaN 2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43   \n",
       "4                NaN 2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43   \n",
       "...              ...                       ...   ...   ...     ...        ...   \n",
       "41629            NaN 2021-06-28 00:00:00+00:00  26.0   6.0  2021.0    2021-26   \n",
       "41630            NaN 2021-06-27 00:00:00+00:00  25.0   6.0  2021.0    2021-26   \n",
       "41631            NaN 2021-06-27 00:00:00+00:00  25.0   6.0  2021.0    2021-26   \n",
       "41632            NaN 2021-06-26 00:00:00+00:00  25.0   6.0  2021.0    2021-25   \n",
       "41633            NaN 2021-06-25 00:00:00+00:00  25.0   6.0  2021.0    2021-25   \n",
       "\n",
       "       year-month  Language  \\\n",
       "0         2022-10    German   \n",
       "1         2022-10    German   \n",
       "2         2022-10    German   \n",
       "3         2022-10    German   \n",
       "4         2022-10    German   \n",
       "...           ...       ...   \n",
       "41629     2021-06   English   \n",
       "41630     2021-06   English   \n",
       "41631     2021-06   English   \n",
       "41632     2021-06   English   \n",
       "41633     2021-06   English   \n",
       "\n",
       "                                    entities.annotations entities.cashtags  \n",
       "0                                                    NaN               NaN  \n",
       "1                                                    NaN               NaN  \n",
       "2                                                    NaN               NaN  \n",
       "3                                                    NaN               NaN  \n",
       "4                                                    NaN               NaN  \n",
       "...                                                  ...               ...  \n",
       "41629                                                NaN               NaN  \n",
       "41630  [{'start': 15, 'end': 20, 'probability': 0.963...               NaN  \n",
       "41631  [{'start': 59, 'end': 63, 'probability': 0.416...               NaN  \n",
       "41632  [{'start': 59, 'end': 68, 'probability': 0.670...               NaN  \n",
       "41633  [{'start': 1, 'end': 9, 'probability': 0.9824,...               NaN  \n",
       "\n",
       "[41634 rows x 40 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_uk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c036de3-b844-43df-88e4-c57fa667224d",
   "metadata": {},
   "source": [
    "## 6. Combining Syrian inflow and Ukrainian inflow  dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07c903e5-5c14-4032-91d1-de758627b9ff",
   "metadata": {},
   "source": [
    "### 6.1. Creating inflow column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "d40e7099-c3c1-41bd-b4ad-a176df888c1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding a new column, inflow, indicating Syrians or Ukrainians\n",
    "\n",
    "df_syr['inflow'] = 'Syrians'\n",
    "df_uk['inflow'] = 'Ukrainians'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4535b6c-7094-4b99-94a9-370aad5d2357",
   "metadata": {},
   "source": [
    "### 6.2. Viewing the dataframes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "ffcab34f-19e3-40f5-a39d-e1de1453bfed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>entities.annotations</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "      <th>inflow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.229216e+17</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>722921572810366977</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>['722921572810366977']</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228995e+17</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>722899547039473665</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>['722899547039473665']</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228974e+17</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>722897370313195521</td>\n",
       "      <td>224607633</td>\n",
       "      <td>['722897370313195521']</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228536e+17</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>722847691101880320</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>['722853635751809025']</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 83, 'end': 106, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>7.228240e+17</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>722824011063799809</td>\n",
       "      <td>606265303</td>\n",
       "      <td>['722824011063799809']</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24626</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.476947e+17</td>\n",
       "      <td>DE-News : Syrian President Bashar Assad, shown...</td>\n",
       "      <td>547694737776320513</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['547694737776320513']</td>\n",
       "      <td>2014-12-24 10:06:15+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 117, 'end': 139, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>[{'start': 27, 'end': 38, 'probability': 0.949...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24627</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.476090e+17</td>\n",
       "      <td>#CelioGermanyDesk German President Gauck calls...</td>\n",
       "      <td>547609017623654400</td>\n",
       "      <td>271776179</td>\n",
       "      <td>['547609017623654400']</td>\n",
       "      <td>2014-12-24 04:25:38+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 96, 'end': 118, 'url': 'http://t.co...</td>\n",
       "      <td>...</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-24 00:00:00+00:00</td>\n",
       "      <td>[{'start': 1, 'end': 16, 'probability': 0.2996...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24628</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.471029e+17</td>\n",
       "      <td>#munich is colourful! welcome #refugees! no #p...</td>\n",
       "      <td>547102939484286976</td>\n",
       "      <td>15105772</td>\n",
       "      <td>['547102939484286976']</td>\n",
       "      <td>2014-12-22 18:54:40+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 65, 'end': 87, 'url': 'http://t.co/...</td>\n",
       "      <td>...</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-22 00:00:00+00:00</td>\n",
       "      <td>[{'start': 1, 'end': 6, 'probability': 0.3992,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24629</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.466994e+17</td>\n",
       "      <td>DE-News : There is little to break the monoton...</td>\n",
       "      <td>546699388316184577</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['546699388316184577']</td>\n",
       "      <td>2014-12-21 16:11:06+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 101, 'end': 123, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>[{'start': 3, 'end': 6, 'probability': 0.4604,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24630</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>5.465444e+17</td>\n",
       "      <td>DE-News : View of a vacant lot earmarked for r...</td>\n",
       "      <td>546544368169918464</td>\n",
       "      <td>186899860</td>\n",
       "      <td>['546544368169918464']</td>\n",
       "      <td>2014-12-21 05:55:06+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 109, 'end': 131, 'url': 'http://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2014.0</td>\n",
       "      <td>2014-51</td>\n",
       "      <td>2014-12</td>\n",
       "      <td>English</td>\n",
       "      <td>2014-12-21 00:00:00+00:00</td>\n",
       "      <td>[{'start': 57, 'end': 62, 'probability': 0.848...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>24631 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      reply_settings possibly_sensitive            id  \\\n",
       "0           everyone              False  7.229216e+17   \n",
       "1           everyone              False  7.228995e+17   \n",
       "2           everyone              False  7.228974e+17   \n",
       "3           everyone              False  7.228536e+17   \n",
       "4           everyone              False  7.228240e+17   \n",
       "...              ...                ...           ...   \n",
       "24626       everyone              False  5.476947e+17   \n",
       "24627       everyone              False  5.476090e+17   \n",
       "24628       everyone              False  5.471029e+17   \n",
       "24629       everyone              False  5.466994e+17   \n",
       "24630       everyone              False  5.465444e+17   \n",
       "\n",
       "                                                    text     conversation_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  722921572810366977   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  722899547039473665   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...  722897370313195521   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  722847691101880320   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...  722824011063799809   \n",
       "...                                                  ...                 ...   \n",
       "24626  DE-News : Syrian President Bashar Assad, shown...  547694737776320513   \n",
       "24627  #CelioGermanyDesk German President Gauck calls...  547609017623654400   \n",
       "24628  #munich is colourful! welcome #refugees! no #p...  547102939484286976   \n",
       "24629  DE-News : There is little to break the monoton...  546699388316184577   \n",
       "24630  DE-News : View of a vacant lot earmarked for r...  546544368169918464   \n",
       "\n",
       "        author_id  edit_history_tweet_ids                created_at lang  \\\n",
       "0      4122038069  ['722921572810366977'] 2016-04-20 22:55:08+00:00   de   \n",
       "1      1179543852  ['722899547039473665'] 2016-04-20 21:27:37+00:00   de   \n",
       "2       224607633  ['722897370313195521'] 2016-04-20 21:18:58+00:00   de   \n",
       "3      2480764313  ['722853635751809025'] 2016-04-20 18:25:11+00:00   de   \n",
       "4       606265303  ['722824011063799809'] 2016-04-20 16:27:28+00:00   de   \n",
       "...           ...                     ...                       ...  ...   \n",
       "24626   186899860  ['547694737776320513'] 2014-12-24 10:06:15+00:00   en   \n",
       "24627   271776179  ['547609017623654400'] 2014-12-24 04:25:38+00:00   en   \n",
       "24628    15105772  ['547102939484286976'] 2014-12-22 18:54:40+00:00   en   \n",
       "24629   186899860  ['546699388316184577'] 2014-12-21 16:11:06+00:00   en   \n",
       "24630   186899860  ['546544368169918464'] 2014-12-21 05:55:06+00:00   en   \n",
       "\n",
       "                                           entities.urls  ...    year  \\\n",
       "0      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...  2016.0   \n",
       "1                                                    NaN  ...  2016.0   \n",
       "2                                                    NaN  ...  2016.0   \n",
       "3      [{'start': 83, 'end': 106, 'url': 'https://t.c...  ...  2016.0   \n",
       "4      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...  2016.0   \n",
       "...                                                  ...  ...     ...   \n",
       "24626  [{'start': 117, 'end': 139, 'url': 'http://t.c...  ...  2014.0   \n",
       "24627  [{'start': 96, 'end': 118, 'url': 'http://t.co...  ...  2014.0   \n",
       "24628  [{'start': 65, 'end': 87, 'url': 'http://t.co/...  ...  2014.0   \n",
       "24629  [{'start': 101, 'end': 123, 'url': 'http://t.c...  ...  2014.0   \n",
       "24630  [{'start': 109, 'end': 131, 'url': 'http://t.c...  ...  2014.0   \n",
       "\n",
       "       year-week year-month Language                      date  \\\n",
       "0        2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "1        2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "2        2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "3        2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "4        2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "...          ...        ...      ...                       ...   \n",
       "24626    2014-51    2014-12  English 2014-12-24 00:00:00+00:00   \n",
       "24627    2014-51    2014-12  English 2014-12-24 00:00:00+00:00   \n",
       "24628    2014-51    2014-12  English 2014-12-22 00:00:00+00:00   \n",
       "24629    2014-51    2014-12  English 2014-12-21 00:00:00+00:00   \n",
       "24630    2014-51    2014-12  English 2014-12-21 00:00:00+00:00   \n",
       "\n",
       "                                    entities.annotations withheld.copyright  \\\n",
       "0                                                    NaN                NaN   \n",
       "1                                                    NaN                NaN   \n",
       "2                                                    NaN                NaN   \n",
       "3                                                    NaN                NaN   \n",
       "4                                                    NaN                NaN   \n",
       "...                                                  ...                ...   \n",
       "24626  [{'start': 27, 'end': 38, 'probability': 0.949...                NaN   \n",
       "24627  [{'start': 1, 'end': 16, 'probability': 0.2996...                NaN   \n",
       "24628  [{'start': 1, 'end': 6, 'probability': 0.3992,...                NaN   \n",
       "24629  [{'start': 3, 'end': 6, 'probability': 0.4604,...                NaN   \n",
       "24630  [{'start': 57, 'end': 62, 'probability': 0.848...                NaN   \n",
       "\n",
       "      withheld.country_codes  withheld.scope   inflow  \n",
       "0                        NaN             NaN  Syrians  \n",
       "1                        NaN             NaN  Syrians  \n",
       "2                        NaN             NaN  Syrians  \n",
       "3                        NaN             NaN  Syrians  \n",
       "4                        NaN             NaN  Syrians  \n",
       "...                      ...             ...      ...  \n",
       "24626                    NaN             NaN  Syrians  \n",
       "24627                    NaN             NaN  Syrians  \n",
       "24628                    NaN             NaN  Syrians  \n",
       "24629                    NaN             NaN  Syrians  \n",
       "24630                    NaN             NaN  Syrians  \n",
       "\n",
       "[24631 rows x 41 columns]"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_syr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "127ffef5-c01a-4fa9-b368-242127a6d6de",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>context_annotations</th>\n",
       "      <th>lang</th>\n",
       "      <th>author_id</th>\n",
       "      <th>text</th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>...</th>\n",
       "      <th>date</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>entities.annotations</th>\n",
       "      <th>entities.cashtags</th>\n",
       "      <th>inflow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[{'domain': {'id': '46', 'name': 'Business Tax...</td>\n",
       "      <td>de</td>\n",
       "      <td>1508097355458961410</td>\n",
       "      <td>Die Ukraine plant eine „False Flag“ Operation ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313222764511232']</td>\n",
       "      <td>1.584313e+18</td>\n",
       "      <td>2022-10-23 22:38:02+00:00</td>\n",
       "      <td>1584313222764511232.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1498603032640167936</td>\n",
       "      <td>@MalteKaufmann Oh wow..Ja, schlimme Zustände w...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584313179298885633']</td>\n",
       "      <td>1.584091e+18</td>\n",
       "      <td>2022-10-23 22:37:51+00:00</td>\n",
       "      <td>1584313179298885632.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>1310099812474331147</td>\n",
       "      <td>@HasnainKazim Ach, es ist ja so einfach mit de...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584258196608192512']</td>\n",
       "      <td>1.583888e+18</td>\n",
       "      <td>2022-10-23 18:59:22+00:00</td>\n",
       "      <td>1584258196608192512.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>NaN</td>\n",
       "      <td>de</td>\n",
       "      <td>2218012226</td>\n",
       "      <td>Hätte er auch gewonnen, wenn er kein Ukrainer ...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584253429714944000']</td>\n",
       "      <td>1.584253e+18</td>\n",
       "      <td>2022-10-23 18:40:26+00:00</td>\n",
       "      <td>1584253429714944000.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[{'domain': {'id': '10', 'name': 'Person', 'de...</td>\n",
       "      <td>de</td>\n",
       "      <td>1272454735</td>\n",
       "      <td>Unsere Politiker sind allmählich komplett verr...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1584212627927941120']</td>\n",
       "      <td>1.584208e+18</td>\n",
       "      <td>2022-10-23 15:58:18+00:00</td>\n",
       "      <td>1584212627927941120.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2022-10-23 00:00:00+00:00</td>\n",
       "      <td>42.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>2022.0</td>\n",
       "      <td>2022-43</td>\n",
       "      <td>2022-10</td>\n",
       "      <td>German</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41629</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>110402493</td>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409522855814041606']</td>\n",
       "      <td>1.407592e+18</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>1409522855814041606</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41630</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409135295895855104']</td>\n",
       "      <td>1.409134e+18</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>1409135295895855104</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 15, 'end': 20, 'probability': 0.963...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41631</th>\n",
       "      <td>[{'domain': {'id': '130', 'name': 'Multimedia ...</td>\n",
       "      <td>en</td>\n",
       "      <td>563381751</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1409068320947593220']</td>\n",
       "      <td>1.409068e+18</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>1409068320947593220</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 59, 'end': 63, 'probability': 0.416...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41632</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>['1408697379063271425']</td>\n",
       "      <td>1.408457e+18</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>1408697379063271425</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 59, 'end': 68, 'probability': 0.670...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41633</th>\n",
       "      <td>NaN</td>\n",
       "      <td>en</td>\n",
       "      <td>134401670</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>everyone</td>\n",
       "      <td>True</td>\n",
       "      <td>['1408531015518326784']</td>\n",
       "      <td>1.408531e+18</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>1408531015518326784</td>\n",
       "      <td>...</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>[{'start': 1, 'end': 9, 'probability': 0.9824,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>41634 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     context_annotations lang  \\\n",
       "0      [{'domain': {'id': '46', 'name': 'Business Tax...   de   \n",
       "1                                                    NaN   de   \n",
       "2                                                    NaN   de   \n",
       "3                                                    NaN   de   \n",
       "4      [{'domain': {'id': '10', 'name': 'Person', 'de...   de   \n",
       "...                                                  ...  ...   \n",
       "41629                                                NaN   en   \n",
       "41630                                                NaN   en   \n",
       "41631  [{'domain': {'id': '130', 'name': 'Multimedia ...   en   \n",
       "41632                                                NaN   en   \n",
       "41633                                                NaN   en   \n",
       "\n",
       "                 author_id                                               text  \\\n",
       "0      1508097355458961410  Die Ukraine plant eine „False Flag“ Operation ...   \n",
       "1      1498603032640167936  @MalteKaufmann Oh wow..Ja, schlimme Zustände w...   \n",
       "2      1310099812474331147  @HasnainKazim Ach, es ist ja so einfach mit de...   \n",
       "3               2218012226  Hätte er auch gewonnen, wenn er kein Ukrainer ...   \n",
       "4               1272454735  Unsere Politiker sind allmählich komplett verr...   \n",
       "...                    ...                                                ...   \n",
       "41629            110402493  For day 1 of week 2, @AnnaMariaKonsta discusse...   \n",
       "41630           2521808908  @ariadneconill Europe is racist, but in a diff...   \n",
       "41631            563381751  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "41632           2591892350  @simongerman600 I must have missed the great f...   \n",
       "41633            134401670  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "      reply_settings possibly_sensitive   edit_history_tweet_ids  \\\n",
       "0           everyone              False  ['1584313222764511232']   \n",
       "1           everyone              False  ['1584313179298885633']   \n",
       "2           everyone              False  ['1584258196608192512']   \n",
       "3           everyone              False  ['1584253429714944000']   \n",
       "4           everyone              False  ['1584212627927941120']   \n",
       "...              ...                ...                      ...   \n",
       "41629       everyone              False  ['1409522855814041606']   \n",
       "41630       everyone              False  ['1409135295895855104']   \n",
       "41631       everyone              False  ['1409068320947593220']   \n",
       "41632       everyone              False  ['1408697379063271425']   \n",
       "41633       everyone               True  ['1408531015518326784']   \n",
       "\n",
       "       conversation_id                created_at                     id  ...  \\\n",
       "0         1.584313e+18 2022-10-23 22:38:02+00:00  1584313222764511232.0  ...   \n",
       "1         1.584091e+18 2022-10-23 22:37:51+00:00  1584313179298885632.0  ...   \n",
       "2         1.583888e+18 2022-10-23 18:59:22+00:00  1584258196608192512.0  ...   \n",
       "3         1.584253e+18 2022-10-23 18:40:26+00:00  1584253429714944000.0  ...   \n",
       "4         1.584208e+18 2022-10-23 15:58:18+00:00  1584212627927941120.0  ...   \n",
       "...                ...                       ...                    ...  ...   \n",
       "41629     1.407592e+18 2021-06-28 14:43:31+00:00    1409522855814041606  ...   \n",
       "41630     1.409134e+18 2021-06-27 13:03:29+00:00    1409135295895855104  ...   \n",
       "41631     1.409068e+18 2021-06-27 08:37:21+00:00    1409068320947593220  ...   \n",
       "41632     1.408457e+18 2021-06-26 08:03:22+00:00    1408697379063271425  ...   \n",
       "41633     1.408531e+18 2021-06-25 21:02:17+00:00    1408531015518326784  ...   \n",
       "\n",
       "                           date  week month    year  year-week  year-month  \\\n",
       "0     2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43     2022-10   \n",
       "1     2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43     2022-10   \n",
       "2     2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43     2022-10   \n",
       "3     2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43     2022-10   \n",
       "4     2022-10-23 00:00:00+00:00  42.0  10.0  2022.0    2022-43     2022-10   \n",
       "...                         ...   ...   ...     ...        ...         ...   \n",
       "41629 2021-06-28 00:00:00+00:00  26.0   6.0  2021.0    2021-26     2021-06   \n",
       "41630 2021-06-27 00:00:00+00:00  25.0   6.0  2021.0    2021-26     2021-06   \n",
       "41631 2021-06-27 00:00:00+00:00  25.0   6.0  2021.0    2021-26     2021-06   \n",
       "41632 2021-06-26 00:00:00+00:00  25.0   6.0  2021.0    2021-25     2021-06   \n",
       "41633 2021-06-25 00:00:00+00:00  25.0   6.0  2021.0    2021-25     2021-06   \n",
       "\n",
       "       Language                               entities.annotations  \\\n",
       "0        German                                                NaN   \n",
       "1        German                                                NaN   \n",
       "2        German                                                NaN   \n",
       "3        German                                                NaN   \n",
       "4        German                                                NaN   \n",
       "...         ...                                                ...   \n",
       "41629   English                                                NaN   \n",
       "41630   English  [{'start': 15, 'end': 20, 'probability': 0.963...   \n",
       "41631   English  [{'start': 59, 'end': 63, 'probability': 0.416...   \n",
       "41632   English  [{'start': 59, 'end': 68, 'probability': 0.670...   \n",
       "41633   English  [{'start': 1, 'end': 9, 'probability': 0.9824,...   \n",
       "\n",
       "      entities.cashtags      inflow  \n",
       "0                   NaN  Ukrainians  \n",
       "1                   NaN  Ukrainians  \n",
       "2                   NaN  Ukrainians  \n",
       "3                   NaN  Ukrainians  \n",
       "4                   NaN  Ukrainians  \n",
       "...                 ...         ...  \n",
       "41629               NaN  Ukrainians  \n",
       "41630               NaN  Ukrainians  \n",
       "41631               NaN  Ukrainians  \n",
       "41632               NaN  Ukrainians  \n",
       "41633               NaN  Ukrainians  \n",
       "\n",
       "[41634 rows x 41 columns]"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_uk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be8a785a-735d-4b4b-aec0-1ffe1ed0d944",
   "metadata": {},
   "source": [
    "### 6.3. Merging "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "a2f85a37-18a7-4181-8af8-cd2015e5feb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a joint data frame\n",
    "\n",
    "df = pd.concat([df_syr, df_uk], ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c353e235-e1e9-4eeb-9113-79dec3ea3c95",
   "metadata": {},
   "source": [
    "### 6.4 Viewing the dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "cac0cb03-c044-480c-8396-539595cf0dd6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reply_settings</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "      <th>conversation_id</th>\n",
       "      <th>author_id</th>\n",
       "      <th>edit_history_tweet_ids</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>entities.urls</th>\n",
       "      <th>...</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>entities.annotations</th>\n",
       "      <th>withheld.copyright</th>\n",
       "      <th>withheld.country_codes</th>\n",
       "      <th>withheld.scope</th>\n",
       "      <th>inflow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>722921572810366976.0</td>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>722921572810366977</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>['722921572810366977']</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>722899547039473664.0</td>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>722899547039473665</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>['722899547039473665']</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>722897370313195520.0</td>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>722897370313195521</td>\n",
       "      <td>224607633</td>\n",
       "      <td>['722897370313195521']</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>722853635751809024.0</td>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>722847691101880320</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>['722853635751809025']</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 83, 'end': 106, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>722824011063799808.0</td>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>722824011063799809</td>\n",
       "      <td>606265303</td>\n",
       "      <td>['722824011063799809']</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>[{'start': 92, 'end': 115, 'url': 'https://t.c...</td>\n",
       "      <td>...</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66260</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>1409522855814041606</td>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>1407591830422687744.0</td>\n",
       "      <td>110402493</td>\n",
       "      <td>['1409522855814041606']</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>1409135295895855104</td>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>1409134355037970432.0</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>['1409135295895855104']</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>[{'start': 15, 'end': 20, 'probability': 0.963...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>1409068320947593220</td>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>1409068228270231552.0</td>\n",
       "      <td>563381751</td>\n",
       "      <td>['1409068320947593220']</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 272, 'end': 295, 'url': 'https://t....</td>\n",
       "      <td>...</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>[{'start': 59, 'end': 63, 'probability': 0.416...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>everyone</td>\n",
       "      <td>False</td>\n",
       "      <td>1408697379063271425</td>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>1408457471707000832.0</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>['1408697379063271425']</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>[{'start': 59, 'end': 68, 'probability': 0.670...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>everyone</td>\n",
       "      <td>True</td>\n",
       "      <td>1408531015518326784</td>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>1408531015518326784.0</td>\n",
       "      <td>134401670</td>\n",
       "      <td>['1408531015518326784']</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>[{'start': 239, 'end': 262, 'url': 'https://t....</td>\n",
       "      <td>...</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>[{'start': 1, 'end': 9, 'probability': 0.9824,...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66265 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      reply_settings possibly_sensitive                    id  \\\n",
       "0           everyone              False  722921572810366976.0   \n",
       "1           everyone              False  722899547039473664.0   \n",
       "2           everyone              False  722897370313195520.0   \n",
       "3           everyone              False  722853635751809024.0   \n",
       "4           everyone              False  722824011063799808.0   \n",
       "...              ...                ...                   ...   \n",
       "66260       everyone              False   1409522855814041606   \n",
       "66261       everyone              False   1409135295895855104   \n",
       "66262       everyone              False   1409068320947593220   \n",
       "66263       everyone              False   1408697379063271425   \n",
       "66264       everyone               True   1408531015518326784   \n",
       "\n",
       "                                                    text  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   \n",
       "...                                                  ...   \n",
       "66260  For day 1 of week 2, @AnnaMariaKonsta discusse...   \n",
       "66261  @ariadneconill Europe is racist, but in a diff...   \n",
       "66262  A labour of love, inspired by Middle-earth.\\n\\...   \n",
       "66263  @simongerman600 I must have missed the great f...   \n",
       "66264  #Wuerzburg today an African immigrant or refug...   \n",
       "\n",
       "             conversation_id   author_id   edit_history_tweet_ids  \\\n",
       "0         722921572810366977  4122038069   ['722921572810366977']   \n",
       "1         722899547039473665  1179543852   ['722899547039473665']   \n",
       "2         722897370313195521   224607633   ['722897370313195521']   \n",
       "3         722847691101880320  2480764313   ['722853635751809025']   \n",
       "4         722824011063799809   606265303   ['722824011063799809']   \n",
       "...                      ...         ...                      ...   \n",
       "66260  1407591830422687744.0   110402493  ['1409522855814041606']   \n",
       "66261  1409134355037970432.0  2521808908  ['1409135295895855104']   \n",
       "66262  1409068228270231552.0   563381751  ['1409068320947593220']   \n",
       "66263  1408457471707000832.0  2591892350  ['1408697379063271425']   \n",
       "66264  1408531015518326784.0   134401670  ['1408531015518326784']   \n",
       "\n",
       "                     created_at lang  \\\n",
       "0     2016-04-20 22:55:08+00:00   de   \n",
       "1     2016-04-20 21:27:37+00:00   de   \n",
       "2     2016-04-20 21:18:58+00:00   de   \n",
       "3     2016-04-20 18:25:11+00:00   de   \n",
       "4     2016-04-20 16:27:28+00:00   de   \n",
       "...                         ...  ...   \n",
       "66260 2021-06-28 14:43:31+00:00   en   \n",
       "66261 2021-06-27 13:03:29+00:00   en   \n",
       "66262 2021-06-27 08:37:21+00:00   en   \n",
       "66263 2021-06-26 08:03:22+00:00   en   \n",
       "66264 2021-06-25 21:02:17+00:00   en   \n",
       "\n",
       "                                           entities.urls  ...    year  \\\n",
       "0      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...  2016.0   \n",
       "1                                                    NaN  ...  2016.0   \n",
       "2                                                    NaN  ...  2016.0   \n",
       "3      [{'start': 83, 'end': 106, 'url': 'https://t.c...  ...  2016.0   \n",
       "4      [{'start': 92, 'end': 115, 'url': 'https://t.c...  ...  2016.0   \n",
       "...                                                  ...  ...     ...   \n",
       "66260                                                NaN  ...  2021.0   \n",
       "66261                                                NaN  ...  2021.0   \n",
       "66262  [{'start': 272, 'end': 295, 'url': 'https://t....  ...  2021.0   \n",
       "66263                                                NaN  ...  2021.0   \n",
       "66264  [{'start': 239, 'end': 262, 'url': 'https://t....  ...  2021.0   \n",
       "\n",
       "      year-week year-month Language                      date  \\\n",
       "0       2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "1       2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "2       2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "3       2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "4       2016-16    2016-04   German 2016-04-20 00:00:00+00:00   \n",
       "...         ...        ...      ...                       ...   \n",
       "66260   2021-26    2021-06  English 2021-06-28 00:00:00+00:00   \n",
       "66261   2021-26    2021-06  English 2021-06-27 00:00:00+00:00   \n",
       "66262   2021-26    2021-06  English 2021-06-27 00:00:00+00:00   \n",
       "66263   2021-25    2021-06  English 2021-06-26 00:00:00+00:00   \n",
       "66264   2021-25    2021-06  English 2021-06-25 00:00:00+00:00   \n",
       "\n",
       "                                    entities.annotations withheld.copyright  \\\n",
       "0                                                    NaN                NaN   \n",
       "1                                                    NaN                NaN   \n",
       "2                                                    NaN                NaN   \n",
       "3                                                    NaN                NaN   \n",
       "4                                                    NaN                NaN   \n",
       "...                                                  ...                ...   \n",
       "66260                                                NaN                NaN   \n",
       "66261  [{'start': 15, 'end': 20, 'probability': 0.963...                NaN   \n",
       "66262  [{'start': 59, 'end': 63, 'probability': 0.416...                NaN   \n",
       "66263  [{'start': 59, 'end': 68, 'probability': 0.670...                NaN   \n",
       "66264  [{'start': 1, 'end': 9, 'probability': 0.9824,...                NaN   \n",
       "\n",
       "      withheld.country_codes withheld.scope      inflow  \n",
       "0                        NaN            NaN     Syrians  \n",
       "1                        NaN            NaN     Syrians  \n",
       "2                        NaN            NaN     Syrians  \n",
       "3                        NaN            NaN     Syrians  \n",
       "4                        NaN            NaN     Syrians  \n",
       "...                      ...            ...         ...  \n",
       "66260                    NaN            NaN  Ukrainians  \n",
       "66261                    NaN            NaN  Ukrainians  \n",
       "66262                    NaN            NaN  Ukrainians  \n",
       "66263                    NaN            NaN  Ukrainians  \n",
       "66264                    NaN            NaN  Ukrainians  \n",
       "\n",
       "[66265 rows x 41 columns]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d105a254-c46a-4b38-ae59-0be989a275eb",
   "metadata": {},
   "source": [
    "## 7. Finding and dropping unnecessary columns\n",
    "Several columns are unnecessary for the analysis and will be dropped.\n",
    "Steps:\n",
    "* View all column names\n",
    "* Inspect some of the columns to see if they are needed or not\n",
    "* Drop the columns identified as unnecessary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41ecab9a-3e52-4e43-9230-6665dd5ad503",
   "metadata": {},
   "source": [
    "### 7.1. Viewing all column names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "5d9186a9-f602-420a-88a3-8ab764e2b855",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['reply_settings', 'possibly_sensitive', 'id', 'text', 'conversation_id',\n",
       "       'author_id', 'edit_history_tweet_ids', 'created_at', 'lang',\n",
       "       'entities.urls', 'geo.place_id', 'edit_controls.edits_remaining',\n",
       "       'edit_controls.is_edit_eligible', 'edit_controls.editable_until',\n",
       "       'public_metrics.retweet_count', 'public_metrics.reply_count',\n",
       "       'public_metrics.like_count', 'public_metrics.quote_count',\n",
       "       'public_metrics.impression_count', 'geo.coordinates.type',\n",
       "       'geo.coordinates.coordinates', 'in_reply_to_user_id',\n",
       "       'referenced_tweets', 'entities.hashtags', 'attachments.media_keys',\n",
       "       'entities.mentions', 'context_annotations', 'attachments.poll_ids',\n",
       "       'entities.cashtags', 'week', 'month', 'year', 'year-week', 'year-month',\n",
       "       'Language', 'date', 'entities.annotations', 'withheld.copyright',\n",
       "       'withheld.country_codes', 'withheld.scope', 'inflow'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing all column names\n",
    "\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbb440ac-9efd-4251-8c44-17886df8fd90",
   "metadata": {},
   "source": [
    "### 7.2. Inspecting column contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "b91f83a2-c693-4f1b-9842-ebde21ddc0aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    everyone\n",
      "1    everyone\n",
      "2    everyone\n",
      "3    everyone\n",
      "4    everyone\n",
      "Name: reply_settings, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'reply_settings' (only looking at first 5 entries)\n",
    "\n",
    "print(df['reply_settings'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "e7827e76-bed0-4f55-ac91-2cfb7fe5f93d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    5.0\n",
      "1    5.0\n",
      "2    5.0\n",
      "3    5.0\n",
      "4    5.0\n",
      "Name: edit_controls.edits_remaining, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'edit_controls.edits_remaining' (only looking at first 5 entries)\n",
    "\n",
    "print(df['edit_controls.edits_remaining'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "5c58691f-356f-4932-94f3-b673c34b16f0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                                                   NaN\n",
      "1                                                   NaN\n",
      "2                                                   NaN\n",
      "3      [{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]\n",
      "4     [{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...\n",
      "5                                                   NaN\n",
      "6     [{'start': 0, 'end': 7, 'tag': 'Boehmi'}, {'st...\n",
      "7     [{'start': 0, 'end': 8, 'tag': 'Bamberg'}, {'s...\n",
      "8     [{'start': 64, 'end': 75, 'tag': 'Mittelmeer'}...\n",
      "9                                                   NaN\n",
      "10                                                  NaN\n",
      "11                                                  NaN\n",
      "12                                                  NaN\n",
      "13                                                  NaN\n",
      "14    [{'start': 0, 'end': 10, 'tag': 'Davutoğlu'}, ...\n",
      "15    [{'start': 66, 'end': 73, 'tag': 'Bochum'}, {'...\n",
      "16    [{'start': 16, 'end': 27, 'tag': 'Länderzeit'}...\n",
      "17    [{'start': 0, 'end': 11, 'tag': 'EHFreiburg'},...\n",
      "18                                                  NaN\n",
      "19                                                  NaN\n",
      "Name: entities.hashtags, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'entities.hashtags' (only looking at first 20 entries)\n",
    "\n",
    "print(df['entities.hashtags'][0:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "1277fe7b-90e5-4c46-ad62-037958ddf509",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                                                   NaN\n",
      "1                                                   NaN\n",
      "2                                                   NaN\n",
      "3                                                   NaN\n",
      "4                                                   NaN\n",
      "5     [{'start': 0, 'end': 8, 'username': 'hataibu',...\n",
      "6                                                   NaN\n",
      "7                                                   NaN\n",
      "8                                                   NaN\n",
      "9                                                   NaN\n",
      "10                                                  NaN\n",
      "11    [{'start': 96, 'end': 101, 'username': 'welt',...\n",
      "12    [{'start': 96, 'end': 101, 'username': 'welt',...\n",
      "13                                                  NaN\n",
      "14                                                  NaN\n",
      "15    [{'start': 0, 'end': 13, 'username': 'AndreasL...\n",
      "16    [{'start': 10, 'end': 14, 'username': 'DLF', '...\n",
      "17                                                  NaN\n",
      "18                                                  NaN\n",
      "19                                                  NaN\n",
      "Name: entities.mentions, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'entities.mentions' (only looking at first 20 entries)\n",
    "\n",
    "print(df['entities.mentions'][0:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "55df6cf1-d518-425d-99ee-3511851b4ff8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     [{'start': 92, 'end': 115, 'url': 'https://t.c...\n",
      "1                                                   NaN\n",
      "2                                                   NaN\n",
      "3     [{'start': 83, 'end': 106, 'url': 'https://t.c...\n",
      "4     [{'start': 92, 'end': 115, 'url': 'https://t.c...\n",
      "5                                                   NaN\n",
      "6                                                   NaN\n",
      "7     [{'start': 70, 'end': 93, 'url': 'https://t.co...\n",
      "8     [{'start': 113, 'end': 136, 'url': 'https://t....\n",
      "9                                                   NaN\n",
      "10    [{'start': 76, 'end': 99, 'url': 'https://t.co...\n",
      "11    [{'start': 68, 'end': 91, 'url': 'https://t.co...\n",
      "12    [{'start': 68, 'end': 91, 'url': 'https://t.co...\n",
      "13    [{'start': 110, 'end': 133, 'url': 'https://t....\n",
      "14                                                  NaN\n",
      "15    [{'start': 110, 'end': 133, 'url': 'https://t....\n",
      "16                                                  NaN\n",
      "17    [{'start': 82, 'end': 105, 'url': 'https://t.c...\n",
      "18    [{'start': 74, 'end': 97, 'url': 'https://t.co...\n",
      "19    [{'start': 60, 'end': 83, 'url': 'https://t.co...\n",
      "Name: entities.urls, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'entities.urls' (only looking at first 20 entries)\n",
    "\n",
    "print(df['entities.urls'][0:20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "8186c4c7-2e4f-44f3-a16a-a17d3d0d141d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0      NaN\n",
      "1    Point\n",
      "2      NaN\n",
      "3      NaN\n",
      "4      NaN\n",
      "Name: geo.coordinates.type, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'geo.coordinates.type' (only looking at first 5 entries)\n",
    "\n",
    "print(df['geo.coordinates.type'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "a8bc2cee-d9bb-4af7-bbce-f11b5a6176dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0                         NaN\n",
      "1                         NaN\n",
      "2                         NaN\n",
      "3    ['7_722852972233912321']\n",
      "4    ['3_722824009939886080']\n",
      "Name: attachments.media_keys, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'attachments.media_keys' (only looking at first 5 entries)\n",
    "\n",
    "print(df['attachments.media_keys'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "7f1eb560-06d1-4917-ba0b-d1c99ad0ae20",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    NaN\n",
      "1    NaN\n",
      "2    NaN\n",
      "3    NaN\n",
      "4    NaN\n",
      "Name: context_annotations, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'context_annotations' (only looking at first 5 entries)\n",
    "\n",
    "print(df['context_annotations'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "f847176a-140d-44b8-89b8-a1f2fc61d97a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    NaN\n",
      "1    NaN\n",
      "2    NaN\n",
      "3    NaN\n",
      "4    NaN\n",
      "Name: attachments.poll_ids, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'attachments.poll_ids' (only looking at first 5 entries)\n",
    "\n",
    "print(df['attachments.poll_ids'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "c7887d3d-5ce4-4322-a931-a9f64eb796dd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    NaN\n",
      "1    NaN\n",
      "2    NaN\n",
      "3    NaN\n",
      "4    NaN\n",
      "Name: entities.cashtags, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'entities.cashtags' (only looking at first 5 entries)\n",
    "\n",
    "print(df['entities.cashtags'][0:5])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "4689cdc9-018a-404b-a5b4-c401d81711c6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    NaN\n",
      "1    NaN\n",
      "2    NaN\n",
      "3    NaN\n",
      "4    NaN\n",
      "Name: entities.annotations, dtype: object\n"
     ]
    }
   ],
   "source": [
    "# inspecting column 'entities.annotations' (only looking at first 5 entries)\n",
    "\n",
    "print(df['entities.annotations'][0:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7edbcb54-5bbf-4c90-aeb3-7988025f9add",
   "metadata": {},
   "source": [
    "### 7.2. Dropping columns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc4b130e-8959-4b21-94e6-d55693bf2e5c",
   "metadata": {},
   "source": [
    "Unnecessary columns include:\n",
    "* referenced_tweets\n",
    "* id\n",
    "* conversation_id\n",
    "* edit_history_tweet_ids\n",
    "* possibly_sensitive\n",
    "* reply_settings\n",
    "* edit_controls.edits_remaining\n",
    "* edit_controls.is_edit_eligible\n",
    "* edit_controls.editable_until\n",
    "* entities.mentions\n",
    "* entities.urls\n",
    "* geo.coordinates.type\n",
    "* attachments.media_keys\n",
    "* context_annotations\n",
    "* attachments.poll_ids\n",
    "* entities.cashtags\n",
    "* entities.annotations\n",
    "* withheld.copyright\n",
    "* withheld.country_codes\n",
    "* withheld.scope\n",
    "\n",
    "**NOTE**: Some remaining columns may not end up being necessary but will be left in the dataframe just in case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "9fb7cd83-f29f-44cd-82fe-94a1750b3acd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dropping the columns\n",
    "\n",
    "df.drop(['referenced_tweets', 'id', 'conversation_id', 'edit_history_tweet_ids', 'possibly_sensitive', 'reply_settings', 'edit_controls.edits_remaining', 'edit_controls.is_edit_eligible', 'edit_controls.editable_until', 'entities.mentions', 'entities.urls', 'geo.coordinates.type', 'attachments.media_keys', 'context_annotations', 'attachments.poll_ids', 'entities.cashtags', 'entities.annotations', 'withheld.copyright', 'withheld.country_codes', 'withheld.scope'], axis=1, inplace=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "b0544895-ba94-4210-a235-0a1339b107bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['text', 'author_id', 'created_at', 'lang', 'geo.place_id',\n",
       "       'public_metrics.retweet_count', 'public_metrics.reply_count',\n",
       "       'public_metrics.like_count', 'public_metrics.quote_count',\n",
       "       'public_metrics.impression_count', 'geo.coordinates.coordinates',\n",
       "       'in_reply_to_user_id', 'entities.hashtags', 'week', 'month', 'year',\n",
       "       'year-week', 'year-month', 'Language', 'date', 'inflow'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing all column names\n",
    "\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "6202ebee-b54f-4874-9a91-6325540c5b9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>author_id</th>\n",
       "      <th>created_at</th>\n",
       "      <th>lang</th>\n",
       "      <th>geo.place_id</th>\n",
       "      <th>public_metrics.retweet_count</th>\n",
       "      <th>public_metrics.reply_count</th>\n",
       "      <th>public_metrics.like_count</th>\n",
       "      <th>public_metrics.quote_count</th>\n",
       "      <th>public_metrics.impression_count</th>\n",
       "      <th>...</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>entities.hashtags</th>\n",
       "      <th>week</th>\n",
       "      <th>month</th>\n",
       "      <th>year</th>\n",
       "      <th>year-week</th>\n",
       "      <th>year-month</th>\n",
       "      <th>Language</th>\n",
       "      <th>date</th>\n",
       "      <th>inflow</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...</td>\n",
       "      <td>4122038069</td>\n",
       "      <td>2016-04-20 22:55:08+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>06d9a7c249c59bcd</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Habe schon lang nicht gehört, daß Flüchtling G...</td>\n",
       "      <td>1179543852</td>\n",
       "      <td>2016-04-20 21:27:37+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e99b714fe65be4fb</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>\"Es kommen kaum noch Flüchtlinge nach Griechen...</td>\n",
       "      <td>224607633</td>\n",
       "      <td>2016-04-20 21:18:58+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>3078869807f9dd36</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Unsere 1. Kochshow für #Flüchtlinge. Super spi...</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>2016-04-20 18:25:11+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>8abc99434d4f5d28</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2480764313</td>\n",
       "      <td>[{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>500 tote #Flüchtlinge im #Mittelmeer – Tragödi...</td>\n",
       "      <td>606265303</td>\n",
       "      <td>2016-04-20 16:27:28+00:00</td>\n",
       "      <td>de</td>\n",
       "      <td>e11a8b8e3771f9fa</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016.0</td>\n",
       "      <td>2016-16</td>\n",
       "      <td>2016-04</td>\n",
       "      <td>German</td>\n",
       "      <td>2016-04-20 00:00:00+00:00</td>\n",
       "      <td>Syrians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66260</th>\n",
       "      <td>For day 1 of week 2, @AnnaMariaKonsta discusse...</td>\n",
       "      <td>110402493</td>\n",
       "      <td>2021-06-28 14:43:31+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>fcbb3c6e0a7eba22</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>110402493.0</td>\n",
       "      <td>[{'start': 95, 'end': 108, 'tag': 'SocialRight...</td>\n",
       "      <td>26.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-28 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66261</th>\n",
       "      <td>@ariadneconill Europe is racist, but in a diff...</td>\n",
       "      <td>2521808908</td>\n",
       "      <td>2021-06-27 13:03:29+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>e385d4d639c6a423</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>15869538.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66262</th>\n",
       "      <td>A labour of love, inspired by Middle-earth.\\n\\...</td>\n",
       "      <td>563381751</td>\n",
       "      <td>2021-06-27 08:37:21+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>257640324f249a73</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>563381751.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-26</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-27 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66263</th>\n",
       "      <td>@simongerman600 I must have missed the great f...</td>\n",
       "      <td>2591892350</td>\n",
       "      <td>2021-06-26 08:03:22+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>000b71538f35fe46</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>359188534.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-26 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66264</th>\n",
       "      <td>#Wuerzburg today an African immigrant or refug...</td>\n",
       "      <td>134401670</td>\n",
       "      <td>2021-06-25 21:02:17+00:00</td>\n",
       "      <td>en</td>\n",
       "      <td>08f94128283f7bc6</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]</td>\n",
       "      <td>25.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2021.0</td>\n",
       "      <td>2021-25</td>\n",
       "      <td>2021-06</td>\n",
       "      <td>English</td>\n",
       "      <td>2021-06-25 00:00:00+00:00</td>\n",
       "      <td>Ukrainians</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>66265 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    text   author_id  \\\n",
       "0      Syrisch-orthodoxer Bischof: \"Was im Nahen Oste...  4122038069   \n",
       "1      Habe schon lang nicht gehört, daß Flüchtling G...  1179543852   \n",
       "2      \"Es kommen kaum noch Flüchtlinge nach Griechen...   224607633   \n",
       "3      Unsere 1. Kochshow für #Flüchtlinge. Super spi...  2480764313   \n",
       "4      500 tote #Flüchtlinge im #Mittelmeer – Tragödi...   606265303   \n",
       "...                                                  ...         ...   \n",
       "66260  For day 1 of week 2, @AnnaMariaKonsta discusse...   110402493   \n",
       "66261  @ariadneconill Europe is racist, but in a diff...  2521808908   \n",
       "66262  A labour of love, inspired by Middle-earth.\\n\\...   563381751   \n",
       "66263  @simongerman600 I must have missed the great f...  2591892350   \n",
       "66264  #Wuerzburg today an African immigrant or refug...   134401670   \n",
       "\n",
       "                     created_at lang      geo.place_id  \\\n",
       "0     2016-04-20 22:55:08+00:00   de  06d9a7c249c59bcd   \n",
       "1     2016-04-20 21:27:37+00:00   de  e99b714fe65be4fb   \n",
       "2     2016-04-20 21:18:58+00:00   de  3078869807f9dd36   \n",
       "3     2016-04-20 18:25:11+00:00   de  8abc99434d4f5d28   \n",
       "4     2016-04-20 16:27:28+00:00   de  e11a8b8e3771f9fa   \n",
       "...                         ...  ...               ...   \n",
       "66260 2021-06-28 14:43:31+00:00   en  fcbb3c6e0a7eba22   \n",
       "66261 2021-06-27 13:03:29+00:00   en  e385d4d639c6a423   \n",
       "66262 2021-06-27 08:37:21+00:00   en  257640324f249a73   \n",
       "66263 2021-06-26 08:03:22+00:00   en  000b71538f35fe46   \n",
       "66264 2021-06-25 21:02:17+00:00   en  08f94128283f7bc6   \n",
       "\n",
       "       public_metrics.retweet_count  public_metrics.reply_count  \\\n",
       "0                               0.0                         0.0   \n",
       "1                               0.0                         0.0   \n",
       "2                               0.0                         0.0   \n",
       "3                               0.0                         0.0   \n",
       "4                               0.0                         1.0   \n",
       "...                             ...                         ...   \n",
       "66260                           0.0                         1.0   \n",
       "66261                           0.0                         1.0   \n",
       "66262                           0.0                         1.0   \n",
       "66263                           0.0                         0.0   \n",
       "66264                           8.0                         6.0   \n",
       "\n",
       "      public_metrics.like_count public_metrics.quote_count  \\\n",
       "0                             0                          0   \n",
       "1                             0                          0   \n",
       "2                             0                          0   \n",
       "3                             4                          0   \n",
       "4                             0                          0   \n",
       "...                         ...                        ...   \n",
       "66260                       2.0                        0.0   \n",
       "66261                       6.0                        0.0   \n",
       "66262                      17.0                        0.0   \n",
       "66263                       1.0                        0.0   \n",
       "66264                      38.0                        6.0   \n",
       "\n",
       "      public_metrics.impression_count  ... in_reply_to_user_id  \\\n",
       "0                                 0.0  ...                 NaN   \n",
       "1                                 0.0  ...                 NaN   \n",
       "2                                 0.0  ...                 NaN   \n",
       "3                                 0.0  ...          2480764313   \n",
       "4                                 0.0  ...                 NaN   \n",
       "...                               ...  ...                 ...   \n",
       "66260                               0  ...         110402493.0   \n",
       "66261                               0  ...          15869538.0   \n",
       "66262                               0  ...         563381751.0   \n",
       "66263                               0  ...         359188534.0   \n",
       "66264                               0  ...                 NaN   \n",
       "\n",
       "                                       entities.hashtags  week  month    year  \\\n",
       "0                                                    NaN  16.0    4.0  2016.0   \n",
       "1                                                    NaN  16.0    4.0  2016.0   \n",
       "2                                                    NaN  16.0    4.0  2016.0   \n",
       "3       [{'start': 23, 'end': 35, 'tag': 'Flüchtlinge'}]  16.0    4.0  2016.0   \n",
       "4      [{'start': 9, 'end': 21, 'tag': 'Flüchtlinge'}...  16.0    4.0  2016.0   \n",
       "...                                                  ...   ...    ...     ...   \n",
       "66260  [{'start': 95, 'end': 108, 'tag': 'SocialRight...  26.0    6.0  2021.0   \n",
       "66261                                                NaN  25.0    6.0  2021.0   \n",
       "66262                                                NaN  25.0    6.0  2021.0   \n",
       "66263                                                NaN  25.0    6.0  2021.0   \n",
       "66264      [{'start': 0, 'end': 10, 'tag': 'Wuerzburg'}]  25.0    6.0  2021.0   \n",
       "\n",
       "       year-week year-month Language                      date      inflow  \n",
       "0        2016-16    2016-04   German 2016-04-20 00:00:00+00:00     Syrians  \n",
       "1        2016-16    2016-04   German 2016-04-20 00:00:00+00:00     Syrians  \n",
       "2        2016-16    2016-04   German 2016-04-20 00:00:00+00:00     Syrians  \n",
       "3        2016-16    2016-04   German 2016-04-20 00:00:00+00:00     Syrians  \n",
       "4        2016-16    2016-04   German 2016-04-20 00:00:00+00:00     Syrians  \n",
       "...          ...        ...      ...                       ...         ...  \n",
       "66260    2021-26    2021-06  English 2021-06-28 00:00:00+00:00  Ukrainians  \n",
       "66261    2021-26    2021-06  English 2021-06-27 00:00:00+00:00  Ukrainians  \n",
       "66262    2021-26    2021-06  English 2021-06-27 00:00:00+00:00  Ukrainians  \n",
       "66263    2021-25    2021-06  English 2021-06-26 00:00:00+00:00  Ukrainians  \n",
       "66264    2021-25    2021-06  English 2021-06-25 00:00:00+00:00  Ukrainians  \n",
       "\n",
       "[66265 rows x 21 columns]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# viewing the dataframe\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde4a29f-fc95-4e95-ac03-9aeae780ae2f",
   "metadata": {},
   "source": [
    "## 8. Saving the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "b0c3c4f6-048c-4723-8a1b-946e947e0f40",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(CASS_thesis / \"02_Pre-processed_limited_merged.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f2c14a1-3aaf-41e6-846f-dc748962e165",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}