notebooks/6_WebScraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Web Scraping and APIs\n",
    "\n",
    "In this notebook, we learn how to scrape data from the Web and get an idea of what Applicaiton Programming Interfaces are (APIs)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web Scraping\n",
    "\n",
    "**Web Scraping** is a technique for the extraction of information from websites by transforming unstructured data (HTML pages) into structured data (databases or spreadsheets). \n",
    "\n",
    "Even if scraping can be manually performed by a user, it is usually implemented using a **web crawler** (i.e., it is usually implemented as an automatic process). For larger scale scraping see, e.g., [Scrapy](https://scrapy.org)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The process is an alternative to using already available **API**s (Application Programming Interface), such as those provided by all the major platforms, like *Facebook*, *Google* and *Twitter*. **More below.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basics of HTML\n",
    "\n",
    "The **HyperText Markup Language (HTML)** is the standard **descriptive markup** language for web pages.\n",
    "\n",
    "\n",
    "- **Markup** language: a human-readable, explicit system for annotating the content of a document. Markdown is another markup language.\n",
    "\n",
    "\n",
    "- **Descriptive** markup languages (e.g. HTML, XML) are used to annotate the structure or the contents of a document, as opposed to **procedural** markup languages (e.g. TEX, Postscript), whose main goal is to describe how a document should be processed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTML provides a means to annotate the <strong>structural</strong> elements of documents like (different kinds of) headings, paragraphs, lists, links, images, quotes, tables and so forth. Similarly, even if with fewer options, does Markdown (which we are <em>using</em> *here*, check the code!).\n",
    "\n",
    "HTML tags **do not mark the logical structure** of a document, but only its format (e.g. *this is a table*, *this is a h3-type heading*...). It is up to the browser to then use HTML (plus other information, such as *Cascading Style Sheets*), to render a webpage appropriately."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "HTML markup relies on a **fixed inventory of tags**, written by using angle brackets. Some tags, e.g. `<p>...</p>`, surround the marked text, and may include subelements. Other tags, e.g. `<br>` or `<img>` introduce content directly.\n",
    "\n",
    "The following is an example of a web page:\n",
    "\n",
    "```html\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "  <head>\n",
    "    <title>The Adventures of Pinocchio</title>\n",
    "  </head>\n",
    "  <body>\n",
    "    <h2>Carlo Collodi</h2>\n",
    "    <h1>The Adventures of Pinocchio</h1>\n",
    "    <hr>\n",
    "    <h4>CHAPTER 1</h4>\n",
    "    <br>\n",
    "    <p><i>How it happened that Mastro Cherry, carpenter, found a piece of wood that wept and laughed like a child</i></p>\n",
    "    <br>\n",
    "    <p>Centuries ago there lived--</p>\n",
    "    <p>\"A king!\" my little readers will say immediately.</p>\n",
    "  </body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scraping Web Pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">The following notes are roughly based on the **Chapters 1-3** of: Mitchell, R. (2015). [Web Scraping with Python](http://shop.oreilly.com/product/0636920034391.do), O'Reilly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Modules and Packages Required for Web Scraping\n",
    "\n",
    "**BeautifulSoup**: this library defines [classes and functions](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) to pull data (e.g. table, lists, paragraphs) out of HTML and XML files. It provides idiomatic ways of navigating, searching, and modifying the parse tree.\n",
    "\n",
    "\n",
    "**lxml**: to function, BeautifulSoup relies on external HTML-XML parsers. Many options are available, among which the html5lib's and the Python's built-in parsers. We'll rely on the [lxml](http://lxml.de/)'s parser, due to its high performance, reliability and flexibility.\n",
    "\n",
    "\n",
    "**Urllib**: BeautifulSoup does not fetch the web page for us. To do this, we'll rely on the [Urllib](https://docs.python.org/3.7/library/urllib.html#module-urllib) module available in the Python Standard Library, that implements classes and functions which help in opening URLs (authentication, redirections, cookies and so on). We will see another option, **requests**, below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib\n",
    "from urllib.request import urlopen\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Retrieve and Parse an HTML page\n",
    "\n",
    "`urllib.request.urlopen()` allows us to retrieve our target HTML page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = urlopen(\"http://www.pythonscraping.com/pages/page1.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if the page doesn't exist?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HTTP Error 404: Not Found\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    html = urlopen(\"http://www.pythonscraping.com/pages/page.html\")\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, let's handle this properly..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    html = urlopen(\"http://www.pythonscraping.com/pages/page.html\")\n",
    "except urllib.request.URLError as e:\n",
    "    pass # code your plan B here\n",
    "except urllib.request.URLError as e:\n",
    "    raise # raise any other exception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use `BeautifulSoup()` in conjunction with `lxml` to parse out `html` page and store it in the Beautiful Soup format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you might need to to the following:\n",
    "#!pip install lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = urlopen(\"http://www.pythonscraping.com/pages/page1.html\")\n",
    "soup_page1 = BeautifulSoup(html, \"lxml\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Let's scrape another couple of pages we'll need in our examples\n",
    "soup_page3 = BeautifulSoup(urlopen(\"http://www.pythonscraping.com/pages/page3.html\"), \"lxml\")\n",
    "soup_wap = BeautifulSoup(urlopen(\"http://www.pythonscraping.com/pages/warandpeace.html\"), \"lxml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Let's look at the nested structure of the page\n",
    "\n",
    "The `prettify()` method allows us to have a look at the structure of the HTML page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      "<head>\n",
      "<title>A Useful Page</title>\n",
      "</head>\n",
      "<body>\n",
      "<h1>An Interesting Title</h1>\n",
      "<div>\n",
      "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n",
      "</div>\n",
      "</body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(soup_page1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <head>\n",
      "  <title>\n",
      "   A Useful Page\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <h1>\n",
      "   An Interesting Title\n",
      "  </h1>\n",
      "  <div>\n",
      "   Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(soup_page1.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Let's play with a HTML tag\n",
    "\n",
    "The notation `soup.<tag>` allows us to retrieve the content marked by a tag (opening and closing tags included)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<div>\n",
       "Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\n",
       "</div>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# note that the first \"<div>\" tag is nested two layers deep (html → body → div).\n",
    "soup_page1.div"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the text is the only thing you're interested into, well, the `soup.<tag>.string` method comes in handy:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\nLorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.\\n'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_page1.div.string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTML markup generated by Beautiful Soup can be modified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's change the content of our div\n",
    "soup_page1.div.string = \"this content has been changed\"\n",
    "# let's change the name of the tag\n",
    "soup_page1.div.name = \"new_div\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <head>\n",
      "  <title>\n",
      "   A Useful Page\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <h1>\n",
      "   An Interesting Title\n",
      "  </h1>\n",
      "  <new_div>\n",
      "   this content has been changed\n",
      "  </new_div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(soup_page1.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In its simplest use, the `find()` method is an alternative to the `soup.<tag>` notation..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<new_div>this content has been changed</new_div>"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_page1.find(\"new_div\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<new_div>this content has been changed</new_div>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_page1.new_div"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...but this function allows for the searching of nodes by exploiting cues in the markup, such as a given **class attribute** value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <head>\n",
      "  <style>\n",
      "   .green{\n",
      "\tcolor:#55ff55;\n",
      "}\n",
      ".red{\n",
      "\tcolor:#ff5555;\n",
      "}\n",
      "#text{\n",
      "\twidth:50%;\n",
      "}\n",
      "  </style>\n",
      " </head>\n",
      " <body>\n",
      "  <h1>\n",
      "   War and Peace\n",
      "  </h1>\n",
      "  <h2>\n",
      "   Chapter 1\n",
      "  </h2>\n",
      "  <div id=\"text\">\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Well, Prince, so Genoa and Lucca are now just family estates of the\n",
      "Buonapartes. But I warn you, if you don't tell me that this means war,\n",
      "if you still try to defend the infamies and horrors perpetrated by\n",
      "that Antichrist- I really believe he is Antichrist- I will have\n",
      "nothing more to do with you and you are no longer my friend, no longer\n",
      "my 'faithful slave,' as you call yourself! But how do you do? I see\n",
      "I have frightened you- sit down and tell me all the news.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   It was in July, 1805, and the speaker was the well-known\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna Scherer\n",
      "   </span>\n",
      "   , maid of honor and favorite of the\n",
      "   <span class=\"green\">\n",
      "    Empress Marya\n",
      "Fedorovna\n",
      "   </span>\n",
      "   . With these words she greeted\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili Kuragin\n",
      "   </span>\n",
      "   , a man\n",
      "of high rank and importance, who was the first to arrive at her\n",
      "reception.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   had had a cough for some days. She was, as\n",
      "she said, suffering from la grippe; grippe being then a new word in\n",
      "   <span class=\"green\">\n",
      "    St. Petersburg\n",
      "   </span>\n",
      "   , used only by the elite.\n",
      "   <p>\n",
      "   </p>\n",
      "   All her invitations without exception, written in French, and\n",
      "delivered by a scarlet-liveried footman that morning, ran as follows:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    If you have nothing better to do, Count [or Prince], and if the\n",
      "prospect of spending an evening with a poor invalid is not too\n",
      "terrible, I shall be very charmed to see you tonight between 7 and 10-\n",
      "Annette Scherer.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Heavens! what a virulent attack!\n",
      "   </span>\n",
      "   \" replied\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   , not in the\n",
      "least disconcerted by this reception. He had just entered, wearing\n",
      "an embroidered court uniform, knee breeches, and shoes, and had\n",
      "stars on his breast and a serene expression on his flat face. He spoke\n",
      "in that refined French in which our grandfathers not only spoke but\n",
      "thought, and with the gentle, patronizing intonation natural to a\n",
      "man of importance who had grown old in society and at court. He went\n",
      "up to\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   , kissed her hand, presenting to her his bald,\n",
      "scented, and shining head, and complacently seated himself on the\n",
      "sofa.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    First of all, dear friend, tell me how you are. Set your friend's\n",
      "mind at rest,\n",
      "   </span>\n",
      "   \" said he without altering his tone, beneath the\n",
      "politeness and affected sympathy of which indifference and even\n",
      "irony could be discerned.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Can one be well while suffering morally? Can one be calm in times\n",
      "like these if one has any feeling?\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    You are\n",
      "staying the whole evening, I hope?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    And the fete at the English ambassador's? Today is Wednesday. I\n",
      "must put in an appearance there,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    My daughter is\n",
      "coming for me to take me there.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I thought today's fete had been canceled. I confess all these\n",
      "festivities and fireworks are becoming wearisome.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    If they had known that you wished it, the entertainment would\n",
      "have been put off,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   , who, like a wound-up clock, by\n",
      "force of habit said things he did not even wish to be believed.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Don't tease! Well, and what has been decided about Novosiltsev's\n",
      "dispatch? You know everything.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    What can one say about it?\n",
      "   </span>\n",
      "   \" replied\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   in a cold,\n",
      "listless tone. \"\n",
      "   <span class=\"red\">\n",
      "    What has been decided? They have decided that\n",
      "Buonaparte has burnt his boats, and I believe that we are ready to\n",
      "burn ours.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili\n",
      "   </span>\n",
      "   always spoke languidly, like an actor repeating a\n",
      "stale part.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   Scherer on the contrary, despite her forty\n",
      "years, overflowed with animation and impulsiveness. To be an\n",
      "enthusiast had become her social vocation and, sometimes even when she\n",
      "did not feel like it, she became enthusiastic in order not to\n",
      "disappoint the expectations of those who knew her. The subdued smile\n",
      "which, though it did not suit her faded features, always played\n",
      "round her lips expressed, as in a spoiled child, a continual\n",
      "consciousness of her charming defect, which she neither wished, nor\n",
      "could, nor considered it necessary, to correct.\n",
      "   <p>\n",
      "   </p>\n",
      "   In the midst of a conversation on political matters\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   burst out:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Oh, don't speak to me of Austria. Perhaps I don't understand\n",
      "things, but Austria never has wished, and does not wish, for war.\n",
      "She is betraying us! Russia alone must save Europe. Our gracious\n",
      "sovereign recognizes his high vocation and will be true to it. That is\n",
      "the one thing I have faith in! Our good and wonderful sovereign has to\n",
      "perform the noblest role on earth, and he is so virtuous and noble\n",
      "that God will not forsake him. He will fulfill his vocation and\n",
      "crush the hydra of revolution, which has become more terrible than\n",
      "ever in the person of this murderer and villain! We alone must\n",
      "avenge the blood of the just one.... Whom, I ask you, can we rely\n",
      "on?... England with her commercial spirit will not and cannot\n",
      "understand the Emperor Alexander's loftiness of soul. She has\n",
      "refused to evacuate Malta. She wanted to find, and still seeks, some\n",
      "secret motive in our actions. What answer did Novosiltsev get? None.\n",
      "The English have not understood and cannot understand the\n",
      "self-abnegation of our Emperor who wants nothing for himself, but only\n",
      "desires the good of mankind. And what have they promised? Nothing! And\n",
      "what little they have promised they will not perform! Prussia has\n",
      "always declared that Buonaparte is invincible, and that all Europe\n",
      "is powerless before him.... And I don't believe a word that Hardenburg\n",
      "says, or Haugwitz either. This famous Prussian neutrality is just a\n",
      "trap. I have faith only in God and the lofty destiny of our adored\n",
      "monarch. He will save Europe!\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   She suddenly paused, smiling at her own impetuosity.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I think,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   with a smile, \"\n",
      "   <span class=\"red\">\n",
      "    that if you had been\n",
      "sent instead of our dear\n",
      "    <span class=\"green\">\n",
      "     Wintzingerode\n",
      "    </span>\n",
      "    you would have captured the\n",
      "    <span class=\"green\">\n",
      "     King of Prussia\n",
      "    </span>\n",
      "    's consent by assault. You are so eloquent. Will you\n",
      "give me a cup of tea?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    In a moment. A propos,\n",
      "   </span>\n",
      "   \" she added, becoming calm again, \"\n",
      "   <span class=\"red\">\n",
      "    I am\n",
      "expecting two very interesting men tonight,\n",
      "    <span class=\"green\">\n",
      "     le Vicomte de Mortemart\n",
      "    </span>\n",
      "    ,\n",
      "who is connected with the\n",
      "    <span class=\"green\">\n",
      "     Montmorencys\n",
      "    </span>\n",
      "    through the\n",
      "    <span class=\"green\">\n",
      "     Rohans\n",
      "    </span>\n",
      "    , one of\n",
      "the best French families. He is one of the genuine emigres, the good\n",
      "ones. And also the\n",
      "    <span class=\"green\">\n",
      "     Abbe Morio\n",
      "    </span>\n",
      "    . Do you know that profound thinker? He\n",
      "has been received by\n",
      "    <span class=\"green\">\n",
      "     the Emperor\n",
      "    </span>\n",
      "    . Had you heard?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I shall be delighted to meet them,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    But tell me,\n",
      "   </span>\n",
      "   \"\n",
      "he added with studied carelessness as if it had only just occurred\n",
      "to him, though the question he was about to ask was the chief motive\n",
      "of his visit, \"\n",
      "   <span class=\"red\">\n",
      "    is it true that the Dowager Empress wants Baron Funke\n",
      "to be appointed first secretary at Vienna? The baron by all accounts\n",
      "is a poor creature.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili\n",
      "   </span>\n",
      "   wished to obtain this post for his son, but others\n",
      "were trying through the\n",
      "   <span class=\"green\">\n",
      "    Dowager Empress Marya Fedorovna\n",
      "   </span>\n",
      "   to secure it\n",
      "for\n",
      "   <span class=\"green\">\n",
      "    the baron\n",
      "   </span>\n",
      "   .\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   almost closed her eyes to indicate that neither she\n",
      "nor anyone else had a right to criticize what\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   desired or\n",
      "was pleased with.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Baron Funke has been recommended to the Dowager Empress by her\n",
      "sister,\n",
      "   </span>\n",
      "   \" was all she said, in a dry and mournful tone.\n",
      "   <p>\n",
      "   </p>\n",
      "   As she named\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   ,\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna's\n",
      "   </span>\n",
      "   face suddenly assumed an\n",
      "expression of profound and sincere devotion and respect mingled with\n",
      "sadness, and this occurred every time she mentioned her illustrious\n",
      "patroness. She added that\n",
      "   <span class=\"green\">\n",
      "    Her Majesty\n",
      "   </span>\n",
      "   had deigned to show\n",
      "   <span class=\"green\">\n",
      "    Baron\n",
      "Funke\n",
      "   </span>\n",
      "   , and again her face clouded over with sadness.\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   was silent and looked indifferent. But, with the\n",
      "womanly and courtierlike quickness and tact habitual to her,\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna\n",
      "   </span>\n",
      "   wished both to rebuke him (for daring to speak he had done of\n",
      "a man recommended to\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   ) and at the same time to console him,\n",
      "so she said:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Now about your family. Do you know that since your daughter came\n",
      "out everyone has been enraptured by her? They say she is amazingly\n",
      "beautiful.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   bowed to signify his respect and gratitude.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I often think,\n",
      "   </span>\n",
      "   \" she continued after a short pause, drawing nearer\n",
      "to the prince and smiling amiably at him as if to show that\n",
      "political and social topics were ended and the time had come for\n",
      "intimate conversation- \"\n",
      "   <span class=\"red\">\n",
      "    I often think how unfairly sometimes the\n",
      "joys of life are distributed. Why has fate given you two such splendid\n",
      "children? I don't speak of\n",
      "    <span class=\"green\">\n",
      "     Anatole\n",
      "    </span>\n",
      "    , your youngest. I don't like\n",
      "him,\n",
      "   </span>\n",
      "   \" she added in a tone admitting of no rejoinder and raising her\n",
      "eyebrows. \"\n",
      "   <span class=\"red\">\n",
      "    Two such charming children. And really you appreciate\n",
      "them less than anyone, and so you don't deserve to have them.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   And she smiled her ecstatic smile.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I can't help it,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    Lavater would have said I\n",
      "lack the bump of paternity.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Don't joke; I mean to have a serious talk with you. Do you know I\n",
      "am dissatisfied with your younger son? Between ourselves\n",
      "   </span>\n",
      "   \" (and her\n",
      "face assumed its melancholy expression), \"\n",
      "   <span class=\"red\">\n",
      "    he was mentioned at Her\n",
      "Majesty's and you were pitied....\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   answered nothing, but she looked at him significantly,\n",
      "awaiting a reply. He frowned.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    What would you have me do?\n",
      "   </span>\n",
      "   \" he said at last. \"\n",
      "   <span class=\"red\">\n",
      "    You know I did all\n",
      "a father could for their education, and they have both turned out\n",
      "fools. Hippolyte is at least a quiet fool, but Anatole is an active\n",
      "one. That is the only difference between them.\n",
      "   </span>\n",
      "   \" He said this smiling\n",
      "in a way more natural and animated than usual, so that the wrinkles\n",
      "round his mouth very clearly revealed something unexpectedly coarse\n",
      "and unpleasant.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    And why are children born to such men as you? If you were not a\n",
      "father there would be nothing I could reproach you with,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna\n",
      "   </span>\n",
      "   , looking up pensively.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I am your faithful slave and to you alone I can confess that my\n",
      "children are the bane of my life. It is the cross I have to bear. That\n",
      "is how I explain it to myself. It can't be helped!\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   He said no more, but expressed his resignation to cruel fate by a\n",
      "gesture.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   meditated.\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(soup_wap.prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<span class=\"green\">Anna\n",
       "Pavlovna Scherer</span>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_wap.find(\"span\", attrs = {\"class\":\"green\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values of an attribute for a given tag instance can be retrieved by using the `get(\"ATTRIBUTE\")` method. For instance, if we want to retrieve the URL of an image we can extract the `src` value from the corresponding `<img>` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../img/gifts/logo.jpg'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_page3.img.get(\"src\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to know all the attibutes associated with a given tag, the `attrs` method is convenient:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'src': '../img/gifts/logo.jpg', 'style': 'float:left;'}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_page3.img.attrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../img/gifts/logo.jpg'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# by returning a dictionary, it is easy to see how \"attrs\" can be used as an alternative to \"get()\"\n",
    "soup_page3.img.attrs[\"src\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'../img/gifts/logo.jpg'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# if you fancy another way to do the same thing...\n",
    "soup_page3.img[\"src\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dealing with multiple HTML tags at once\n",
    "\n",
    "When the same tag is used multiple time in the same page, however, both the `soup.<tag>` notation and the `find()` method allow you to access **only one instance** (i.e. the first):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <head>\n",
      "  <style>\n",
      "   .green{\n",
      "\tcolor:#55ff55;\n",
      "}\n",
      ".red{\n",
      "\tcolor:#ff5555;\n",
      "}\n",
      "#text{\n",
      "\twidth:50%;\n",
      "}\n",
      "  </style>\n",
      " </head>\n",
      " <body>\n",
      "  <h1>\n",
      "   War and Peace\n",
      "  </h1>\n",
      "  <h2>\n",
      "   Chapter 1\n",
      "  </h2>\n",
      "  <div id=\"text\">\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Well, Prince, so Genoa and Lucca are now just family estates of the\n",
      "Buonapartes. But I warn you, if you don't tell me that this means war,\n",
      "if you still try to defend the infamies and horrors perpetrated by\n",
      "that Antichrist- I really believe he is Antichrist- I will have\n",
      "nothing more to do with you and you are no longer my friend, no longer\n",
      "my 'faithful slave,' as you call yourself! But how do you do? I see\n",
      "I have frightened you- sit down and tell me all the news.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   It was in July, 1805, and the speaker was the well-known\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna Scherer\n",
      "   </span>\n",
      "   , maid of honor and favorite of the\n",
      "   <span class=\"green\">\n",
      "    Empress Marya\n",
      "Fedorovna\n",
      "   </span>\n",
      "   . With these words she greeted\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili Kuragin\n",
      "   </span>\n",
      "   , a man\n",
      "of high rank and importance, who was the first to arrive at her\n",
      "reception.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   had had a cough for some days. She was, as\n",
      "she said, suffering from la grippe; grippe being then a new word in\n",
      "   <span class=\"green\">\n",
      "    St. Petersburg\n",
      "   </span>\n",
      "   , used only by the elite.\n",
      "   <p>\n",
      "   </p>\n",
      "   All her invitations without exception, written in French, and\n",
      "delivered by a scarlet-liveried footman that morning, ran as follows:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    If you have nothing better to do, Count [or Prince], and if the\n",
      "prospect of spending an evening with a poor invalid is not too\n",
      "terrible, I shall be very charmed to see you tonight between 7 and 10-\n",
      "Annette Scherer.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Heavens! what a virulent attack!\n",
      "   </span>\n",
      "   \" replied\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   , not in the\n",
      "least disconcerted by this reception. He had just entered, wearing\n",
      "an embroidered court uniform, knee breeches, and shoes, and had\n",
      "stars on his breast and a serene expression on his flat face. He spoke\n",
      "in that refined French in which our grandfathers not only spoke but\n",
      "thought, and with the gentle, patronizing intonation natural to a\n",
      "man of importance who had grown old in society and at court. He went\n",
      "up to\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   , kissed her hand, presenting to her his bald,\n",
      "scented, and shining head, and complacently seated himself on the\n",
      "sofa.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    First of all, dear friend, tell me how you are. Set your friend's\n",
      "mind at rest,\n",
      "   </span>\n",
      "   \" said he without altering his tone, beneath the\n",
      "politeness and affected sympathy of which indifference and even\n",
      "irony could be discerned.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Can one be well while suffering morally? Can one be calm in times\n",
      "like these if one has any feeling?\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    You are\n",
      "staying the whole evening, I hope?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    And the fete at the English ambassador's? Today is Wednesday. I\n",
      "must put in an appearance there,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    My daughter is\n",
      "coming for me to take me there.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I thought today's fete had been canceled. I confess all these\n",
      "festivities and fireworks are becoming wearisome.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    If they had known that you wished it, the entertainment would\n",
      "have been put off,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   , who, like a wound-up clock, by\n",
      "force of habit said things he did not even wish to be believed.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Don't tease! Well, and what has been decided about Novosiltsev's\n",
      "dispatch? You know everything.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    What can one say about it?\n",
      "   </span>\n",
      "   \" replied\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   in a cold,\n",
      "listless tone. \"\n",
      "   <span class=\"red\">\n",
      "    What has been decided? They have decided that\n",
      "Buonaparte has burnt his boats, and I believe that we are ready to\n",
      "burn ours.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili\n",
      "   </span>\n",
      "   always spoke languidly, like an actor repeating a\n",
      "stale part.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   Scherer on the contrary, despite her forty\n",
      "years, overflowed with animation and impulsiveness. To be an\n",
      "enthusiast had become her social vocation and, sometimes even when she\n",
      "did not feel like it, she became enthusiastic in order not to\n",
      "disappoint the expectations of those who knew her. The subdued smile\n",
      "which, though it did not suit her faded features, always played\n",
      "round her lips expressed, as in a spoiled child, a continual\n",
      "consciousness of her charming defect, which she neither wished, nor\n",
      "could, nor considered it necessary, to correct.\n",
      "   <p>\n",
      "   </p>\n",
      "   In the midst of a conversation on political matters\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   burst out:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Oh, don't speak to me of Austria. Perhaps I don't understand\n",
      "things, but Austria never has wished, and does not wish, for war.\n",
      "She is betraying us! Russia alone must save Europe. Our gracious\n",
      "sovereign recognizes his high vocation and will be true to it. That is\n",
      "the one thing I have faith in! Our good and wonderful sovereign has to\n",
      "perform the noblest role on earth, and he is so virtuous and noble\n",
      "that God will not forsake him. He will fulfill his vocation and\n",
      "crush the hydra of revolution, which has become more terrible than\n",
      "ever in the person of this murderer and villain! We alone must\n",
      "avenge the blood of the just one.... Whom, I ask you, can we rely\n",
      "on?... England with her commercial spirit will not and cannot\n",
      "understand the Emperor Alexander's loftiness of soul. She has\n",
      "refused to evacuate Malta. She wanted to find, and still seeks, some\n",
      "secret motive in our actions. What answer did Novosiltsev get? None.\n",
      "The English have not understood and cannot understand the\n",
      "self-abnegation of our Emperor who wants nothing for himself, but only\n",
      "desires the good of mankind. And what have they promised? Nothing! And\n",
      "what little they have promised they will not perform! Prussia has\n",
      "always declared that Buonaparte is invincible, and that all Europe\n",
      "is powerless before him.... And I don't believe a word that Hardenburg\n",
      "says, or Haugwitz either. This famous Prussian neutrality is just a\n",
      "trap. I have faith only in God and the lofty destiny of our adored\n",
      "monarch. He will save Europe!\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   She suddenly paused, smiling at her own impetuosity.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I think,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   with a smile, \"\n",
      "   <span class=\"red\">\n",
      "    that if you had been\n",
      "sent instead of our dear\n",
      "    <span class=\"green\">\n",
      "     Wintzingerode\n",
      "    </span>\n",
      "    you would have captured the\n",
      "    <span class=\"green\">\n",
      "     King of Prussia\n",
      "    </span>\n",
      "    's consent by assault. You are so eloquent. Will you\n",
      "give me a cup of tea?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    In a moment. A propos,\n",
      "   </span>\n",
      "   \" she added, becoming calm again, \"\n",
      "   <span class=\"red\">\n",
      "    I am\n",
      "expecting two very interesting men tonight,\n",
      "    <span class=\"green\">\n",
      "     le Vicomte de Mortemart\n",
      "    </span>\n",
      "    ,\n",
      "who is connected with the\n",
      "    <span class=\"green\">\n",
      "     Montmorencys\n",
      "    </span>\n",
      "    through the\n",
      "    <span class=\"green\">\n",
      "     Rohans\n",
      "    </span>\n",
      "    , one of\n",
      "the best French families. He is one of the genuine emigres, the good\n",
      "ones. And also the\n",
      "    <span class=\"green\">\n",
      "     Abbe Morio\n",
      "    </span>\n",
      "    . Do you know that profound thinker? He\n",
      "has been received by\n",
      "    <span class=\"green\">\n",
      "     the Emperor\n",
      "    </span>\n",
      "    . Had you heard?\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I shall be delighted to meet them,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    But tell me,\n",
      "   </span>\n",
      "   \"\n",
      "he added with studied carelessness as if it had only just occurred\n",
      "to him, though the question he was about to ask was the chief motive\n",
      "of his visit, \"\n",
      "   <span class=\"red\">\n",
      "    is it true that the Dowager Empress wants Baron Funke\n",
      "to be appointed first secretary at Vienna? The baron by all accounts\n",
      "is a poor creature.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Prince Vasili\n",
      "   </span>\n",
      "   wished to obtain this post for his son, but others\n",
      "were trying through the\n",
      "   <span class=\"green\">\n",
      "    Dowager Empress Marya Fedorovna\n",
      "   </span>\n",
      "   to secure it\n",
      "for\n",
      "   <span class=\"green\">\n",
      "    the baron\n",
      "   </span>\n",
      "   .\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   almost closed her eyes to indicate that neither she\n",
      "nor anyone else had a right to criticize what\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   desired or\n",
      "was pleased with.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Baron Funke has been recommended to the Dowager Empress by her\n",
      "sister,\n",
      "   </span>\n",
      "   \" was all she said, in a dry and mournful tone.\n",
      "   <p>\n",
      "   </p>\n",
      "   As she named\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   ,\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna's\n",
      "   </span>\n",
      "   face suddenly assumed an\n",
      "expression of profound and sincere devotion and respect mingled with\n",
      "sadness, and this occurred every time she mentioned her illustrious\n",
      "patroness. She added that\n",
      "   <span class=\"green\">\n",
      "    Her Majesty\n",
      "   </span>\n",
      "   had deigned to show\n",
      "   <span class=\"green\">\n",
      "    Baron\n",
      "Funke\n",
      "   </span>\n",
      "   , and again her face clouded over with sadness.\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   was silent and looked indifferent. But, with the\n",
      "womanly and courtierlike quickness and tact habitual to her,\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna\n",
      "   </span>\n",
      "   wished both to rebuke him (for daring to speak he had done of\n",
      "a man recommended to\n",
      "   <span class=\"green\">\n",
      "    the Empress\n",
      "   </span>\n",
      "   ) and at the same time to console him,\n",
      "so she said:\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Now about your family. Do you know that since your daughter came\n",
      "out everyone has been enraptured by her? They say she is amazingly\n",
      "beautiful.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   bowed to signify his respect and gratitude.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I often think,\n",
      "   </span>\n",
      "   \" she continued after a short pause, drawing nearer\n",
      "to the prince and smiling amiably at him as if to show that\n",
      "political and social topics were ended and the time had come for\n",
      "intimate conversation- \"\n",
      "   <span class=\"red\">\n",
      "    I often think how unfairly sometimes the\n",
      "joys of life are distributed. Why has fate given you two such splendid\n",
      "children? I don't speak of\n",
      "    <span class=\"green\">\n",
      "     Anatole\n",
      "    </span>\n",
      "    , your youngest. I don't like\n",
      "him,\n",
      "   </span>\n",
      "   \" she added in a tone admitting of no rejoinder and raising her\n",
      "eyebrows. \"\n",
      "   <span class=\"red\">\n",
      "    Two such charming children. And really you appreciate\n",
      "them less than anyone, and so you don't deserve to have them.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   And she smiled her ecstatic smile.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I can't help it,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    the prince\n",
      "   </span>\n",
      "   . \"\n",
      "   <span class=\"red\">\n",
      "    Lavater would have said I\n",
      "lack the bump of paternity.\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    Don't joke; I mean to have a serious talk with you. Do you know I\n",
      "am dissatisfied with your younger son? Between ourselves\n",
      "   </span>\n",
      "   \" (and her\n",
      "face assumed its melancholy expression), \"\n",
      "   <span class=\"red\">\n",
      "    he was mentioned at Her\n",
      "Majesty's and you were pitied....\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   <span class=\"green\">\n",
      "    The prince\n",
      "   </span>\n",
      "   answered nothing, but she looked at him significantly,\n",
      "awaiting a reply. He frowned.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    What would you have me do?\n",
      "   </span>\n",
      "   \" he said at last. \"\n",
      "   <span class=\"red\">\n",
      "    You know I did all\n",
      "a father could for their education, and they have both turned out\n",
      "fools. Hippolyte is at least a quiet fool, but Anatole is an active\n",
      "one. That is the only difference between them.\n",
      "   </span>\n",
      "   \" He said this smiling\n",
      "in a way more natural and animated than usual, so that the wrinkles\n",
      "round his mouth very clearly revealed something unexpectedly coarse\n",
      "and unpleasant.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    And why are children born to such men as you? If you were not a\n",
      "father there would be nothing I could reproach you with,\n",
      "   </span>\n",
      "   \" said\n",
      "   <span class=\"green\">\n",
      "    Anna\n",
      "Pavlovna\n",
      "   </span>\n",
      "   , looking up pensively.\n",
      "   <p>\n",
      "   </p>\n",
      "   \"\n",
      "   <span class=\"red\">\n",
      "    I am your faithful slave and to you alone I can confess that my\n",
      "children are the bane of my life. It is the cross I have to bear. That\n",
      "is how I explain it to myself. It can't be helped!\n",
      "   </span>\n",
      "   \"\n",
      "   <p>\n",
      "   </p>\n",
      "   He said no more, but expressed his resignation to cruel fate by a\n",
      "gesture.\n",
      "   <span class=\"green\">\n",
      "    Anna Pavlovna\n",
      "   </span>\n",
      "   meditated.\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not subscriptable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-22-f4f7e6da6685>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msoup_wap\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mprettify\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m180\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;36m1190\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not subscriptable"
     ]
    }
   ],
   "source": [
    "print(soup_wap.prettify())[180:1190]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<span class=\"red\">Well, Prince, so Genoa and Lucca are now just family estates of the\n",
       "Buonapartes. But I warn you, if you don't tell me that this means war,\n",
       "if you still try to defend the infamies and horrors perpetrated by\n",
       "that Antichrist- I really believe he is Antichrist- I will have\n",
       "nothing more to do with you and you are no longer my friend, no longer\n",
       "my 'faithful slave,' as you call yourself! But how do you do? I see\n",
       "I have frightened you- sit down and tell me all the news.</span>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_wap.span"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to extract the **sequence of all the instances of a tag** in a file, we can use the `find_all()` method (previously known as `findAll()` and `findChildren()` in BS 3 and BS 2, respectively)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<span class=\"red\">Well, Prince, so Genoa and Lucca are now just family estates of the\n",
       " Buonapartes. But I warn you, if you don't tell me that this means war,\n",
       " if you still try to defend the infamies and horrors perpetrated by\n",
       " that Antichrist- I really believe he is Antichrist- I will have\n",
       " nothing more to do with you and you are no longer my friend, no longer\n",
       " my 'faithful slave,' as you call yourself! But how do you do? I see\n",
       " I have frightened you- sit down and tell me all the news.</span>,\n",
       " <span class=\"green\">Anna\n",
       " Pavlovna Scherer</span>,\n",
       " <span class=\"green\">Empress Marya\n",
       " Fedorovna</span>,\n",
       " <span class=\"green\">Prince Vasili Kuragin</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"green\">St. Petersburg</span>,\n",
       " <span class=\"red\">If you have nothing better to do, Count [or Prince], and if the\n",
       " prospect of spending an evening with a poor invalid is not too\n",
       " terrible, I shall be very charmed to see you tonight between 7 and 10-\n",
       " Annette Scherer.</span>,\n",
       " <span class=\"red\">Heavens! what a virulent attack!</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"red\">First of all, dear friend, tell me how you are. Set your friend's\n",
       " mind at rest,</span>,\n",
       " <span class=\"red\">Can one be well while suffering morally? Can one be calm in times\n",
       " like these if one has any feeling?</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"red\">You are\n",
       " staying the whole evening, I hope?</span>,\n",
       " <span class=\"red\">And the fete at the English ambassador's? Today is Wednesday. I\n",
       " must put in an appearance there,</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">My daughter is\n",
       " coming for me to take me there.</span>,\n",
       " <span class=\"red\">I thought today's fete had been canceled. I confess all these\n",
       " festivities and fireworks are becoming wearisome.</span>,\n",
       " <span class=\"red\">If they had known that you wished it, the entertainment would\n",
       " have been put off,</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">Don't tease! Well, and what has been decided about Novosiltsev's\n",
       " dispatch? You know everything.</span>,\n",
       " <span class=\"red\">What can one say about it?</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">What has been decided? They have decided that\n",
       " Buonaparte has burnt his boats, and I believe that we are ready to\n",
       " burn ours.</span>,\n",
       " <span class=\"green\">Prince Vasili</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"red\">Oh, don't speak to me of Austria. Perhaps I don't understand\n",
       " things, but Austria never has wished, and does not wish, for war.\n",
       " She is betraying us! Russia alone must save Europe. Our gracious\n",
       " sovereign recognizes his high vocation and will be true to it. That is\n",
       " the one thing I have faith in! Our good and wonderful sovereign has to\n",
       " perform the noblest role on earth, and he is so virtuous and noble\n",
       " that God will not forsake him. He will fulfill his vocation and\n",
       " crush the hydra of revolution, which has become more terrible than\n",
       " ever in the person of this murderer and villain! We alone must\n",
       " avenge the blood of the just one.... Whom, I ask you, can we rely\n",
       " on?... England with her commercial spirit will not and cannot\n",
       " understand the Emperor Alexander's loftiness of soul. She has\n",
       " refused to evacuate Malta. She wanted to find, and still seeks, some\n",
       " secret motive in our actions. What answer did Novosiltsev get? None.\n",
       " The English have not understood and cannot understand the\n",
       " self-abnegation of our Emperor who wants nothing for himself, but only\n",
       " desires the good of mankind. And what have they promised? Nothing! And\n",
       " what little they have promised they will not perform! Prussia has\n",
       " always declared that Buonaparte is invincible, and that all Europe\n",
       " is powerless before him.... And I don't believe a word that Hardenburg\n",
       " says, or Haugwitz either. This famous Prussian neutrality is just a\n",
       " trap. I have faith only in God and the lofty destiny of our adored\n",
       " monarch. He will save Europe!</span>,\n",
       " <span class=\"red\">I think,</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">that if you had been\n",
       " sent instead of our dear <span class=\"green\">Wintzingerode</span> you would have captured the\n",
       " <span class=\"green\">King of Prussia</span>'s consent by assault. You are so eloquent. Will you\n",
       " give me a cup of tea?</span>,\n",
       " <span class=\"green\">Wintzingerode</span>,\n",
       " <span class=\"green\">King of Prussia</span>,\n",
       " <span class=\"red\">In a moment. A propos,</span>,\n",
       " <span class=\"red\">I am\n",
       " expecting two very interesting men tonight, <span class=\"green\">le Vicomte de Mortemart</span>,\n",
       " who is connected with the <span class=\"green\">Montmorencys</span> through the <span class=\"green\">Rohans</span>, one of\n",
       " the best French families. He is one of the genuine emigres, the good\n",
       " ones. And also the <span class=\"green\">Abbe Morio</span>. Do you know that profound thinker? He\n",
       " has been received by <span class=\"green\">the Emperor</span>. Had you heard?</span>,\n",
       " <span class=\"green\">le Vicomte de Mortemart</span>,\n",
       " <span class=\"green\">Montmorencys</span>,\n",
       " <span class=\"green\">Rohans</span>,\n",
       " <span class=\"green\">Abbe Morio</span>,\n",
       " <span class=\"green\">the Emperor</span>,\n",
       " <span class=\"red\">I shall be delighted to meet them,</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">But tell me,</span>,\n",
       " <span class=\"red\">is it true that the Dowager Empress wants Baron Funke\n",
       " to be appointed first secretary at Vienna? The baron by all accounts\n",
       " is a poor creature.</span>,\n",
       " <span class=\"green\">Prince Vasili</span>,\n",
       " <span class=\"green\">Dowager Empress Marya Fedorovna</span>,\n",
       " <span class=\"green\">the baron</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>,\n",
       " <span class=\"green\">the Empress</span>,\n",
       " <span class=\"red\">Baron Funke has been recommended to the Dowager Empress by her\n",
       " sister,</span>,\n",
       " <span class=\"green\">the Empress</span>,\n",
       " <span class=\"green\">Anna Pavlovna's</span>,\n",
       " <span class=\"green\">Her Majesty</span>,\n",
       " <span class=\"green\">Baron\n",
       " Funke</span>,\n",
       " <span class=\"green\">The prince</span>,\n",
       " <span class=\"green\">Anna\n",
       " Pavlovna</span>,\n",
       " <span class=\"green\">the Empress</span>,\n",
       " <span class=\"red\">Now about your family. Do you know that since your daughter came\n",
       " out everyone has been enraptured by her? They say she is amazingly\n",
       " beautiful.</span>,\n",
       " <span class=\"green\">The prince</span>,\n",
       " <span class=\"red\">I often think,</span>,\n",
       " <span class=\"red\">I often think how unfairly sometimes the\n",
       " joys of life are distributed. Why has fate given you two such splendid\n",
       " children? I don't speak of <span class=\"green\">Anatole</span>, your youngest. I don't like\n",
       " him,</span>,\n",
       " <span class=\"green\">Anatole</span>,\n",
       " <span class=\"red\">Two such charming children. And really you appreciate\n",
       " them less than anyone, and so you don't deserve to have them.</span>,\n",
       " <span class=\"red\">I can't help it,</span>,\n",
       " <span class=\"green\">the prince</span>,\n",
       " <span class=\"red\">Lavater would have said I\n",
       " lack the bump of paternity.</span>,\n",
       " <span class=\"red\">Don't joke; I mean to have a serious talk with you. Do you know I\n",
       " am dissatisfied with your younger son? Between ourselves</span>,\n",
       " <span class=\"red\">he was mentioned at Her\n",
       " Majesty's and you were pitied....</span>,\n",
       " <span class=\"green\">The prince</span>,\n",
       " <span class=\"red\">What would you have me do?</span>,\n",
       " <span class=\"red\">You know I did all\n",
       " a father could for their education, and they have both turned out\n",
       " fools. Hippolyte is at least a quiet fool, but Anatole is an active\n",
       " one. That is the only difference between them.</span>,\n",
       " <span class=\"red\">And why are children born to such men as you? If you were not a\n",
       " father there would be nothing I could reproach you with,</span>,\n",
       " <span class=\"green\">Anna\n",
       " Pavlovna</span>,\n",
       " <span class=\"red\">I am your faithful slave and to you alone I can confess that my\n",
       " children are the bane of my life. It is the cross I have to bear. That\n",
       " is how I explain it to myself. It can't be helped!</span>,\n",
       " <span class=\"green\">Anna Pavlovna</span>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_wap.find_all(\"span\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The `find_all()` method as well allows for  the extraction of  all tags by exploiting cues in the markup, such as a given **class attribute** value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<span class=\"green\">Anna\n",
       " Pavlovna Scherer</span>, <span class=\"green\">Empress Marya\n",
       " Fedorovna</span>, <span class=\"green\">Prince Vasili Kuragin</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">St. Petersburg</span>, <span class=\"green\">the prince</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">the prince</span>, <span class=\"green\">the prince</span>, <span class=\"green\">the prince</span>, <span class=\"green\">Prince Vasili</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">the prince</span>, <span class=\"green\">Wintzingerode</span>, <span class=\"green\">King of Prussia</span>, <span class=\"green\">le Vicomte de Mortemart</span>, <span class=\"green\">Montmorencys</span>, <span class=\"green\">Rohans</span>, <span class=\"green\">Abbe Morio</span>, <span class=\"green\">the Emperor</span>, <span class=\"green\">the prince</span>, <span class=\"green\">Prince Vasili</span>, <span class=\"green\">Dowager Empress Marya Fedorovna</span>, <span class=\"green\">the baron</span>, <span class=\"green\">Anna Pavlovna</span>, <span class=\"green\">the Empress</span>, <span class=\"green\">the Empress</span>, <span class=\"green\">Anna Pavlovna's</span>, <span class=\"green\">Her Majesty</span>, <span class=\"green\">Baron\n",
       " Funke</span>, <span class=\"green\">The prince</span>, <span class=\"green\">Anna\n",
       " Pavlovna</span>, <span class=\"green\">the Empress</span>, <span class=\"green\">The prince</span>, <span class=\"green\">Anatole</span>, <span class=\"green\">the prince</span>, <span class=\"green\">The prince</span>, <span class=\"green\">Anna\n",
       " Pavlovna</span>, <span class=\"green\">Anna Pavlovna</span>]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup_wap.find_all(\"span\",  attrs = {\"class\":\"green\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Web Crawling\n",
    "\n",
    "Web Crawlers are softwares designed to collect pages from the Web. In essence, they recursively implement the following steps: \n",
    "\n",
    "- they start by retrieving the page content for an URL \n",
    "\n",
    "\n",
    "- they then parse it to retrieve other URLs of interest\n",
    "\n",
    "\n",
    "- they then focus on these new URLs, for each of which they repeat the whole process, ad infinitum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, if you want to crawl and **entire site**:\n",
    "\n",
    "- start with a top-level page\n",
    "\n",
    "\n",
    "- parse the page (retrieve the data your application need) and extract all the internal links, by ignoring already visited URLs\n",
    "\n",
    "\n",
    "- for each new link, move to the corresponding page and repeat the previous step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A Random walk through Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's set our starting page URL, fetch it and parse its HTML:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "starting_page = urlopen(\"https://en.wikipedia.org/wiki/Chris_Cornell\")\n",
    "soup = BeautifulSoup(starting_page, \"lxml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, it should be easy to extract all the links in the page:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a id=\"top\"></a>\n",
      "<a href=\"/wiki/Wikipedia:Protection_policy#semi\" title=\"This article is semi-protected until June 9, 2020 at 11:57 UTC.\"><img alt=\"Page semi-protected\" data-file-height=\"512\" data-file-width=\"512\" decoding=\"async\" height=\"20\" src=\"//upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/20px-Semi-protection-shackle.svg.png\" srcset=\"//upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/30px-Semi-protection-shackle.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/1/1b/Semi-protection-shackle.svg/40px-Semi-protection-shackle.svg.png 2x\" width=\"20\"/></a>\n",
      "<a class=\"mw-jump-link\" href=\"#mw-head\">Jump to navigation</a>\n",
      "<a class=\"mw-jump-link\" href=\"#p-search\">Jump to search</a>\n",
      "<a class=\"image\" href=\"/wiki/File:Chris_Cornell.jpg\"><img alt=\"Chris Cornell.jpg\" data-file-height=\"900\" data-file-width=\"600\" decoding=\"async\" height=\"330\" src=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Chris_Cornell.jpg/220px-Chris_Cornell.jpg\" srcset=\"//upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Chris_Cornell.jpg/330px-Chris_Cornell.jpg 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/a/ab/Chris_Cornell.jpg/440px-Chris_Cornell.jpg 2x\" width=\"220\"/></a>\n",
      "<a class=\"mw-redirect\" href=\"/wiki/Seattle,_Washington\" title=\"Seattle, Washington\">Seattle, Washington</a>\n",
      "<a class=\"mw-redirect\" href=\"/wiki/Detroit,_Michigan\" title=\"Detroit, Michigan\">Detroit, Michigan</a>\n",
      "<a href=\"/wiki/Suicide_by_hanging\" title=\"Suicide by hanging\">Suicide by hanging</a>\n",
      "<a href=\"/wiki/Hollywood_Forever_Cemetery\" title=\"Hollywood Forever Cemetery\">Hollywood Forever Cemetery</a>\n",
      "<a href=\"/wiki/Susan_Silver\" title=\"Susan Silver\">Susan Silver</a>\n"
     ]
    }
   ],
   "source": [
    "# links are defined by <a> tag\n",
    "for link in soup.find_all(\"a\")[:10]:\n",
    "    print(link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's ignore all the \"a\" tags without an \"href\" attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/wiki/Wikipedia:Protection_policy#semi\n",
      "#mw-head\n",
      "#p-search\n",
      "/wiki/File:Chris_Cornell.jpg\n",
      "/wiki/Seattle,_Washington\n",
      "/wiki/Detroit,_Michigan\n",
      "/wiki/Suicide_by_hanging\n",
      "/wiki/Hollywood_Forever_Cemetery\n",
      "/wiki/Susan_Silver\n",
      "/wiki/List_of_awards_and_nominations_received_by_Chris_Cornell\n"
     ]
    }
   ],
   "source": [
    "for link in [tag for tag in soup.find_all(\"a\") if 'href' in tag.attrs][:10]:\n",
    "    print(link.attrs['href'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wikipedia is full of sidebar, footer, and header links that appear on every page, along with links to the category pages, talk pages, and other pages that do not contain different articles:\n",
    "\n",
    "```\n",
    "/wiki/Template_talk:Chris_Cornell\n",
    "```\n",
    "\n",
    "```\n",
    "#cite_note-147\n",
    "```\n",
    "\n",
    "Moreover, we don't want to visit pages outside of Wikipedia:\n",
    "\n",
    "```\n",
    "http://www.chriscornell.com/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Relevant links have three thing in common:\n",
    "\n",
    "- they reside within the `div` with the `id` set to `bodyContent`\n",
    "\n",
    "\n",
    "- the URLs do not contain semicolons\n",
    "\n",
    "\n",
    "- the URLs begin with `/wiki/`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/wiki/Seattle,_Washington\n",
      "/wiki/Detroit,_Michigan\n",
      "/wiki/Suicide_by_hanging\n",
      "/wiki/Hollywood_Forever_Cemetery\n",
      "/wiki/Susan_Silver\n",
      "/wiki/List_of_awards_and_nominations_received_by_Chris_Cornell\n",
      "/wiki/Alternative_metal\n",
      "/wiki/Heavy_metal_music\n",
      "/wiki/Grunge\n",
      "/wiki/Alternative_rock\n",
      "/wiki/Hard_rock\n",
      "/wiki/SST_Records\n",
      "/wiki/Sub_Pop\n",
      "/wiki/A%26M_Records\n",
      "/wiki/Epic_Records\n",
      "/wiki/Interscope_Records\n",
      "/wiki/Mosley_Music_Group\n",
      "/wiki/Soundgarden\n",
      "/wiki/Audioslave\n",
      "/wiki/Pearl_Jam\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Center_for_Disease_Control_Boys\n",
      "/wiki/Alice_Mudgarden\n",
      "/wiki/Heart_(band)\n",
      "/wiki/Mad_Season_(band)\n",
      "/wiki/N%C3%A9\n",
      "/wiki/Rock_music\n",
      "/wiki/Soundgarden\n",
      "/wiki/Audioslave\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Andrew_Wood_(singer)\n",
      "/wiki/Grunge\n",
      "/wiki/Octave\n",
      "/wiki/Belting_(music)\n",
      "/wiki/Euphoria_Morning\n",
      "/wiki/Carry_On_(Chris_Cornell_album)\n",
      "/wiki/Scream_(Chris_Cornell_album)\n",
      "/wiki/Higher_Truth\n",
      "/wiki/Songbook_(Chris_Cornell_album)\n",
      "/wiki/The_Roads_We_Choose_%E2%80%93_A_Retrospective\n",
      "/wiki/Chris_Cornell_(album)\n",
      "/wiki/Golden_Globe_Award\n",
      "/wiki/Machine_Gun_Preacher\n",
      "/wiki/You_Know_My_Name_(Chris_Cornell_song)\n",
      "/wiki/James_Bond\n",
      "/wiki/Casino_Royale_(2006_film)\n",
      "/wiki/The_Promise_(Chris_Cornell_song)\n",
      "/wiki/The_Promise_(2016_American_film)\n",
      "/wiki/Grammy\n",
      "/wiki/Guitar_World\n",
      "/wiki/Hit_Parader\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Depression_(mood)\n",
      "/wiki/Detroit\n",
      "/wiki/Fox_Theater_(Detroit)\n",
      "/wiki/Suicide_by_hanging\n",
      "/wiki/Seattle\n",
      "/wiki/Irish_Catholic\n",
      "/wiki/One_Tin_Soldier\n",
      "/wiki/Shorewood_High_School_(Washington)\n",
      "/wiki/Little_Richard\n",
      "/wiki/The_Beatles\n",
      "/wiki/Alcohol_(drug)\n",
      "/wiki/Cannabis\n",
      "/wiki/Lysergic_acid_diethylamide\n",
      "/wiki/Psilocybin_mushrooms\n",
      "/wiki/Prescription_drugs\n",
      "/wiki/Phencyclidine\n",
      "/wiki/Panic_disorder\n",
      "/wiki/Agoraphobia\n",
      "/wiki/Snare_drum\n",
      "/wiki/Chef#Sous-chef\n",
      "/wiki/Ray%27s_Boathouse\n",
      "/wiki/Hiro_Yamamoto\n",
      "/wiki/Kim_Thayil\n",
      "/wiki/Jam_session\n",
      "/wiki/Soundgarden\n",
      "/wiki/Soundgarden\n",
      "/wiki/Scott_Sundquist\n",
      "/wiki/C/Z_Records\n",
      "/wiki/Deep_Six_(album)\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Skin_Yard\n",
      "/wiki/Kim_Thayil\n",
      "/wiki/Sub_Pop\n",
      "/wiki/Screaming_Life\n",
      "/wiki/Extended_play\n",
      "/wiki/Fopp_(EP)\n",
      "/wiki/Screaming_Life/Fopp\n",
      "/wiki/SST_Records\n",
      "/wiki/Ultramega_OK\n",
      "/wiki/Grammy_Award\n",
      "/wiki/Grammy_Award_for_Best_Metal_Performance\n",
      "/wiki/A%26M_Records\n",
      "/wiki/Grunge\n",
      "/wiki/Louder_Than_Love\n",
      "/wiki/Western_Washington_University\n",
      "/wiki/Nirvana_(band)\n",
      "/wiki/Jason_Everman\n",
      "/wiki/Ben_Shepherd\n",
      "/wiki/Alice_in_Chains\n",
      "/wiki/Pearl_Jam\n",
      "/wiki/Badmotorfinger\n",
      "/wiki/Jesus_Christ_Pose\n",
      "/wiki/Outshined\n",
      "/wiki/Rusty_Cage\n",
      "/wiki/Johnny_Cash\n",
      "/wiki/Unchained_(Johnny_Cash_album)\n",
      "/wiki/Road_Rash\n",
      "/wiki/Room_a_Thousand_Years_Wide\n",
      "/wiki/Guitar_World\n",
      "/wiki/Superunknown\n",
      "/wiki/Billboard_200\n",
      "/wiki/Spoonman\n",
      "/wiki/Black_Hole_Sun\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Jon_Pareles\n",
      "/wiki/The_New_York_Times\n",
      "/wiki/Heavy_metal_music\n",
      "/wiki/Entertainment_Weekly\n",
      "/wiki/Grammy_Award_for_Best_Rock_Album\n",
      "/wiki/MTV_Video_Music_Award\n",
      "/wiki/Clio_Awards\n",
      "/wiki/Rolling_Stone%27s_500_Greatest_Albums_of_All_Time\n",
      "/wiki/VH1\n",
      "/wiki/Down_on_the_Upside\n",
      "/wiki/Pretty_Noose\n",
      "/wiki/Burden_in_My_Hand\n",
      "/wiki/Blow_Up_the_Outside_World\n",
      "/wiki/Lollapalooza\n",
      "/wiki/Louder_Than_Love\n",
      "/wiki/Lollapalooza\n",
      "/wiki/The_Showbox\n",
      "/wiki/Anagram\n",
      "/wiki/Conan_O%27Brien\n",
      "/wiki/Conan_(talk_show)\n",
      "/wiki/The_Avengers_(2012_film)\n",
      "/wiki/King_Animal\n",
      "/wiki/The_Seattle_Times\n",
      "/wiki/Alain_Johannes\n",
      "/wiki/Natasha_Shneider\n",
      "/wiki/Eleven_(band)\n",
      "/wiki/Euphoria_Morning\n",
      "/wiki/Henry_Fonda_Theater\n",
      "/wiki/Can%27t_Change_Me\n",
      "/wiki/Grammy_Award_for_Best_Male_Rock_Vocal_Performance\n",
      "/wiki/42nd_Grammy_Awards\n",
      "/wiki/Jeff_Buckley\n",
      "/wiki/Titan_A.E.\n",
      "/wiki/Stockholm\n",
      "/wiki/Music_download\n",
      "/wiki/The_Roads_We_Choose_%E2%80%93_A_Retrospective\n",
      "/wiki/Carry_On_(Chris_Cornell_album)\n",
      "/wiki/Steve_Lillywhite\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Gary_Lucas\n",
      "/wiki/Acoustic_guitar\n",
      "/wiki/Rear-end_collision\n",
      "/wiki/Studio_City,_Los_Angeles\n",
      "/wiki/Melkweg\n",
      "/wiki/Amsterdam\n",
      "/wiki/Aerosmith\n",
      "/wiki/Dublin\n",
      "/wiki/Hyde_Park,_London\n",
      "/wiki/Linkin_Park\n",
      "/wiki/Jason_Sutter\n",
      "/wiki/Projekt_Revolution\n",
      "/wiki/Chester_Bennington\n",
      "/wiki/Crawling_(song)\n",
      "/wiki/Aaron_Lewis_(musician)\n",
      "/wiki/Reanimation_(Linkin_Park_album)\n",
      "/wiki/Timbaland\n",
      "/wiki/Scream_(Chris_Cornell_album)\n",
      "/wiki/Billboard_200\n",
      "/wiki/WWPW\n",
      "/wiki/John_Lennon\n",
      "/wiki/Imagine_(John_Lennon_song)\n",
      "/wiki/The_Tonight_Show_with_Conan_O%27Brien\n",
      "/wiki/Songbook_(Chris_Cornell_album)\n",
      "/wiki/Led_Zeppelin\n",
      "/wiki/Thank_You_(Led_Zeppelin_song)\n",
      "/wiki/John_Lennon\n",
      "/wiki/Imagine_(John_Lennon_song)\n",
      "/wiki/AllMusic\n",
      "/wiki/White_House\n",
      "/wiki/Barack_Obama\n",
      "/wiki/Inaugural_Ball\n",
      "/wiki/Twitter\n",
      "/wiki/Higher_Truth\n",
      "/wiki/The_Promise_(Chris_Cornell_song)\n",
      "/wiki/The_Promise_(2016_American_film)\n",
      "/wiki/Armenian_Genocide\n",
      "/wiki/Johnny_Cash\n",
      "/wiki/Grammy_award\n",
      "/wiki/61st_Grammy_Awards\n",
      "/wiki/Audioslave\n",
      "/wiki/Montreux_Jazz_Festival\n",
      "/wiki/Audioslave\n",
      "/wiki/Zack_de_la_Rocha\n",
      "/wiki/Rage_Against_the_Machine\n",
      "/wiki/Rick_Rubin\n",
      "/wiki/Tom_Morello\n",
      "/wiki/Tim_Commerford\n",
      "/wiki/Brad_Wilk\n",
      "/wiki/Audioslave_(album)\n",
      "/wiki/Cochise_(song)\n",
      "/wiki/Like_a_Stone\n",
      "/wiki/Show_Me_How_to_Live_(song)\n",
      "/wiki/Ozzfest\n",
      "/wiki/Drug_rehabilitation\n",
      "/wiki/Metal_Hammer\n",
      "/wiki/San_Diego_CityBeat\n",
      "/wiki/Sobriety\n",
      "/wiki/Out_of_Exile\n",
      "/wiki/Out_of_Exile_(song)\n",
      "/wiki/Be_Yourself_(Audioslave_song)\n",
      "/wiki/Your_Time_Has_Come\n",
      "/wiki/Doesn%27t_Remind_Me\n",
      "/wiki/AllMusic\n",
      "/wiki/Havana\n",
      "/wiki/Cuba\n",
      "/wiki/Revelations_(Audioslave_album)\n",
      "/wiki/Funk\n",
      "/wiki/Rhythm_and_blues\n",
      "/wiki/Original_Fire\n",
      "/wiki/Revelations_(Audioslave_song)\n",
      "/wiki/Michael_Mann_(director)\n",
      "/wiki/Miami_Vice_(film)\n",
      "/wiki/The_Nightwatchman\n",
      "/wiki/Existentialism\n",
      "/wiki/Hedonism\n",
      "/wiki/Spirituality\n",
      "/wiki/Prophets_of_Rage\n",
      "/wiki/Donald_Trump\n",
      "/wiki/President_of_the_United_States\n",
      "/wiki/Center_for_Disease_Control_Boys\n",
      "/wiki/Western_swing\n",
      "/wiki/Center_for_Disease_Control_Boys\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Temple_of_the_Dog_(album)\n",
      "/wiki/Andrew_Wood_(singer)\n",
      "/wiki/Mother_Love_Bone\n",
      "/wiki/Jeff_Ament\n",
      "/wiki/Stone_Gossard\n",
      "/wiki/Mike_McCready\n",
      "/wiki/Eddie_Vedder\n",
      "/wiki/Dave_Krusen\n",
      "/wiki/Say_Hello_2_Heaven\n",
      "/wiki/Hunger_Strike_(song)\n",
      "/wiki/Santa_Barbara_Bowl\n",
      "/wiki/Jerry_Cantrell\n",
      "/wiki/Pearl_Jam_Twenty_Tour\n",
      "/wiki/Alpine_Valley_Music_Theatre\n",
      "/wiki/Shoreline_Amphitheater\n",
      "/wiki/Mountain_View,_California\n",
      "/wiki/Bridge_School_Benefit\n",
      "/wiki/Paramount_Theatre_(Seattle)\n",
      "/wiki/Alice_Mudgarden\n",
      "/wiki/Layne_Staley\n",
      "/wiki/Jerry_Cantrell\n",
      "/wiki/Alice_in_Chains\n",
      "/wiki/Mark_Arm\n",
      "/wiki/Mudhoney\n",
      "/wiki/Sap_(EP)\n",
      "/wiki/Alice_Mudgarden\n",
      "/wiki/Black_Hawk_Down_(film)\n",
      "/wiki/M.A.C.C.\n",
      "/wiki/Mike_McCready\n",
      "/wiki/Jeff_Ament\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Screaming_Trees\n",
      "/wiki/Uncle_Anesthesia\n",
      "/wiki/Alice_Cooper\n",
      "/wiki/The_Last_Temptation_(Alice_Cooper_album)\n",
      "/wiki/Eleven_(band)\n",
      "/wiki/Ave_Maria_(musical_settings)\n",
      "/wiki/A_Very_Special_Christmas_3\n",
      "/wiki/Heart_(band)\n",
      "/wiki/Ann_Wilson\n",
      "/wiki/Nancy_Wilson_(rock_musician)\n",
      "/wiki/The_Rolling_Stones\n",
      "/wiki/Wild_Horses_(The_Rolling_Stones_song)\n",
      "/wiki/Whirlygig_(album)\n",
      "/wiki/Layne_Staley\n",
      "/wiki/Eleven_(band)\n",
      "/wiki/Spider-Man_2\n",
      "/wiki/Spider-Man_2_(soundtrack)\n",
      "/wiki/Jimmy_Gnecco\n",
      "/wiki/Ours_(band)\n",
      "/wiki/Brian_May\n",
      "/wiki/Queen_(band)\n",
      "/wiki/American_Society_of_Composers,_Authors_and_Publishers\n",
      "/wiki/Street_team\n",
      "/wiki/Brian_Howes\n",
      "/wiki/David_Cook_(singer)\n",
      "/wiki/American_Idol\n",
      "/wiki/Light_On\n",
      "/wiki/Sandy_Hook_Elementary_School_shooting\n",
      "/wiki/Slash_(album)\n",
      "/wiki/Slash_(musician)\n",
      "/wiki/Gabin_(Italian_band)\n",
      "/wiki/Carlos_Santana\n",
      "/wiki/Whole_Lotta_Love\n",
      "/wiki/The_Everly_Brothers\n",
      "/wiki/All_I_Have_to_Do_Is_Dream\n",
      "/wiki/Rita_Wilson\n",
      "/wiki/Heart_(band)\n",
      "/wiki/Rock_and_Roll_Hall_of_Fame\n",
      "/wiki/Ann_Wilson\n",
      "/wiki/Nancy_Wilson_(rock_musician)\n",
      "/wiki/Barracuda_(song)\n",
      "/wiki/Duff_McKagan\n",
      "/wiki/Sean_Kinney\n",
      "/wiki/Seattle_Symphony\n",
      "/wiki/Mad_Season_(band)\n",
      "/wiki/Seattle_Symphony\n",
      "/wiki/Great_Expectations_(1998_film)\n",
      "/wiki/David_Arnold\n",
      "/wiki/You_Know_My_Name\n",
      "/wiki/James_Bond\n",
      "/wiki/Casino_Royale_(2006_film)\n",
      "/wiki/Octopussy\n",
      "/wiki/Satellite_Awards_2006\n",
      "/wiki/Satellite_Award\n",
      "/wiki/Satellite_Award_for_Best_Original_Song\n",
      "/wiki/World_Soundtrack_Awards_2007\n",
      "/wiki/World_Soundtrack_Academy\n",
      "/wiki/World_Soundtrack_Award_for_Best_Original_Song_Written_Directly_for_a_Film\n",
      "/wiki/Grammy_Award_for_Best_Song_Written_for_a_Motion_Picture,_Television_or_Other_Visual_Media\n",
      "/wiki/50th_Grammy_Awards\n",
      "/wiki/Marc_Forster\n",
      "/wiki/Machine_Gun_Preacher\n",
      "/wiki/Golden_Globe_Award\n",
      "/wiki/Sam_Childers\n",
      "/wiki/The_Avengers_(2012_film)\n",
      "/wiki/12_Years_a_Slave_(film)\n",
      "/wiki/Joy_Williams_(singer)\n",
      "/wiki/Stay_with_Me_(Lorraine_Ellison_song)\n",
      "/wiki/HBO\n",
      "/wiki/Vinyl_(TV_series)\n",
      "/wiki/Terry_Reid\n",
      "/wiki/Drive_My_Car\n",
      "/wiki/Beat_Bugs\n",
      "/wiki/The_Promise_(Chris_Cornell_song)\n",
      "/wiki/The_Promise_(2016_American_film)\n",
      "/wiki/Lisbon\n",
      "/wiki/Paul_McCartney\n",
      "/wiki/XTC_(band)\n",
      "/wiki/Siouxsie_and_the_Banshees\n",
      "/wiki/Ultravox\n",
      "/wiki/Bauhaus_(band)\n",
      "/wiki/Interval_(music)\n",
      "/wiki/Baritone\n",
      "/wiki/Tenor\n",
      "/wiki/Blues\n",
      "/wiki/Neo-soul\n",
      "/wiki/Singles_(1992_film)\n",
      "/wiki/Cameron_Crowe\n",
      "/wiki/Matt_Dillon\n",
      "/wiki/Bridget_Fonda\n",
      "/wiki/John_Varvatos\n",
      "/wiki/Philip_Carlo\n",
      "/wiki/Richard_Ramirez\n",
      "/wiki/James_Franco\n",
      "/wiki/Garbage_(band)\n",
      "/wiki/Garbage_(album)\n",
      "/wiki/Shirley_Manson\n",
      "/wiki/Black_Hole_Sun\n",
      "/wiki/Outshined\n",
      "/wiki/Steven_Baigelman\n",
      "/wiki/Feeling_Minnesota\n",
      "/wiki/Keanu_Reeves\n",
      "/wiki/Cameron_Diaz\n",
      "/wiki/Euphoria_Morning\n",
      "/wiki/Lorene_Scafaria\n",
      "/wiki/Seeking_a_Friend_for_the_End_of_the_World\n",
      "/wiki/Steve_Carell\n",
      "/wiki/Keira_Knightley\n",
      "/wiki/Jeopardy!\n",
      "/wiki/Susan_Silver\n",
      "/wiki/Alice_in_Chains\n",
      "/wiki/Soundgarden\n",
      "/wiki/Screaming_Trees\n",
      "/wiki/Paris\n",
      "/wiki/Freethought\n",
      "/wiki/Andrew_Wood_(singer)\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/The_Guardian\n",
      "/wiki/Eddie_Vedder\n",
      "/wiki/Alpine_Valley\n",
      "/wiki/Pearl_Jam_Twenty\n",
      "/wiki/OxyContin\n",
      "/wiki/Cannabis\n",
      "/wiki/Lysergic_acid_diethylamide\n",
      "/wiki/Psilocybin_mushrooms\n",
      "/wiki/Phencyclidine\n",
      "/wiki/Panic_disorder\n",
      "/wiki/Agoraphobia\n",
      "/wiki/Stevie_Ray_Vaughan\n",
      "/wiki/MusiCares\n",
      "/wiki/Alice_Cooper\n",
      "/wiki/Jack_Daniel%27s\n",
      "/wiki/MGM_Grand_Detroit\n",
      "/wiki/Detroit,_Michigan\n",
      "/wiki/Fox_Theatre_(Detroit)\n",
      "/wiki/Resistance_band\n",
      "/wiki/Emergency_medical_services\n",
      "/wiki/Suicide_by_hanging\n",
      "/wiki/Butalbital\n",
      "/wiki/Headache\n",
      "/wiki/Lorazepam\n",
      "/wiki/Pseudoephedrine\n",
      "/wiki/Norpseudoephedrine\n",
      "/wiki/Caffeine\n",
      "/wiki/No-Doz\n",
      "/wiki/Naloxone\n",
      "/wiki/Nearly_Forgot_My_Broken_Heart\n",
      "/wiki/Old_West\n",
      "/wiki/Detroit_Police_Department\n",
      "/wiki/The_Detroit_News\n",
      "/wiki/Hollywood_Forever_Cemetery\n",
      "/wiki/Kim_Thayil\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Ben_Shepherd\n",
      "/wiki/Hiro_Yamamoto\n",
      "/wiki/Scott_Sundquist\n",
      "/wiki/Tom_Morello\n",
      "/wiki/Jeff_Ament\n",
      "/wiki/Mike_McCready\n",
      "/wiki/Chester_Bennington\n",
      "/wiki/Leonard_Cohen\n",
      "/wiki/Hallelujah_(Leonard_Cohen_song)\n",
      "/wiki/Johnny_Ramone\n",
      "/wiki/Cenotaph\n",
      "/wiki/Space_Needle\n",
      "/wiki/Ann_Wilson\n",
      "/wiki/Jimmy_Kimmel_Live!\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Alice_in_Chains\n",
      "/wiki/Faith_No_More\n",
      "/wiki/Seattle_Mariners\n",
      "/wiki/Chicago_White_Sox\n",
      "/wiki/Oakland_Athletics\n",
      "/wiki/Trevor_Plouffe\n",
      "/wiki/Linkin_Park\n",
      "/wiki/One_More_Light_(song)\n",
      "/wiki/Jimmy_Kimmel\n",
      "/wiki/Billboard_Music_Awards\n",
      "/wiki/Imagine_Dragons\n",
      "/wiki/Dan_Reynolds_(musician)\n",
      "/wiki/Norah_Jones\n",
      "/wiki/Fox_Theatre_(Detroit)\n",
      "/wiki/Prophets_of_Rage\n",
      "/wiki/Serj_Tankian\n",
      "/wiki/System_of_a_Down\n",
      "/wiki/Alternative_Press_Music_Awards\n",
      "/wiki/Pierce_The_Veil\n",
      "/wiki/Stone_Gossard\n",
      "/wiki/Shawn_Smith\n",
      "/wiki/Foo_Fighters\n",
      "/wiki/Taylor_Hawkins\n",
      "/wiki/American_Satan\n",
      "/wiki/Neil_Young\n",
      "/wiki/Hey_Hey,_My_My_(Into_The_Black)\n",
      "/wiki/Pearl_Jam_(album)\n",
      "/wiki/Rock_and_Roll_Hall_of_Fame\n",
      "/wiki/Black_Hole_Sun\n",
      "/wiki/U2_(band)\n",
      "/wiki/Running_to_Stand_Still\n",
      "/wiki/Rose_Bowl_(stadium)\n",
      "/wiki/Hunted_Down\n",
      "/wiki/Down_on_the_Upside\n",
      "/wiki/Tool_(band)\n",
      "/wiki/Guns_N%27_Roses\n",
      "/wiki/Not_In_This_Lifetime..._Tour\n",
      "/wiki/Never_Fade_(Alice_in_Chains_song)\n",
      "/wiki/Rainier_Fog\n",
      "/wiki/William_DuVall\n",
      "/wiki/I_Am_The_Highway\n",
      "/wiki/Statue_of_Chris_Cornell\n",
      "/wiki/Seattle_Center\n",
      "/wiki/Foo_Fighters\n",
      "/wiki/Metallica\n",
      "/wiki/Melvins\n",
      "/wiki/Fiona_Apple\n",
      "/wiki/Jerry_Cantrell\n",
      "/wiki/William_DuVall\n",
      "/wiki/Miguel_(singer)\n",
      "/wiki/Nikka_Costa\n",
      "/wiki/Adam_Levine\n",
      "/wiki/Jesse_Carmichael\n",
      "/wiki/Jack_Black\n",
      "/wiki/Geezer_Butler\n",
      "/wiki/Ryan_Adams\n",
      "/wiki/Taylor_Momsen\n",
      "/wiki/Brandi_Carlile\n",
      "/wiki/Perry_Farrell\n",
      "/wiki/Juliette_Lewis\n",
      "/wiki/Josh_Homme\n",
      "/wiki/Miley_Cyrus\n",
      "/wiki/Alain_Johannes\n",
      "/wiki/Wayne_Kramer_(guitarist)\n",
      "/wiki/Peter_Frampton\n",
      "/wiki/Ziggy_Marley\n",
      "/wiki/Columbus_Dispatch\n",
      "/wiki/Columbus,_Ohio\n",
      "/wiki/Ian_Curtis\n",
      "/wiki/Post-punk\n",
      "/wiki/Joy_Division\n",
      "/wiki/Rolling_Stone_(magazine)\n",
      "/wiki/Stephen_Morris_(musician)\n",
      "/wiki/New_Order_(band)\n",
      "/wiki/Chester_Bennington\n",
      "/wiki/Linkin_Park\n",
      "/wiki/Leonard_Cohen\n",
      "/wiki/Hallelujah_(Leonard_Cohen_song)\n",
      "/wiki/Mike_Shinoda\n",
      "/wiki/Jimmy_Kimmel_Live!\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Axl_Rose\n",
      "/wiki/Ronnie_James_Dio\n",
      "/wiki/Scott_Stapp\n",
      "/wiki/Alice_Cooper\n",
      "/wiki/Living_Colour\n",
      "/wiki/Vernon_Reid\n",
      "/wiki/Jeff_Ament\n",
      "/wiki/Jimi_Hendrix\n",
      "/wiki/Chris_Cornell_discography\n",
      "/wiki/Soundgarden_discography\n",
      "/wiki/Audioslave_discography\n",
      "/wiki/Chris_Cornell_(album)\n",
      "/wiki/Soundgarden\n",
      "/wiki/Audioslave\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Screaming_Trees\n",
      "/wiki/Uncle_Anesthesia\n",
      "/wiki/Euphoria_Morning\n",
      "/wiki/Carry_On_(Chris_Cornell_album)\n",
      "/wiki/Scream_(Chris_Cornell_album)\n",
      "/wiki/Songbook_(Chris_Cornell_album)\n",
      "/wiki/Higher_Truth\n",
      "/wiki/The_Roads_We_Choose_%E2%80%93_A_Retrospective\n",
      "/wiki/Chris_Cornell_(album)\n",
      "/wiki/Ultramega_OK\n",
      "/wiki/Louder_Than_Love\n",
      "/wiki/Badmotorfinger\n",
      "/wiki/Superunknown\n",
      "/wiki/Down_on_the_Upside\n",
      "/wiki/King_Animal\n",
      "/wiki/Audioslave_(album)\n",
      "/wiki/Out_of_Exile\n",
      "/wiki/Revelations_(Audioslave_album)\n",
      "/wiki/Temple_of_the_Dog_(album)\n",
      "/wiki/List_of_awards_and_nominations_received_by_Chris_Cornell\n",
      "/wiki/List_of_songs_recorded_by_Chris_Cornell\n",
      "/wiki/List_of_songs_recorded_by_Soundgarden\n",
      "/wiki/List_of_songs_recorded_by_Audioslave\n",
      "/wiki/List_of_suicides_in_the_21st_century\n",
      "/wiki/Spin_(magazine)\n",
      "/wiki/USA_Today\n",
      "/wiki/Grammy\n",
      "/wiki/Grammy\n",
      "/wiki/AllMusic\n",
      "/wiki/The_Guardian\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/Entertainment_Weekly\n",
      "/wiki/Recording_Industry_Association_of_America\n",
      "/wiki/Music_Canada\n",
      "/wiki/British_Phonographic_Industry\n",
      "/wiki/IFPI\n",
      "/wiki/NVPI\n",
      "/wiki/The_New_York_Times\n",
      "/wiki/Entertainment_Weekly\n",
      "/wiki/VH1\n",
      "/wiki/Entertainment_Weekly\n",
      "/wiki/John_Colapinto\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Guitar_World\n",
      "/wiki/Blabbermouth.net\n",
      "/wiki/Metacritic\n",
      "/wiki/NME\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/The_Seattle_Times\n",
      "/wiki/YouTube\n",
      "/wiki/Newsweek\n",
      "/wiki/Spin_(magazine)\n",
      "/wiki/SpinMedia\n",
      "/wiki/MLive\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/Seattle_Post-Intelligencer\n",
      "/wiki/Last.fm\n",
      "/wiki/Last.fm\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/AllMusic\n",
      "/wiki/The_New_York_Times\n",
      "/wiki/YouTube\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Yahoo!_Music\n",
      "/wiki/Metal_Hammer\n",
      "/wiki/San_Diego_CityBeat\n",
      "/wiki/Blabbermouth.net\n",
      "/wiki/Cleveland_Scene\n",
      "/wiki/Slant_Magazine\n",
      "/wiki/AllMusic\n",
      "/wiki/Ithaca_College\n",
      "/wiki/AOL\n",
      "/wiki/Christianity_Today\n",
      "/wiki/Virgin_Media\n",
      "/wiki/Blabbermouth.net\n",
      "/wiki/The_Age\n",
      "/wiki/MTV_News\n",
      "/wiki/YouTube\n",
      "/wiki/Blabbermouth.net\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Eldridge_Industries\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/MusicRadar\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Prometheus_Global_Media\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/YouTube\n",
      "/wiki/ISBN_(identifier)\n",
      "/wiki/YouTube\n",
      "/wiki/Rolling_Stone\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Winston-Salem_Journal\n",
      "/wiki/ISSN_(identifier)\n",
      "/wiki/YouTube\n",
      "/wiki/People_(magazine)\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Billboard_(magazine)\n",
      "/wiki/Consequence_of_Sound\n",
      "/wiki/YouTube\n",
      "/wiki/Columbus_Dispatch\n",
      "/wiki/YouTube\n",
      "/wiki/YouTube\n",
      "/wiki/AllMusic\n",
      "/wiki/AllMusic\n",
      "/wiki/IMDb\n",
      "/wiki/Chris_Cornell_discography\n",
      "/wiki/List_of_songs_recorded_by_Chris_Cornell\n",
      "/wiki/Euphoria_Morning\n",
      "/wiki/Carry_On_(Chris_Cornell_album)\n",
      "/wiki/Scream_(Chris_Cornell_album)\n",
      "/wiki/Higher_Truth\n",
      "/wiki/Songbook_(Chris_Cornell_album)\n",
      "/wiki/The_Roads_We_Choose_%E2%80%93_A_Retrospective\n",
      "/wiki/Chris_Cornell_(album)\n",
      "/wiki/Part_of_Me_Remix_EP\n",
      "/wiki/Can%27t_Change_Me\n",
      "/wiki/You_Know_My_Name\n",
      "/wiki/No_Such_Thing_(Chris_Cornell_song)\n",
      "/wiki/Arms_Around_Your_Love\n",
      "/wiki/Scream_(Chris_Cornell_song)\n",
      "/wiki/Part_of_Me_(Chris_Cornell_song)\n",
      "/wiki/Nearly_Forgot_My_Broken_Heart\n",
      "/wiki/The_Promise_(Chris_Cornell_song)\n",
      "/wiki/Heavy_Is_the_Head_(Zac_Brown_Band_song)\n",
      "/wiki/Soundgarden\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Audioslave\n",
      "/wiki/List_of_awards_and_nominations_received_by_Chris_Cornell\n",
      "/wiki/Statue_of_Chris_Cornell\n",
      "/wiki/Soundgarden\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Jason_Everman\n",
      "/wiki/Ben_Shepherd\n",
      "/wiki/Scott_Sundquist\n",
      "/wiki/Kim_Thayil\n",
      "/wiki/Hiro_Yamamoto\n",
      "/wiki/Ultramega_OK\n",
      "/wiki/Louder_Than_Love\n",
      "/wiki/Badmotorfinger\n",
      "/wiki/Superunknown\n",
      "/wiki/Down_on_the_Upside\n",
      "/wiki/King_Animal\n",
      "/wiki/Louder_Than_Live\n",
      "/wiki/Live_on_I-5\n",
      "/wiki/Screaming_Life\n",
      "/wiki/Fopp_(EP)\n",
      "/wiki/Loudest_Love\n",
      "/wiki/Badmotorfinger#Satanoscillatemymetallicsonatas\n",
      "/wiki/Songs_from_the_Superunknown\n",
      "/wiki/King_Animal_Demos\n",
      "/wiki/Screaming_Life/Fopp\n",
      "/wiki/A-Sides\n",
      "/wiki/Telephantasm\n",
      "/wiki/Louder_Than_Live\n",
      "/wiki/Motorvision\n",
      "/wiki/Hunted_Down\n",
      "/wiki/Flower_(Soundgarden_song)\n",
      "/wiki/Loud_Love\n",
      "/wiki/Hands_All_Over_(Soundgarden_song)\n",
      "/wiki/Room_a_Thousand_Years_Wide\n",
      "/wiki/Jesus_Christ_Pose\n",
      "/wiki/Outshined\n",
      "/wiki/Rusty_Cage\n",
      "/wiki/Spoonman\n",
      "/wiki/The_Day_I_Tried_to_Live\n",
      "/wiki/Black_Hole_Sun\n",
      "/wiki/My_Wave\n",
      "/wiki/Fell_on_Black_Days\n",
      "/wiki/Pretty_Noose\n",
      "/wiki/Burden_in_My_Hand\n",
      "/wiki/Blow_Up_the_Outside_World\n",
      "/wiki/Ty_Cobb_(song)\n",
      "/wiki/Bleed_Together\n",
      "/wiki/Black_Rain_(Soundgarden_song)\n",
      "/wiki/The_Telephantasm\n",
      "/wiki/Live_to_Rise\n",
      "/wiki/Been_Away_Too_Long\n",
      "/wiki/By_Crooked_Steps\n",
      "/wiki/Slaves_%26_Bulldozers\n",
      "/wiki/Rhinosaur\n",
      "/wiki/Taree_(song)\n",
      "/wiki/6_Songs_for_Bruce\n",
      "/wiki/Soundgarden_discography\n",
      "/wiki/List_of_songs_recorded_by_Soundgarden\n",
      "/wiki/List_of_Soundgarden_band_members\n",
      "/wiki/A_Sound_Garden\n",
      "/wiki/Artis_the_Spoonman\n",
      "/wiki/Susan_Silver\n",
      "/wiki/Deep_Six_(album)\n",
      "/wiki/Skin_Yard\n",
      "/wiki/Truly\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Pearl_Jam\n",
      "/wiki/Tone_Dogs\n",
      "/wiki/Hater_(band)\n",
      "/wiki/Wellwater_Conspiracy\n",
      "/wiki/The_No_WTO_Combo\n",
      "/wiki/Audioslave\n",
      "/wiki/Audioslave\n",
      "/wiki/Tom_Morello\n",
      "/wiki/Tim_Commerford\n",
      "/wiki/Brad_Wilk\n",
      "/wiki/Audioslave_(album)\n",
      "/wiki/Out_of_Exile\n",
      "/wiki/Revelations_(Audioslave_album)\n",
      "/wiki/Audioslave_(video)\n",
      "/wiki/Live_in_Cuba_(Audioslave_album)\n",
      "/wiki/Cochise_(song)\n",
      "/wiki/Like_a_Stone\n",
      "/wiki/Show_Me_How_to_Live_(song)\n",
      "/wiki/I_Am_the_Highway\n",
      "/wiki/What_You_Are\n",
      "/wiki/Be_Yourself_(Audioslave_song)\n",
      "/wiki/Your_Time_Has_Come\n",
      "/wiki/Doesn%27t_Remind_Me\n",
      "/wiki/Original_Fire\n",
      "/wiki/Revelations_(Audioslave_song)\n",
      "/wiki/Audioslave_discography\n",
      "/wiki/List_of_songs_recorded_by_Audioslave\n",
      "/wiki/Rage_Against_the_Machine\n",
      "/wiki/Soundgarden\n",
      "/wiki/Axis_of_Justice\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/WAKRAT\n",
      "/wiki/Temple_of_the_Dog\n",
      "/wiki/Jeff_Ament\n",
      "/wiki/Matt_Cameron\n",
      "/wiki/Stone_Gossard\n",
      "/wiki/Mike_McCready\n",
      "/wiki/Eddie_Vedder\n",
      "/wiki/Temple_of_the_Dog_(album)\n",
      "/wiki/Hunger_Strike_(song)\n",
      "/wiki/Say_Hello_2_Heaven\n",
      "/wiki/Pushin_Forward_Back\n",
      "/wiki/Green_River_(band)\n",
      "/wiki/Soundgarden\n",
      "/wiki/Skin_Yard\n",
      "/wiki/Bad_Radio\n",
      "/wiki/Mother_Love_Bone\n",
      "/wiki/Pearl_Jam\n",
      "/wiki/Hater_(band)\n",
      "/wiki/Audioslave\n",
      "/wiki/Bibsys\n",
      "/wiki/BNF_(identifier)\n",
      "/wiki/GND_(identifier)\n",
      "/wiki/ISNI_(identifier)\n",
      "/wiki/LCCN_(identifier)\n",
      "/wiki/MusicBrainz\n",
      "/wiki/National_Library_of_the_Czech_Republic\n",
      "/wiki/Royal_Library_of_the_Netherlands\n",
      "/wiki/VIAF_(identifier)\n",
      "/wiki/WorldCat_Identities\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "for link in soup.find(\"div\", {\"id\": \"bodyContent\"}).findAll(\"a\", href=re.compile(\"^(/wiki/)((?!:).)*$\")):\n",
    "    if 'href' in link.attrs:\n",
    "        print(link.attrs['href'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code returns the list of all the Wikipedia articles linked to our starting page. \n",
    "\n",
    "This is not enough, we want to be recursively repeat this process for all these links. That is, we need a function that takes as input a Wikipedia article URL of the form `/wiki/<Article_Name>` and returns a list of all linked articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getLinks(articleUrl):\n",
    "    page = urlopen(\"http://en.wikipedia.org\" + articleUrl)\n",
    "    soup = BeautifulSoup(page, \"lxml\")\n",
    "    links = soup.find(\"div\", {\"id\":\"bodyContent\"}).findAll(\"a\", href=re.compile(\"^(/wiki/)((?!:).)*$\"))\n",
    "    return links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test our function by calling it in a script that randomly select, for each iteration, a random link and that stops after 10 URLs have been retrieved (or when it bumps into a page without link):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/wiki/Miami_Vice_(film)\n",
      "/wiki/The_Washington_Post\n",
      "/wiki/Joan_Vennochi\n",
      "/wiki/David_Barstow\n",
      "/wiki/The_Providence_Journal\n",
      "/wiki/The_Daily_Republican\n",
      "/wiki/The_Ithaca_Journal\n",
      "/wiki/The_American_News\n",
      "/wiki/Kirksville_Daily_Express\n",
      "/wiki/Daily_newspaper\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "links = getLinks(\"/wiki/Chris_Cornell\")\n",
    "\n",
    "for _ in range(10):\n",
    "    if len(links) > 0:\n",
    "        newArticle = links[random.randint(0, len(links)-1)].attrs[\"href\"]\n",
    "        print(newArticle)\n",
    "        links = getLinks(newArticle)\n",
    "    else:\n",
    "        print(\"no links in this page\")\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1.\n",
    "\n",
    "Write code to retrieve the official address of the Internationally Ranked Universities in the Netherlands by starting from the following Wikipedia article:\n",
    "\n",
    "https://en.wikipedia.org/wiki/List_of_universities_in_the_Netherlands"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Working with APIs\n",
    "\n",
    "An **Application Programming Interface** is a set of protocols that defines how software programs communicate among eachother. Without APIs, we have to scrape the Web or get the data directly. With APIs, we often can get structured data: it is a much more convenient way to work.\n",
    "\n",
    "APIs are a great option in that they implement extensively tested routines (**high reliability**). However, you should spend time in learning how they work and, in some cases, they don't allow you to access the piece of information you may need (**low flexibility**)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Example of a Google search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"Tesla\"\n",
    "r = requests.get('https://www.google.com/search', params={'q': query})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text/html; charset=ISO-8859-1\n",
      "ISO-8859-1\n",
      "https://www.google.com/search?q=Tesla\n"
     ]
    }
   ],
   "source": [
    "print(r.headers['content-type'])\n",
    "print(r.encoding)\n",
    "print(r.url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!doctype html><html lang=\"nl\"><head><meta charset=\"UTF-8\"><meta content=\"/images/branding/googleg/1x/googleg_standard_color_128dp.png\" itemprop=\"image\"><title>Tesla - Google zoeken</title><script nonce=\"v0ZtdUW0V1jl5ahhFaacow==\">(function(){\\ndocument.documentElement.addEventListener(\"submit\",function(b){var a;if(a=b.target){var c=a.getAttribute(\"data-submitfalse\");a=\"1\"==c||\"q\"==c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);document.documentElement.addEventListener(\"click\",function(b){var a;a:{for(a=b.target;a&&a!=document.documentElement;a=a.parentElement)if(\"A\"==a.tagName){a=\"1\"==a.getAttribute(\"data-nohref\");break a}a=!1}a&&b.preventDefault()},!0);}).call(this);(function(){\\nvar a=window.performance;window.start=Date.now();a:{var b=window;if(a){var c=a.timing;if(c){var d=c.navigationStart,f=c.responseStart;if(f>d&&f<=window.start){window.start=f;b.wsrt=f-d;break a}}a.now&&(b.wsrt=Math.floor(a.now()))}}window.google=window.google||{};var h=fu'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.text[:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2.\n",
    "\n",
    "1. Inspect the Google search results page and understand how results are displayed.\n",
    "\n",
    "\n",
    "2. Use BeautifulSoup to get the link of the first 10 results of this search out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about using `requests` to query APIs? Easy using the param dictionary. Responses then follow the starndard format of the API (or you can request the one you like if available)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'{\"current_user_url\":\"https://api.github.com/user\",\"current_user_authorizations_html_url\":\"https://github.com/settings/connections/applications{/client_id}\",\"authorizations_url\":\"https://api.github.com/authorizations\",\"code_search_url\":\"https://api.github.com/search/code?q={query}{&page,per_page,sort,order}\",\"commit_search_url\":\"https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}\",\"emails_url\":\"https://api.github.com/user/emails\",\"emojis_url\":\"https://api.github.com/emojis\",\"events_url\":\"https://api.github.com/events\",\"feeds_url\":\"https://api.github.com/feeds\",\"followers_url\":\"https://api.github.com/user/followers\",\"following_url\":\"https://api.github.com/user/following{/target}\",\"gists_url\":\"https://api.github.com/gists{/gist_id}\",\"hub_url\":\"https://api.github.com/hub\",\"issue_search_url\":\"https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}\",\"issues_url\":\"https://api.github.com/issues\",\"keys_url\":\"https://api.github.com/user/keys\",\"label_search_url\":\"https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}\",\"notifications_url\":\"https://api.github.com/notifications\",\"organization_url\":\"https://api.github.com/orgs/{org}\",\"organization_repositories_url\":\"https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}\",\"organization_teams_url\":\"https://api.github.com/orgs/{org}/teams\",\"public_gists_url\":\"https://api.github.com/gists/public\",\"rate_limit_url\":\"https://api.github.com/rate_limit\",\"repository_url\":\"https://api.github.com/repos/{owner}/{repo}\",\"repository_search_url\":\"https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}\",\"current_user_repositories_url\":\"https://api.github.com/user/repos{?type,page,per_page,sort}\",\"starred_url\":\"https://api.github.com/user/starred{/owner}{/repo}\",\"starred_gists_url\":\"https://api.github.com/gists/starred\",\"user_url\":\"https://api.github.com/users/{user}\",\"user_organizations_url\":\"https://api.github.com/user/orgs\",\"user_repositories_url\":\"https://api.github.com/users/{user}/repos{?type,page,per_page,sort}\",\"user_search_url\":\"https://api.github.com/search/users?q={query}{&page,per_page,sort,order}\"}'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get('https://api.github.com')\n",
    "\n",
    "# raw\n",
    "r.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'current_user_url': 'https://api.github.com/user',\n",
       " 'current_user_authorizations_html_url': 'https://github.com/settings/connections/applications{/client_id}',\n",
       " 'authorizations_url': 'https://api.github.com/authorizations',\n",
       " 'code_search_url': 'https://api.github.com/search/code?q={query}{&page,per_page,sort,order}',\n",
       " 'commit_search_url': 'https://api.github.com/search/commits?q={query}{&page,per_page,sort,order}',\n",
       " 'emails_url': 'https://api.github.com/user/emails',\n",
       " 'emojis_url': 'https://api.github.com/emojis',\n",
       " 'events_url': 'https://api.github.com/events',\n",
       " 'feeds_url': 'https://api.github.com/feeds',\n",
       " 'followers_url': 'https://api.github.com/user/followers',\n",
       " 'following_url': 'https://api.github.com/user/following{/target}',\n",
       " 'gists_url': 'https://api.github.com/gists{/gist_id}',\n",
       " 'hub_url': 'https://api.github.com/hub',\n",
       " 'issue_search_url': 'https://api.github.com/search/issues?q={query}{&page,per_page,sort,order}',\n",
       " 'issues_url': 'https://api.github.com/issues',\n",
       " 'keys_url': 'https://api.github.com/user/keys',\n",
       " 'label_search_url': 'https://api.github.com/search/labels?q={query}&repository_id={repository_id}{&page,per_page}',\n",
       " 'notifications_url': 'https://api.github.com/notifications',\n",
       " 'organization_url': 'https://api.github.com/orgs/{org}',\n",
       " 'organization_repositories_url': 'https://api.github.com/orgs/{org}/repos{?type,page,per_page,sort}',\n",
       " 'organization_teams_url': 'https://api.github.com/orgs/{org}/teams',\n",
       " 'public_gists_url': 'https://api.github.com/gists/public',\n",
       " 'rate_limit_url': 'https://api.github.com/rate_limit',\n",
       " 'repository_url': 'https://api.github.com/repos/{owner}/{repo}',\n",
       " 'repository_search_url': 'https://api.github.com/search/repositories?q={query}{&page,per_page,sort,order}',\n",
       " 'current_user_repositories_url': 'https://api.github.com/user/repos{?type,page,per_page,sort}',\n",
       " 'starred_url': 'https://api.github.com/user/starred{/owner}{/repo}',\n",
       " 'starred_gists_url': 'https://api.github.com/gists/starred',\n",
       " 'user_url': 'https://api.github.com/users/{user}',\n",
       " 'user_organizations_url': 'https://api.github.com/user/orgs',\n",
       " 'user_repositories_url': 'https://api.github.com/users/{user}/repos{?type,page,per_page,sort}',\n",
       " 'user_search_url': 'https://api.github.com/search/users?q={query}{&page,per_page,sort,order}'}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# json\n",
    "r.json()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Twitter API (OPT)\n",
    "\n",
    "Two main APIs:\n",
    "\n",
    "* **Streaming API**: a sample of public tweets and events as they published on Twitter, provides only real-time data without limits.\n",
    "\n",
    "\n",
    "* **REST API**: allows to search, follow trends, read author profile and follower data, post / modify. It provides historical data up to a week (for the free account, more by paying), rwquires a one-time request and has rate limit (varies for different requests and subscriptions).\n",
    "\n",
    "\n",
    "REST APIs (it is a style for developing Web services which is widely used): https://en.wikipedia.org/wiki/Representational_state_transfer\n",
    "\n",
    "Some more basic info: https://developer.twitter.com/en/docs/basics/things-every-developer-should-know\n",
    "\n",
    "Tutorials: https://developer.twitter.com/en/docs/tutorials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the API: authentication\n",
    "\n",
    "**For this part, you will need credentials from the Twitter dev website.**\n",
    "\n",
    "A good way to store your keys is using `.conf` files and `configparser`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['stuff/conf.conf']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import configparser\n",
    "config = configparser.ConfigParser()\n",
    "config.read(\"stuff/conf.conf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hBn3fPoa7TXlL4fEEbZ5l1cbd'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config['twitter']['api_key']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is how my `conf.conf` file looks like (also in `stuff/conf_public.conf`):\n",
    "\n",
    "```\n",
    "[twitter]\n",
    "api_key = YOURS\n",
    "api_secret_key = YOURS\n",
    "access_token = YOURS\n",
    "access_secret_token = YOURS\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A useful package: Tweepy\n",
    "\n",
    "https://tweepy.readthedocs.io/en/latest/index.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tweepy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Great work by our friends @scite https://t.co/TtVZcQoWSL\n",
      "RT @mjczies: Good morning and happy weekend! @Eckyo8 inspired me this morning! If we want to stand tall, we must grow and thrive even throu…\n",
      "RT @SoManyBooks24: “The novel’s polyphony builds into a panoramic picture of late 13th-century England, divided by wealth and status but su…\n",
      "In dit artikel vatten we het belangrijkste coronanieuws van de afgelopen uren voor je samen.\n",
      "https://t.co/5M6d8qlLVp\n",
      "RT @Nisa00: Still a few days to submit nominations to EASST Amsterdamska, Freeman and Ziman prizes! @STSeasst Full information at https://t…\n"
     ]
    }
   ],
   "source": [
    "# Tweepy Hello World\n",
    "\n",
    "# authentication (OAuth)\n",
    "auth = tweepy.OAuthHandler(config['twitter']['api_key'], config['twitter']['api_secret_key'])\n",
    "auth.set_access_token(config['twitter']['access_token'], config['twitter']['access_secret_token'])\n",
    "\n",
    "api = tweepy.API(auth)\n",
    "\n",
    "public_tweets = api.home_timeline()\n",
    "for tweet in public_tweets[:5]:\n",
    "    print(tweet.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Interlude: JSON\n",
    "\n",
    "The Twitter API returns data structured in the JSON format. [JSON](https://www.json.org) (JavaScript Object Notation) is a lightweight data-interchange format. It is easy for humans to read and write. It is easy for machines to parse and generate. **It is basically a list of nested Python dictionaries.**\n",
    "\n",
    "\n",
    "Minimal example:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"firstName\": \"John\",\n",
    "  \"lastName\": \"Doe\",\n",
    "  \"age\": 21\n",
    "}\n",
    "```\n",
    "\n",
    "Extended example:\n",
    "\n",
    "```json\n",
    "{\n",
    "  \"$id\": \"https://example.com/person.schema.json\",\n",
    "  \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n",
    "  \"title\": \"Person\",\n",
    "  \"type\": \"object\",\n",
    "  \"properties\": {\n",
    "    \"firstName\": {\n",
    "      \"type\": \"string\",\n",
    "      \"description\": \"The person's first name.\"\n",
    "    },\n",
    "    \"lastName\": {\n",
    "      \"type\": \"string\",\n",
    "      \"description\": \"The person's last name.\"\n",
    "    },\n",
    "    \"age\": {\n",
    "      \"description\": \"Age in years which must be equal to or greater than zero.\",\n",
    "      \"type\": \"integer\",\n",
    "      \"minimum\": 0\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "\n",
    "Online viewer: http://jsonviewer.stack.hu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the API: search\n",
    "\n",
    "All the most recent Tweets from a given hashtag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'created_at': 'Sat Jun 06 08:49:10 +0000 2020', 'id': 1269189578423820288, 'id_str': '1269189578423820288', 'text': 'RT @Takuma_Kato51: #acl2020nlp の併設ワークショップ(SRW)に採択された論文を公開しました！\\nhttps://t.co/aEVnBqij7N\\nNERデータセットのラベルを複数の要素に分解(例：B-person -&gt; B, Person)し，各要素…', 'truncated': False, 'entities': {'hashtags': [{'text': 'acl2020nlp', 'indices': [19, 30]}], 'symbols': [], 'user_mentions': [{'screen_name': 'Takuma_Kato51', 'name': 'Takuma Kato', 'id': 1267657369200128000, 'id_str': '1267657369200128000', 'indices': [3, 17]}], 'urls': [{'url': 'https://t.co/aEVnBqij7N', 'expanded_url': 'https://arxiv.org/abs/2006.01372v2', 'display_url': 'arxiv.org/abs/2006.01372…', 'indices': [63, 86]}]}, 'metadata': {'iso_language_code': 'ja', 'result_type': 'recent'}, 'source': '<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 1256161151009751040, 'id_str': '1256161151009751040', 'name': 'Jun Suzuki', 'screen_name': 'drJunSuzuki', 'location': '', 'description': '', 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 27, 'friends_count': 23, 'listed_count': 1, 'created_at': 'Fri May 01 09:59:34 +0000 2020', 'favourites_count': 20, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 27, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'F5F8FA', 'profile_background_image_url': None, 'profile_background_image_url_https': None, 'profile_background_tile': False, 'profile_image_url': 'http://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png', 'profile_image_url_https': 'https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png', 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': True, 'default_profile_image': True, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'retweeted_status': {'created_at': 'Fri Jun 05 09:55:47 +0000 2020', 'id': 1268843956437020672, 'id_str': '1268843956437020672', 'text': '#acl2020nlp の併設ワークショップ(SRW)に採択された論文を公開しました！\\nhttps://t.co/aEVnBqij7N\\nNERデータセットのラベルを複数の要素に分解(例：B-person -&gt; B, Person)… https://t.co/BxeX3XvvrH', 'truncated': True, 'entities': {'hashtags': [{'text': 'acl2020nlp', 'indices': [0, 11]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/aEVnBqij7N', 'expanded_url': 'https://arxiv.org/abs/2006.01372v2', 'display_url': 'arxiv.org/abs/2006.01372…', 'indices': [44, 67]}, {'url': 'https://t.co/BxeX3XvvrH', 'expanded_url': 'https://twitter.com/i/web/status/1268843956437020672', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [120, 143]}]}, 'metadata': {'iso_language_code': 'ja', 'result_type': 'recent'}, 'source': '<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 1267657369200128000, 'id_str': '1267657369200128000', 'name': 'Takuma Kato', 'screen_name': 'Takuma_Kato51', 'location': '', 'description': \"Master's student at Tohoku University (Tohoku NLP lab. @NlpTohoku) #NLProc\", 'url': None, 'entities': {'description': {'urls': []}}, 'protected': False, 'followers_count': 59, 'friends_count': 69, 'listed_count': 1, 'created_at': 'Tue Jun 02 03:20:53 +0000 2020', 'favourites_count': 0, 'utc_offset': None, 'time_zone': None, 'geo_enabled': False, 'verified': False, 'statuses_count': 2, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': 'F5F8FA', 'profile_background_image_url': None, 'profile_background_image_url_https': None, 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/1267661999695003649/_3hBGtJ1_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/1267661999695003649/_3hBGtJ1_normal.jpg', 'profile_link_color': '1DA1F2', 'profile_sidebar_border_color': 'C0DEED', 'profile_sidebar_fill_color': 'DDEEF6', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': True, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 4, 'favorite_count': 30, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'ja'}, 'is_quote_status': False, 'retweet_count': 4, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'ja'}\n",
      "{'created_at': 'Sat Jun 06 08:45:56 +0000 2020', 'id': 1269188764712099840, 'id_str': '1269188764712099840', 'text': 'RT @gp_pulipaka: Top Colab Notebooks to Kickstart Your #MachineLearning Journey. #BigData #Analytics #DataScience #AI #IoT #IIoT #NLProc #P…', 'truncated': False, 'entities': {'hashtags': [{'text': 'MachineLearning', 'indices': [55, 71]}, {'text': 'BigData', 'indices': [81, 89]}, {'text': 'Analytics', 'indices': [90, 100]}, {'text': 'DataScience', 'indices': [101, 113]}, {'text': 'AI', 'indices': [114, 117]}, {'text': 'IoT', 'indices': [118, 122]}, {'text': 'IIoT', 'indices': [123, 128]}, {'text': 'NLProc', 'indices': [129, 136]}], 'symbols': [], 'user_mentions': [{'screen_name': 'gp_pulipaka', 'name': 'Dr. Ganapathi Pulipaka', 'id': 4263007693, 'id_str': '4263007693', 'indices': [3, 15]}], 'urls': []}, 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'}, 'source': '<a href=\"http://twitter.com/download/android\" rel=\"nofollow\">Twitter for Android</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 352584360, 'id_str': '352584360', 'name': '⭐Irsyad MFA⭐', 'screen_name': 'ibnumfa', 'location': 'SELANGOR', 'description': '~justice~\\n\\n🇺🇸🇲🇾🇲🇨🇸🇦🇳🇿🇵🇭🇸🇬🇹🇷🇦🇪🏴\\U000e0067\\U000e0062\\U000e0065\\U000e006e\\U000e0067\\U000e007f🏴\\U000e0067\\U000e0062\\U000e0073\\U000e0063\\U000e0074\\U000e007f🇪🇸🇫🇷🇧🇪🇳🇱🇩🇪🇨🇭🇹🇭', 'url': 'https://t.co/OfYGTPIHEK', 'entities': {'url': {'urls': [{'url': 'https://t.co/OfYGTPIHEK', 'expanded_url': 'http://www.linkedin.com/pub/iqmal-irsyad-mohammad-fuad/43/b3/816', 'display_url': 'linkedin.com/pub/iqmal-irsy…', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 312, 'friends_count': 213, 'listed_count': 8, 'created_at': 'Wed Aug 10 20:39:38 +0000 2011', 'favourites_count': 7862, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 7596, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '131516', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme14/bg.gif', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme14/bg.gif', 'profile_background_tile': True, 'profile_image_url': 'http://pbs.twimg.com/profile_images/935888958923325440/6iFy-zwX_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/935888958923325440/6iFy-zwX_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/352584360/1431691812', 'profile_link_color': '009999', 'profile_sidebar_border_color': 'EEEEEE', 'profile_sidebar_fill_color': 'EFEFEF', 'profile_text_color': '333333', 'profile_use_background_image': True, 'has_extended_profile': False, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'retweeted_status': {'created_at': 'Sat Jun 06 00:26:03 +0000 2020', 'id': 1269062964650348545, 'id_str': '1269062964650348545', 'text': 'Top Colab Notebooks to Kickstart Your #MachineLearning Journey. #BigData #Analytics #DataScience #AI #IoT #IIoT… https://t.co/YpN7NyUZKo', 'truncated': True, 'entities': {'hashtags': [{'text': 'MachineLearning', 'indices': [38, 54]}, {'text': 'BigData', 'indices': [64, 72]}, {'text': 'Analytics', 'indices': [73, 83]}, {'text': 'DataScience', 'indices': [84, 96]}, {'text': 'AI', 'indices': [97, 100]}, {'text': 'IoT', 'indices': [101, 105]}, {'text': 'IIoT', 'indices': [106, 111]}], 'symbols': [], 'user_mentions': [], 'urls': [{'url': 'https://t.co/YpN7NyUZKo', 'expanded_url': 'https://twitter.com/i/web/status/1269062964650348545', 'display_url': 'twitter.com/i/web/status/1…', 'indices': [113, 136]}]}, 'metadata': {'iso_language_code': 'en', 'result_type': 'recent'}, 'source': '<a href=\"https://buffer.com\" rel=\"nofollow\">Buffer</a>', 'in_reply_to_status_id': None, 'in_reply_to_status_id_str': None, 'in_reply_to_user_id': None, 'in_reply_to_user_id_str': None, 'in_reply_to_screen_name': None, 'user': {'id': 4263007693, 'id_str': '4263007693', 'name': 'Dr. Ganapathi Pulipaka', 'screen_name': 'gp_pulipaka', 'location': 'Redondo Beach, CA', 'description': 'Global Premier Speaker | PostDoc CS, PhD | Bestselling Author | 18.4 M Views Per Month | Top Data Science, IIoT Influencer @Onalytica', 'url': 'https://t.co/eYyaJsEcfw', 'entities': {'url': {'urls': [{'url': 'https://t.co/eYyaJsEcfw', 'expanded_url': 'http://www.gppulipaka.org', 'display_url': 'gppulipaka.org', 'indices': [0, 23]}]}, 'description': {'urls': []}}, 'protected': False, 'followers_count': 89881, 'friends_count': 26528, 'listed_count': 3420, 'created_at': 'Tue Nov 17 02:19:09 +0000 2015', 'favourites_count': 9, 'utc_offset': None, 'time_zone': None, 'geo_enabled': True, 'verified': False, 'statuses_count': 56196, 'lang': None, 'contributors_enabled': False, 'is_translator': False, 'is_translation_enabled': False, 'profile_background_color': '000000', 'profile_background_image_url': 'http://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_image_url_https': 'https://abs.twimg.com/images/themes/theme1/bg.png', 'profile_background_tile': False, 'profile_image_url': 'http://pbs.twimg.com/profile_images/998301258413850625/9BZwTjgv_normal.jpg', 'profile_image_url_https': 'https://pbs.twimg.com/profile_images/998301258413850625/9BZwTjgv_normal.jpg', 'profile_banner_url': 'https://pbs.twimg.com/profile_banners/4263007693/1583526136', 'profile_link_color': 'DD2E44', 'profile_sidebar_border_color': '000000', 'profile_sidebar_fill_color': '000000', 'profile_text_color': '000000', 'profile_use_background_image': False, 'has_extended_profile': True, 'default_profile': False, 'default_profile_image': False, 'following': False, 'follow_request_sent': False, 'notifications': False, 'translator_type': 'none'}, 'geo': None, 'coordinates': None, 'place': None, 'contributors': None, 'is_quote_status': False, 'retweet_count': 49, 'favorite_count': 50, 'favorited': False, 'retweeted': False, 'possibly_sensitive': False, 'lang': 'en'}, 'is_quote_status': False, 'retweet_count': 49, 'favorite_count': 0, 'favorited': False, 'retweeted': False, 'lang': 'en'}\n"
     ]
    }
   ],
   "source": [
    "# queries\n",
    "\n",
    "tweets = tweepy.Cursor(api.search, q=\"#nlproc\")\n",
    "\n",
    "for item in tweets.items(2):\n",
    "    print(item._json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"created_at\": \"Sat Jun 06 08:49:10 +0000 2020\",\n",
      "    \"id\": 1269189578423820288,\n",
      "    \"id_str\": \"1269189578423820288\",\n",
      "    \"text\": \"RT @Takuma_Kato51: #acl2020nlp \\u306e\\u4f75\\u8a2d\\u30ef\\u30fc\\u30af\\u30b7\\u30e7\\u30c3\\u30d7(SRW)\\u306b\\u63a1\\u629e\\u3055\\u308c\\u305f\\u8ad6\\u6587\\u3092\\u516c\\u958b\\u3057\\u307e\\u3057\\u305f\\uff01\\nhttps://t.co/aEVnBqij7N\\nNER\\u30c7\\u30fc\\u30bf\\u30bb\\u30c3\\u30c8\\u306e\\u30e9\\u30d9\\u30eb\\u3092\\u8907\\u6570\\u306e\\u8981\\u7d20\\u306b\\u5206\\u89e3(\\u4f8b\\uff1aB-person -&gt; B, Person)\\u3057\\uff0c\\u5404\\u8981\\u7d20\\u2026\",\n",
      "    \"truncated\": false,\n",
      "    \"entities\": {\n",
      "        \"hashtags\": [\n",
      "            {\n",
      "                \"text\": \"acl2020nlp\",\n",
      "                \"indices\": [\n",
      "                    19,\n",
      "                    30\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"symbols\": [],\n",
      "        \"user_mentions\": [\n",
      "            {\n",
      "                \"screen_name\": \"Takuma_Kato51\",\n",
      "                \"name\": \"Takuma Kato\",\n",
      "                \"id\": 1267657369200128000,\n",
      "                \"id_str\": \"1267657369200128000\",\n",
      "                \"indices\": [\n",
      "                    3,\n",
      "                    17\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"urls\": [\n",
      "            {\n",
      "                \"url\": \"https://t.co/aEVnBqij7N\",\n",
      "                \"expanded_url\": \"https://arxiv.org/abs/2006.01372v2\",\n",
      "                \"display_url\": \"arxiv.org/abs/2006.01372\\u2026\",\n",
      "                \"indices\": [\n",
      "                    63,\n",
      "                    86\n",
      "                ]\n",
      "            }\n",
      "        ]\n",
      "    },\n",
      "    \"metadata\": {\n",
      "        \"iso_language_code\": \"ja\",\n",
      "        \"result_type\": \"recent\"\n",
      "    },\n",
      "    \"source\": \"<a href=\\\"https://mobile.twitter.com\\\" rel=\\\"nofollow\\\">Twitter Web App</a>\",\n",
      "    \"in_reply_to_status_id\": null,\n",
      "    \"in_reply_to_status_id_str\": null,\n",
      "    \"in_reply_to_user_id\": null,\n",
      "    \"in_reply_to_user_id_str\": null,\n",
      "    \"in_reply_to_screen_name\": null,\n",
      "    \"user\": {\n",
      "        \"id\": 1256161151009751040,\n",
      "        \"id_str\": \"1256161151009751040\",\n",
      "        \"name\": \"Jun Suzuki\",\n",
      "        \"screen_name\": \"drJunSuzuki\",\n",
      "        \"location\": \"\",\n",
      "        \"description\": \"\",\n",
      "        \"url\": null,\n",
      "        \"entities\": {\n",
      "            \"description\": {\n",
      "                \"urls\": []\n",
      "            }\n",
      "        },\n",
      "        \"protected\": false,\n",
      "        \"followers_count\": 27,\n",
      "        \"friends_count\": 23,\n",
      "        \"listed_count\": 1,\n",
      "        \"created_at\": \"Fri May 01 09:59:34 +0000 2020\",\n",
      "        \"favourites_count\": 20,\n",
      "        \"utc_offset\": null,\n",
      "        \"time_zone\": null,\n",
      "        \"geo_enabled\": false,\n",
      "        \"verified\": false,\n",
      "        \"statuses_count\": 27,\n",
      "        \"lang\": null,\n",
      "        \"contributors_enabled\": false,\n",
      "        \"is_translator\": false,\n",
      "        \"is_translation_enabled\": false,\n",
      "        \"profile_background_color\": \"F5F8FA\",\n",
      "        \"profile_background_image_url\": null,\n",
      "        \"profile_background_image_url_https\": null,\n",
      "        \"profile_background_tile\": false,\n",
      "        \"profile_image_url\": \"http://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png\",\n",
      "        \"profile_image_url_https\": \"https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png\",\n",
      "        \"profile_link_color\": \"1DA1F2\",\n",
      "        \"profile_sidebar_border_color\": \"C0DEED\",\n",
      "        \"profile_sidebar_fill_color\": \"DDEEF6\",\n",
      "        \"profile_text_color\": \"333333\",\n",
      "        \"profile_use_background_image\": true,\n",
      "        \"has_extended_profile\": false,\n",
      "        \"default_profile\": true,\n",
      "        \"default_profile_image\": true,\n",
      "        \"following\": false,\n",
      "        \"follow_request_sent\": false,\n",
      "        \"notifications\": false,\n",
      "        \"translator_type\": \"none\"\n",
      "    },\n",
      "    \"geo\": null,\n",
      "    \"coordinates\": null,\n",
      "    \"place\": null,\n",
      "    \"contributors\": null,\n",
      "    \"retweeted_status\": {\n",
      "        \"created_at\": \"Fri Jun 05 09:55:47 +0000 2020\",\n",
      "        \"id\": 1268843956437020672,\n",
      "        \"id_str\": \"1268843956437020672\",\n",
      "        \"text\": \"#acl2020nlp \\u306e\\u4f75\\u8a2d\\u30ef\\u30fc\\u30af\\u30b7\\u30e7\\u30c3\\u30d7(SRW)\\u306b\\u63a1\\u629e\\u3055\\u308c\\u305f\\u8ad6\\u6587\\u3092\\u516c\\u958b\\u3057\\u307e\\u3057\\u305f\\uff01\\nhttps://t.co/aEVnBqij7N\\nNER\\u30c7\\u30fc\\u30bf\\u30bb\\u30c3\\u30c8\\u306e\\u30e9\\u30d9\\u30eb\\u3092\\u8907\\u6570\\u306e\\u8981\\u7d20\\u306b\\u5206\\u89e3(\\u4f8b\\uff1aB-person -&gt; B, Person)\\u2026 https://t.co/BxeX3XvvrH\",\n",
      "        \"truncated\": true,\n",
      "        \"entities\": {\n",
      "            \"hashtags\": [\n",
      "                {\n",
      "                    \"text\": \"acl2020nlp\",\n",
      "                    \"indices\": [\n",
      "                        0,\n",
      "                        11\n",
      "                    ]\n",
      "                }\n",
      "            ],\n",
      "            \"symbols\": [],\n",
      "            \"user_mentions\": [],\n",
      "            \"urls\": [\n",
      "                {\n",
      "                    \"url\": \"https://t.co/aEVnBqij7N\",\n",
      "                    \"expanded_url\": \"https://arxiv.org/abs/2006.01372v2\",\n",
      "                    \"display_url\": \"arxiv.org/abs/2006.01372\\u2026\",\n",
      "                    \"indices\": [\n",
      "                        44,\n",
      "                        67\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"url\": \"https://t.co/BxeX3XvvrH\",\n",
      "                    \"expanded_url\": \"https://twitter.com/i/web/status/1268843956437020672\",\n",
      "                    \"display_url\": \"twitter.com/i/web/status/1\\u2026\",\n",
      "                    \"indices\": [\n",
      "                        120,\n",
      "                        143\n",
      "                    ]\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"metadata\": {\n",
      "            \"iso_language_code\": \"ja\",\n",
      "            \"result_type\": \"recent\"\n",
      "        },\n",
      "        \"source\": \"<a href=\\\"https://mobile.twitter.com\\\" rel=\\\"nofollow\\\">Twitter Web App</a>\",\n",
      "        \"in_reply_to_status_id\": null,\n",
      "        \"in_reply_to_status_id_str\": null,\n",
      "        \"in_reply_to_user_id\": null,\n",
      "        \"in_reply_to_user_id_str\": null,\n",
      "        \"in_reply_to_screen_name\": null,\n",
      "        \"user\": {\n",
      "            \"id\": 1267657369200128000,\n",
      "            \"id_str\": \"1267657369200128000\",\n",
      "            \"name\": \"Takuma Kato\",\n",
      "            \"screen_name\": \"Takuma_Kato51\",\n",
      "            \"location\": \"\",\n",
      "            \"description\": \"Master's student at Tohoku University (Tohoku NLP lab. @NlpTohoku) #NLProc\",\n",
      "            \"url\": null,\n",
      "            \"entities\": {\n",
      "                \"description\": {\n",
      "                    \"urls\": []\n",
      "                }\n",
      "            },\n",
      "            \"protected\": false,\n",
      "            \"followers_count\": 59,\n",
      "            \"friends_count\": 69,\n",
      "            \"listed_count\": 1,\n",
      "            \"created_at\": \"Tue Jun 02 03:20:53 +0000 2020\",\n",
      "            \"favourites_count\": 0,\n",
      "            \"utc_offset\": null,\n",
      "            \"time_zone\": null,\n",
      "            \"geo_enabled\": false,\n",
      "            \"verified\": false,\n",
      "            \"statuses_count\": 2,\n",
      "            \"lang\": null,\n",
      "            \"contributors_enabled\": false,\n",
      "            \"is_translator\": false,\n",
      "            \"is_translation_enabled\": false,\n",
      "            \"profile_background_color\": \"F5F8FA\",\n",
      "            \"profile_background_image_url\": null,\n",
      "            \"profile_background_image_url_https\": null,\n",
      "            \"profile_background_tile\": false,\n",
      "            \"profile_image_url\": \"http://pbs.twimg.com/profile_images/1267661999695003649/_3hBGtJ1_normal.jpg\",\n",
      "            \"profile_image_url_https\": \"https://pbs.twimg.com/profile_images/1267661999695003649/_3hBGtJ1_normal.jpg\",\n",
      "            \"profile_link_color\": \"1DA1F2\",\n",
      "            \"profile_sidebar_border_color\": \"C0DEED\",\n",
      "            \"profile_sidebar_fill_color\": \"DDEEF6\",\n",
      "            \"profile_text_color\": \"333333\",\n",
      "            \"profile_use_background_image\": true,\n",
      "            \"has_extended_profile\": false,\n",
      "            \"default_profile\": true,\n",
      "            \"default_profile_image\": false,\n",
      "            \"following\": false,\n",
      "            \"follow_request_sent\": false,\n",
      "            \"notifications\": false,\n",
      "            \"translator_type\": \"none\"\n",
      "        },\n",
      "        \"geo\": null,\n",
      "        \"coordinates\": null,\n",
      "        \"place\": null,\n",
      "        \"contributors\": null,\n",
      "        \"is_quote_status\": false,\n",
      "        \"retweet_count\": 4,\n",
      "        \"favorite_count\": 30,\n",
      "        \"favorited\": false,\n",
      "        \"retweeted\": false,\n",
      "        \"possibly_sensitive\": false,\n",
      "        \"lang\": \"ja\"\n",
      "    },\n",
      "    \"is_quote_status\": false,\n",
      "    \"retweet_count\": 4,\n",
      "    \"favorite_count\": 0,\n",
      "    \"favorited\": false,\n",
      "    \"retweeted\": false,\n",
      "    \"possibly_sensitive\": false,\n",
      "    \"lang\": \"ja\"\n",
      "}\n",
      "{\n",
      "    \"created_at\": \"Sat Jun 06 08:45:56 +0000 2020\",\n",
      "    \"id\": 1269188764712099840,\n",
      "    \"id_str\": \"1269188764712099840\",\n",
      "    \"text\": \"RT @gp_pulipaka: Top Colab Notebooks to Kickstart Your #MachineLearning Journey. #BigData #Analytics #DataScience #AI #IoT #IIoT #NLProc #P\\u2026\",\n",
      "    \"truncated\": false,\n",
      "    \"entities\": {\n",
      "        \"hashtags\": [\n",
      "            {\n",
      "                \"text\": \"MachineLearning\",\n",
      "                \"indices\": [\n",
      "                    55,\n",
      "                    71\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"BigData\",\n",
      "                \"indices\": [\n",
      "                    81,\n",
      "                    89\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"Analytics\",\n",
      "                \"indices\": [\n",
      "                    90,\n",
      "                    100\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"DataScience\",\n",
      "                \"indices\": [\n",
      "                    101,\n",
      "                    113\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"AI\",\n",
      "                \"indices\": [\n",
      "                    114,\n",
      "                    117\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"IoT\",\n",
      "                \"indices\": [\n",
      "                    118,\n",
      "                    122\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"IIoT\",\n",
      "                \"indices\": [\n",
      "                    123,\n",
      "                    128\n",
      "                ]\n",
      "            },\n",
      "            {\n",
      "                \"text\": \"NLProc\",\n",
      "                \"indices\": [\n",
      "                    129,\n",
      "                    136\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"symbols\": [],\n",
      "        \"user_mentions\": [\n",
      "            {\n",
      "                \"screen_name\": \"gp_pulipaka\",\n",
      "                \"name\": \"Dr. Ganapathi Pulipaka\",\n",
      "                \"id\": 4263007693,\n",
      "                \"id_str\": \"4263007693\",\n",
      "                \"indices\": [\n",
      "                    3,\n",
      "                    15\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"urls\": []\n",
      "    },\n",
      "    \"metadata\": {\n",
      "        \"iso_language_code\": \"en\",\n",
      "        \"result_type\": \"recent\"\n",
      "    },\n",
      "    \"source\": \"<a href=\\\"http://twitter.com/download/android\\\" rel=\\\"nofollow\\\">Twitter for Android</a>\",\n",
      "    \"in_reply_to_status_id\": null,\n",
      "    \"in_reply_to_status_id_str\": null,\n",
      "    \"in_reply_to_user_id\": null,\n",
      "    \"in_reply_to_user_id_str\": null,\n",
      "    \"in_reply_to_screen_name\": null,\n",
      "    \"user\": {\n",
      "        \"id\": 352584360,\n",
      "        \"id_str\": \"352584360\",\n",
      "        \"name\": \"\\u2b50Irsyad MFA\\u2b50\",\n",
      "        \"screen_name\": \"ibnumfa\",\n",
      "        \"location\": \"SELANGOR\",\n",
      "        \"description\": \"~justice~\\n\\n\\ud83c\\uddfa\\ud83c\\uddf8\\ud83c\\uddf2\\ud83c\\uddfe\\ud83c\\uddf2\\ud83c\\udde8\\ud83c\\uddf8\\ud83c\\udde6\\ud83c\\uddf3\\ud83c\\uddff\\ud83c\\uddf5\\ud83c\\udded\\ud83c\\uddf8\\ud83c\\uddec\\ud83c\\uddf9\\ud83c\\uddf7\\ud83c\\udde6\\ud83c\\uddea\\ud83c\\udff4\\udb40\\udc67\\udb40\\udc62\\udb40\\udc65\\udb40\\udc6e\\udb40\\udc67\\udb40\\udc7f\\ud83c\\udff4\\udb40\\udc67\\udb40\\udc62\\udb40\\udc73\\udb40\\udc63\\udb40\\udc74\\udb40\\udc7f\\ud83c\\uddea\\ud83c\\uddf8\\ud83c\\uddeb\\ud83c\\uddf7\\ud83c\\udde7\\ud83c\\uddea\\ud83c\\uddf3\\ud83c\\uddf1\\ud83c\\udde9\\ud83c\\uddea\\ud83c\\udde8\\ud83c\\udded\\ud83c\\uddf9\\ud83c\\udded\",\n",
      "        \"url\": \"https://t.co/OfYGTPIHEK\",\n",
      "        \"entities\": {\n",
      "            \"url\": {\n",
      "                \"urls\": [\n",
      "                    {\n",
      "                        \"url\": \"https://t.co/OfYGTPIHEK\",\n",
      "                        \"expanded_url\": \"http://www.linkedin.com/pub/iqmal-irsyad-mohammad-fuad/43/b3/816\",\n",
      "                        \"display_url\": \"linkedin.com/pub/iqmal-irsy\\u2026\",\n",
      "                        \"indices\": [\n",
      "                            0,\n",
      "                            23\n",
      "                        ]\n",
      "                    }\n",
      "                ]\n",
      "            },\n",
      "            \"description\": {\n",
      "                \"urls\": []\n",
      "            }\n",
      "        },\n",
      "        \"protected\": false,\n",
      "        \"followers_count\": 312,\n",
      "        \"friends_count\": 213,\n",
      "        \"listed_count\": 8,\n",
      "        \"created_at\": \"Wed Aug 10 20:39:38 +0000 2011\",\n",
      "        \"favourites_count\": 7862,\n",
      "        \"utc_offset\": null,\n",
      "        \"time_zone\": null,\n",
      "        \"geo_enabled\": true,\n",
      "        \"verified\": false,\n",
      "        \"statuses_count\": 7596,\n",
      "        \"lang\": null,\n",
      "        \"contributors_enabled\": false,\n",
      "        \"is_translator\": false,\n",
      "        \"is_translation_enabled\": false,\n",
      "        \"profile_background_color\": \"131516\",\n",
      "        \"profile_background_image_url\": \"http://abs.twimg.com/images/themes/theme14/bg.gif\",\n",
      "        \"profile_background_image_url_https\": \"https://abs.twimg.com/images/themes/theme14/bg.gif\",\n",
      "        \"profile_background_tile\": true,\n",
      "        \"profile_image_url\": \"http://pbs.twimg.com/profile_images/935888958923325440/6iFy-zwX_normal.jpg\",\n",
      "        \"profile_image_url_https\": \"https://pbs.twimg.com/profile_images/935888958923325440/6iFy-zwX_normal.jpg\",\n",
      "        \"profile_banner_url\": \"https://pbs.twimg.com/profile_banners/352584360/1431691812\",\n",
      "        \"profile_link_color\": \"009999\",\n",
      "        \"profile_sidebar_border_color\": \"EEEEEE\",\n",
      "        \"profile_sidebar_fill_color\": \"EFEFEF\",\n",
      "        \"profile_text_color\": \"333333\",\n",
      "        \"profile_use_background_image\": true,\n",
      "        \"has_extended_profile\": false,\n",
      "        \"default_profile\": false,\n",
      "        \"default_profile_image\": false,\n",
      "        \"following\": false,\n",
      "        \"follow_request_sent\": false,\n",
      "        \"notifications\": false,\n",
      "        \"translator_type\": \"none\"\n",
      "    },\n",
      "    \"geo\": null,\n",
      "    \"coordinates\": null,\n",
      "    \"place\": null,\n",
      "    \"contributors\": null,\n",
      "    \"retweeted_status\": {\n",
      "        \"created_at\": \"Sat Jun 06 00:26:03 +0000 2020\",\n",
      "        \"id\": 1269062964650348545,\n",
      "        \"id_str\": \"1269062964650348545\",\n",
      "        \"text\": \"Top Colab Notebooks to Kickstart Your #MachineLearning Journey. #BigData #Analytics #DataScience #AI #IoT #IIoT\\u2026 https://t.co/YpN7NyUZKo\",\n",
      "        \"truncated\": true,\n",
      "        \"entities\": {\n",
      "            \"hashtags\": [\n",
      "                {\n",
      "                    \"text\": \"MachineLearning\",\n",
      "                    \"indices\": [\n",
      "                        38,\n",
      "                        54\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"BigData\",\n",
      "                    \"indices\": [\n",
      "                        64,\n",
      "                        72\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"Analytics\",\n",
      "                    \"indices\": [\n",
      "                        73,\n",
      "                        83\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"DataScience\",\n",
      "                    \"indices\": [\n",
      "                        84,\n",
      "                        96\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"AI\",\n",
      "                    \"indices\": [\n",
      "                        97,\n",
      "                        100\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"IoT\",\n",
      "                    \"indices\": [\n",
      "                        101,\n",
      "                        105\n",
      "                    ]\n",
      "                },\n",
      "                {\n",
      "                    \"text\": \"IIoT\",\n",
      "                    \"indices\": [\n",
      "                        106,\n",
      "                        111\n",
      "                    ]\n",
      "                }\n",
      "            ],\n",
      "            \"symbols\": [],\n",
      "            \"user_mentions\": [],\n",
      "            \"urls\": [\n",
      "                {\n",
      "                    \"url\": \"https://t.co/YpN7NyUZKo\",\n",
      "                    \"expanded_url\": \"https://twitter.com/i/web/status/1269062964650348545\",\n",
      "                    \"display_url\": \"twitter.com/i/web/status/1\\u2026\",\n",
      "                    \"indices\": [\n",
      "                        113,\n",
      "                        136\n",
      "                    ]\n",
      "                }\n",
      "            ]\n",
      "        },\n",
      "        \"metadata\": {\n",
      "            \"iso_language_code\": \"en\",\n",
      "            \"result_type\": \"recent\"\n",
      "        },\n",
      "        \"source\": \"<a href=\\\"https://buffer.com\\\" rel=\\\"nofollow\\\">Buffer</a>\",\n",
      "        \"in_reply_to_status_id\": null,\n",
      "        \"in_reply_to_status_id_str\": null,\n",
      "        \"in_reply_to_user_id\": null,\n",
      "        \"in_reply_to_user_id_str\": null,\n",
      "        \"in_reply_to_screen_name\": null,\n",
      "        \"user\": {\n",
      "            \"id\": 4263007693,\n",
      "            \"id_str\": \"4263007693\",\n",
      "            \"name\": \"Dr. Ganapathi Pulipaka\",\n",
      "            \"screen_name\": \"gp_pulipaka\",\n",
      "            \"location\": \"Redondo Beach, CA\",\n",
      "            \"description\": \"Global Premier Speaker | PostDoc CS, PhD | Bestselling Author | 18.4 M Views Per Month | Top Data Science, IIoT Influencer @Onalytica\",\n",
      "            \"url\": \"https://t.co/eYyaJsEcfw\",\n",
      "            \"entities\": {\n",
      "                \"url\": {\n",
      "                    \"urls\": [\n",
      "                        {\n",
      "                            \"url\": \"https://t.co/eYyaJsEcfw\",\n",
      "                            \"expanded_url\": \"http://www.gppulipaka.org\",\n",
      "                            \"display_url\": \"gppulipaka.org\",\n",
      "                            \"indices\": [\n",
      "                                0,\n",
      "                                23\n",
      "                            ]\n",
      "                        }\n",
      "                    ]\n",
      "                },\n",
      "                \"description\": {\n",
      "                    \"urls\": []\n",
      "                }\n",
      "            },\n",
      "            \"protected\": false,\n",
      "            \"followers_count\": 89881,\n",
      "            \"friends_count\": 26528,\n",
      "            \"listed_count\": 3420,\n",
      "            \"created_at\": \"Tue Nov 17 02:19:09 +0000 2015\",\n",
      "            \"favourites_count\": 9,\n",
      "            \"utc_offset\": null,\n",
      "            \"time_zone\": null,\n",
      "            \"geo_enabled\": true,\n",
      "            \"verified\": false,\n",
      "            \"statuses_count\": 56196,\n",
      "            \"lang\": null,\n",
      "            \"contributors_enabled\": false,\n",
      "            \"is_translator\": false,\n",
      "            \"is_translation_enabled\": false,\n",
      "            \"profile_background_color\": \"000000\",\n",
      "            \"profile_background_image_url\": \"http://abs.twimg.com/images/themes/theme1/bg.png\",\n",
      "            \"profile_background_image_url_https\": \"https://abs.twimg.com/images/themes/theme1/bg.png\",\n",
      "            \"profile_background_tile\": false,\n",
      "            \"profile_image_url\": \"http://pbs.twimg.com/profile_images/998301258413850625/9BZwTjgv_normal.jpg\",\n",
      "            \"profile_image_url_https\": \"https://pbs.twimg.com/profile_images/998301258413850625/9BZwTjgv_normal.jpg\",\n",
      "            \"profile_banner_url\": \"https://pbs.twimg.com/profile_banners/4263007693/1583526136\",\n",
      "            \"profile_link_color\": \"DD2E44\",\n",
      "            \"profile_sidebar_border_color\": \"000000\",\n",
      "            \"profile_sidebar_fill_color\": \"000000\",\n",
      "            \"profile_text_color\": \"000000\",\n",
      "            \"profile_use_background_image\": false,\n",
      "            \"has_extended_profile\": true,\n",
      "            \"default_profile\": false,\n",
      "            \"default_profile_image\": false,\n",
      "            \"following\": false,\n",
      "            \"follow_request_sent\": false,\n",
      "            \"notifications\": false,\n",
      "            \"translator_type\": \"none\"\n",
      "        },\n",
      "        \"geo\": null,\n",
      "        \"coordinates\": null,\n",
      "        \"place\": null,\n",
      "        \"contributors\": null,\n",
      "        \"is_quote_status\": false,\n",
      "        \"retweet_count\": 49,\n",
      "        \"favorite_count\": 50,\n",
      "        \"favorited\": false,\n",
      "        \"retweeted\": false,\n",
      "        \"possibly_sensitive\": false,\n",
      "        \"lang\": \"en\"\n",
      "    },\n",
      "    \"is_quote_status\": false,\n",
      "    \"retweet_count\": 49,\n",
      "    \"favorite_count\": 0,\n",
      "    \"favorited\": false,\n",
      "    \"retweeted\": false,\n",
      "    \"lang\": \"en\"\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# queries with Boolean operators\n",
    "import json\n",
    "\n",
    "tweets = tweepy.Cursor(api.search, q=\"#nlproc\")\n",
    "\n",
    "for item in tweets.items(2):\n",
    "    print(json.dumps(item._json, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the API: users\n",
    "\n",
    "Get some info on a given user, and explore their friends/followers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User: elonmusk\n",
      "------\n",
      "Friends: 92\n",
      "Followers: 35673217\n",
      "------\n",
      "jack\n",
      "TalulahRiley\n",
      "justinemusk\n",
      "MKBHD\n",
      "jk_rowling\n",
      "joerogan\n",
      "Blklivesmatter\n",
      "Erdayastronaut\n",
      "engineeringvids\n",
      "TheWeirdHistory\n",
      "------\n",
      "x_funeral_x\n",
      "StandWithPower1\n",
      "jamesglackin10\n",
      "wernerfinch5\n",
      "_katkouture\n",
      "VASOLAIYAPPAN\n",
      "lemonthewoozi\n",
      "karlwreed1\n",
      "MichaelChao0522\n",
      "bruce78398060\n"
     ]
    }
   ],
   "source": [
    "user = api.get_user(\"elonmusk\")\n",
    "\n",
    "print(\"User:\",user.screen_name)\n",
    "print(\"------\")\n",
    "print(\"Friends:\",user.friends_count)\n",
    "print(\"Followers:\",user.followers_count)\n",
    "print(\"------\")\n",
    "for friend in user.friends(count=10):\n",
    "    print(friend.screen_name)\n",
    "print(\"------\")\n",
    "for friend in user.followers(count=10):\n",
    "    print(friend.screen_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using the API: tweets from user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@akidesir Thursday\n",
      "@CodingMark I guess more people need to get more involved in the party primaries (which is a chore)\n",
      "The gerontocracy is out of touch with the people\n",
      "Selling weed literally went from major felony to essential business (open during pandemic) in much of America &amp; yet… https://t.co/32J0z4qDTI\n",
      "This will probably get me into trouble, but I feel I have to say it\n"
     ]
    }
   ],
   "source": [
    "user = api.get_user(\"elonmusk\", tweet_mode=\"extended\") # extended tweetmode gets also the longer 280/char tweets\n",
    "elon_tweets = user.timeline()\n",
    "\n",
    "for tweet in elon_tweets[:5]:\n",
    "    print(tweet.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For those who don't have a Twitter account and app, here are some tweets on and by Boris Johnson!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "user = \"@BorisJohnson\"\n",
    "tweets_on_user = tweepy.Cursor(api.search, q=user, tweet_mode=\"extended\")\n",
    "\n",
    "on_boris = list()\n",
    "for tweet in tweets_on_user.items(100):\n",
    "    on_boris.append(tweet.full_text)\n",
    "    \n",
    "#print(\"\\n------\\n\")\n",
    "# from user\n",
    "\n",
    "user = api.get_user(\"BorisJohnson\", tweet_mode=\"extended\") # extended tweetmode gets also the longer 280/char tweets\n",
    "tweets_from_user = user.timeline(count=100)\n",
    "\n",
    "from_boris = list()\n",
    "for tweet in tweets_from_user:\n",
    "    from_boris.append(tweet.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save to file\n",
    "f_on_boris = \"stuff/tweets_on_boris.csv\"\n",
    "f_from_boris = \"stuff/tweets_from_boris.csv\"\n",
    "\n",
    "# note we are using the \"\" as text delimiter\n",
    "with open(f_on_boris, \"w\") as f:\n",
    "    for t in on_boris:\n",
    "        f.write('\"'+t+'\"\\n')\n",
    "        \n",
    "with open(f_from_boris, \"w\") as f:\n",
    "    for t in from_boris:\n",
    "        f.write('\"'+t+'\"\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3.\n",
    "\n",
    "1. Download the last 100 (or another number) tweets mentioning a user you are interested into and the last 100 from the user itself. Alternatively, use the tweets in the on_boris and from_boris files.\n",
    "\n",
    "\n",
    "2. Create a minimal pipeline to normalize the tweets into lists of tokens.\n",
    "\n",
    "\n",
    "3. Count and compare from the two datasets, the most frequent (top 10):\n",
    "    - tokens\n",
    "    - hashtags\n",
    "    - other user mentions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}