04-crawler-snscrape-twitter.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7861d98a",
   "metadata": {},
   "source": [
    "# snscrape\n",
    "\n",
    "snscrape is a scraper for social networking services (SNS). It scrapes things like user profiles, hashtags, or searches and returns the discovered items, e.g. the relevant posts.\n",
    "\n",
    "\n",
    "https://github.com/JustAnotherArchivist/snscrape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8141ac5c",
   "metadata": {},
   "source": [
    "The following services are currently supported:\n",
    "\n",
    "- Facebook: user profiles, groups, and communities (aka visitor posts)\n",
    "- Instagram: user profiles, hashtags, and locations\n",
    "- Mastodon: user profiles and toots (single or thread)\n",
    "- Reddit: users, subreddits, and searches (via Pushshift)\n",
    "- Telegram: channels\n",
    "- Twitter: users, user profiles, hashtags, searches, tweets (single or surrounding thread), list posts, and trends\n",
    "- VKontakte: user profiles\n",
    "- Weibo (Sina Weibo): user profiles"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd283009",
   "metadata": {},
   "source": [
    "## Requirements\n",
    "snscrape requires Python 3.8 or higher. The Python package dependencies are installed automatically when you install snscrape.\n",
    "\n",
    "Note that one of the dependencies, lxml, also requires libxml2 and libxslt to be installed.\n",
    "\n",
    "## Installation\n",
    "> pip3 install snscrape\n",
    "\n",
    "If you want to use the development version:\n",
    "\n",
    "> pip3 install git+https://github.com/JustAnotherArchivist/snscrape.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "c6ed78cf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import lxml\n",
    "#import libxml2\n",
    "#import libxslt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "52b58b63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting snscrape\n",
      "  Downloading snscrape-0.4.3.20220106-py3-none-any.whl (59 kB)\n",
      "\u001b[K     |████████████████████████████████| 59 kB 733 kB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: beautifulsoup4 in /opt/anaconda3/lib/python3.9/site-packages (from snscrape) (4.10.0)\n",
      "Requirement already satisfied: lxml in /opt/anaconda3/lib/python3.9/site-packages (from snscrape) (4.6.3)\n",
      "Requirement already satisfied: filelock in /opt/anaconda3/lib/python3.9/site-packages (from snscrape) (3.3.1)\n",
      "Requirement already satisfied: requests[socks] in /opt/anaconda3/lib/python3.9/site-packages (from snscrape) (2.26.0)\n",
      "Requirement already satisfied: soupsieve>1.2 in /opt/anaconda3/lib/python3.9/site-packages (from beautifulsoup4->snscrape) (2.2.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/anaconda3/lib/python3.9/site-packages (from requests[socks]->snscrape) (3.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.9/site-packages (from requests[socks]->snscrape) (2021.10.8)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/anaconda3/lib/python3.9/site-packages (from requests[socks]->snscrape) (1.26.7)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/anaconda3/lib/python3.9/site-packages (from requests[socks]->snscrape) (2.0.4)\n",
      "Requirement already satisfied: PySocks!=1.5.7,>=1.5.6 in /opt/anaconda3/lib/python3.9/site-packages (from requests[socks]->snscrape) (1.7.1)\n",
      "Installing collected packages: snscrape\n",
      "Successfully installed snscrape-0.4.3.20220106\n"
     ]
    }
   ],
   "source": [
    "!pip3 install snscrape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "62bd0ddd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T09:17:04.027907Z",
     "start_time": "2022-10-17T09:17:03.951039Z"
    }
   },
   "outputs": [],
   "source": [
    "## 导入所需要的包\n",
    "import pandas as pd\n",
    "import datetime\n",
    "import os\n",
    "import snscrape.modules.twitter as sntwitter\n",
    "import pandas as pd\n",
    "import time\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "a6a5de67",
   "metadata": {},
   "outputs": [],
   "source": [
    "os.environ['http_proxy'] = \"http://127.0.0.1:9999\" \n",
    "os.environ['https_proxy'] = \"http://127.0.0.1:9999\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a00163f5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-17T09:17:06.208737Z",
     "start_time": "2022-10-17T09:17:06.076687Z"
    }
   },
   "outputs": [],
   "source": [
    "##获取日期列表\n",
    "def get_date_list(begin_date, end_date):\n",
    "    begin_date_dt = datetime.datetime.strptime(begin_date, \"%Y-%m-%d\")\n",
    "    end_date_dt = datetime.datetime.strptime(end_date, \"%Y-%m-%d\")\n",
    "    begin_date_second = (begin_date_dt + datetime.timedelta(days=1)).strftime(\"%Y-%m-%d\")\n",
    "    end_date_last = (end_date_dt + datetime.timedelta(days=-1)).strftime(\"%Y-%m-%d\")\n",
    "    since_date = pd.date_range(f\"{begin_date}\",f\"{end_date_last}\",freq=\"D\").strftime(\"%Y%m%d\").to_list()\n",
    "    since_date = [dt[:4]+'-'+dt[4:6]+'-'+dt[6:8] for dt in since_date]\n",
    "    until_date = pd.date_range(f\"{begin_date_second}\",f\"{end_date}\",freq=\"D\").strftime(\"%Y%m%d\").to_list()\n",
    "    until_date = [dt[:4]+'-'+dt[4:6]+'-'+dt[6:8] for dt in until_date]\n",
    "    return since_date,until_date\n",
    "\n",
    "##保存文件到本地\n",
    "def saveFile(df, path, filename):\n",
    "    '''\n",
    "    功能：将爬取保存到本地文件中\n",
    "    参数：要保存的内容，路径，文件名\n",
    "    '''\n",
    "    # 如果没有该文件夹，则自动生成\n",
    "    if not os.path.exists(path):\n",
    "        os.makedirs(path)\n",
    "        \n",
    "    # 保存文件\n",
    "    df.to_csv(path + filename)#index = False\n",
    "    \n",
    "##随机休眠\n",
    "def random_sleep(mu=60,sigma=60):\n",
    "    '''正态分布随机睡眠\n",
    "    :param mu: 平均值\n",
    "    :param sigma: 标准差，决定波动范围\n",
    "    '''\n",
    "    secs = random.normalvariate(mu, sigma)\n",
    "    if secs <= 60:\n",
    "        secs = mu  # 太小则重置为平均值\n",
    "    time.sleep(secs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "287c37d1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-18T11:15:51.712105Z",
     "start_time": "2022-10-18T05:10:37.720694Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "请输入开始日期:2022-02-10\n",
      "请输入结束日期:2022-02-15\n",
      "请输入爬取关键词:bitcoin\n",
      "爬取完成：2022-02-10\n",
      "爬取完成：2022-02-11\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Tweet 1491943423862972423 contains an app icon medium key '4_1582211007291834369' on app 'android_app'/'com.gemini.android.app', but the corresponding medium is missing; dropping\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "爬取完成：2022-02-12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Tweet 1455815359609675778 contains an app icon medium key '4_1582070568056242197' on app 'android_app'/'de.traderepublic.app', but the corresponding medium is missing; dropping\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "爬取完成：2022-02-13\n",
      "爬取完成：2022-02-14\n"
     ]
    }
   ],
   "source": [
    "if __name__ == '__main__':\n",
    "    # 输入开始日期、结束日期、爬取关键词\n",
    "    begin_date = input('请输入开始日期:')\n",
    "    end_date = input('请输入结束日期:')\n",
    "    keyword = input('请输入爬取关键词:')\n",
    "    since_date,until_date = get_date_list(begin_date, end_date)\n",
    "    for start_date,end_date in zip(since_date,until_date):\n",
    "        year = start_date[:4]\n",
    "        month = start_date[5:7]\n",
    "        day = start_date[-2:]\n",
    "        path = f\"{keyword}\" + '/' + year + '-' + month + '/'\n",
    "        # Creating list to append tweet data\n",
    "        tweets_list = []\n",
    "        try:\n",
    "        # Using TwitterSearchScraper to scrape data and append tweets to list\n",
    "            for i,tweet in enumerate(sntwitter.TwitterSearchScraper('Bitcoin since:'+f\"{start_date}\"+' until:'+f\"{end_date}\").get_items()):\n",
    "                tweets_list.append([\n",
    "                                    tweet.date, tweet.url, tweet.id, tweet.rawContent, tweet.replyCount, tweet.retweetCount, tweet.likeCount, \n",
    "                                    tweet.quoteCount, tweet.conversationId, tweet.lang, tweet.source, tweet.sourceUrl, tweet.links, tweet.media,\n",
    "                                    tweet.retweetedTweet, tweet.quotedTweet, tweet.inReplyToTweetId, tweet.inReplyToUser, tweet.mentionedUsers, \n",
    "                                    tweet.coordinates, tweet.place, tweet.hashtags, tweet.cashtags, tweet.card, #tweet相关的字段\n",
    "                                    tweet.user.username, tweet.user.displayname, tweet.user.id, tweet.user.rawDescription, tweet.user.descriptionLinks,\n",
    "                                    tweet.user.verified, tweet.user.created,tweet.user.followersCount, tweet.user.friendsCount, tweet.user.statusesCount, \n",
    "                                    tweet.user.favouritesCount,tweet.user.listedCount, tweet.user.mediaCount, tweet.user.location, tweet.user.protected, \n",
    "                                    tweet.user.link,tweet.user.profileImageUrl, tweet.user.profileBannerUrl, tweet.user.label ##user相关的字段\n",
    "                                    ])\n",
    "            # Creating a dataframe from the tweets list above\n",
    "            tweets_df = pd.DataFrame(tweets_list, columns=[\n",
    "                                                            'Datetime', 'Tweet_url', 'Tweet_id', 'Tweet_content', 'Tweet_reply_count', 'Tweet_retweet_count',\n",
    "                                                            'Tweet_like_count', 'Tweet_quote_count', 'Tweet_conversation_id', 'Tweet_language', 'Tweet_source', \n",
    "                                                            'Tweet_source_url', 'Tweet_links', 'Tweet_mdeia', 'Tweet_retweeted_tweet', 'Tweet_quoted_tweet', \n",
    "                                                            'Tweet_inReplyToTweetId','Tweet_inReplyToUser','Tweet_mentioned_users','Tweet_coordinates', 'Tweet_place', \n",
    "                                                            'Tweet_hashtags', 'Tweet_cashtags','Tweet_card',#tweet 相关的字段\n",
    "                                                            'Username','User_displayname', 'User_id', 'User_profile_description', 'User_description_link', 'User_verified',\n",
    "                                                            'User_created', 'User_followers_count', 'User_friends_count', 'User_statuses_count', 'User_favourites_count',\n",
    "                                                            'User_listed_count', 'User_media_count', 'User_location', 'User_protected', 'User_profile_link', 'User_profile_image_url',\n",
    "                                                            'User_profile_banner_url', 'User_label' #user 相关的字段\n",
    "                                                          ])\n",
    "\n",
    "            fileName = year + '-' + month + '-' + day + '.csv'\n",
    "            saveFile(tweets_df, path, fileName)\n",
    "            print(\"爬取完成：\" + year + '-' + month + '-' + day)\n",
    "            random_sleep()       # 随机休眠\n",
    "        except KeyError as error_msg:\n",
    "            print(f\"Sorry,{error_msg} is not a valid key!\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "40113ec8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-19T07:41:08.338199Z",
     "start_time": "2022-10-19T07:41:04.108862Z"
    }
   },
   "outputs": [],
   "source": [
    "tweets_list = []\n",
    "for i,tweet in enumerate(sntwitter.TwitterTweetScraper(1565127293961248773).get_items()):\n",
    "    tweets_list.append([\n",
    "                        tweet.date, tweet.url, tweet.id, tweet.rawContent, tweet.replyCount, tweet.retweetCount, tweet.likeCount, \n",
    "                        tweet.quoteCount, tweet.conversationId, tweet.lang, tweet.source, tweet.sourceUrl, tweet.links, tweet.media,\n",
    "                        tweet.retweetedTweet, tweet.quotedTweet, tweet.inReplyToTweetId, tweet.inReplyToUser, tweet.mentionedUsers, \n",
    "                        tweet.coordinates, tweet.place, tweet.hashtags, tweet.cashtags, tweet.card, #tweet相关的字段\n",
    "                        tweet.user.username, tweet.user.displayname, tweet.user.id, tweet.user.rawDescription, tweet.user.descriptionLinks,\n",
    "                        tweet.user.verified, tweet.user.created,tweet.user.followersCount, tweet.user.friendsCount, tweet.user.statusesCount, \n",
    "                        tweet.user.favouritesCount,tweet.user.listedCount, tweet.user.mediaCount, tweet.user.location, tweet.user.protected, \n",
    "                        tweet.user.link,tweet.user.profileImageUrl, tweet.user.profileBannerUrl, tweet.user.label ##user相关的字段\n",
    "                        ])\n",
    "            # Creating a dataframe from the tweets list above\n",
    "tweets_df = pd.DataFrame(tweets_list, columns=[\n",
    "                                                'Datetime', 'Tweet_url', 'Tweet_id', 'Tweet_content', 'Tweet_reply_count', 'Tweet_retweet_count',\n",
    "                                                'Tweet_like_count', 'Tweet_quote_count', 'Tweet_conversation_id', 'Tweet_language', 'Tweet_source', \n",
    "                                                'Tweet_source_url', 'Tweet_links', 'Tweet_mdeia', 'Tweet_retweeted_tweet', 'Tweet_quoted_tweet', \n",
    "                                                'Tweet_inReplyToTweetId','Tweet_inReplyToUser','Tweet_mentioned_users','Tweet_coordinates', 'Tweet_place', \n",
    "                                                'Tweet_hashtags', 'Tweet_cashtags','Tweet_card',#tweet 相关的字段\n",
    "                                                'Username','User_displayname', 'User_id', 'User_profile_description', 'User_description_link', 'User_verified',\n",
    "                                                'User_created', 'User_followers_count', 'User_friends_count', 'User_statuses_count', 'User_favourites_count',\n",
    "                                                'User_listed_count', 'User_media_count', 'User_location', 'User_protected', 'User_profile_link', 'User_profile_image_url',\n",
    "                                                'User_profile_banner_url', 'User_label' #user 相关的字段\n",
    "                                              ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "101de160",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-19T07:41:20.107599Z",
     "start_time": "2022-10-19T07:41:20.041825Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Datetime</th>\n",
       "      <th>Tweet_url</th>\n",
       "      <th>Tweet_id</th>\n",
       "      <th>Tweet_content</th>\n",
       "      <th>Tweet_reply_count</th>\n",
       "      <th>Tweet_retweet_count</th>\n",
       "      <th>Tweet_like_count</th>\n",
       "      <th>Tweet_quote_count</th>\n",
       "      <th>Tweet_conversation_id</th>\n",
       "      <th>Tweet_language</th>\n",
       "      <th>...</th>\n",
       "      <th>User_statuses_count</th>\n",
       "      <th>User_favourites_count</th>\n",
       "      <th>User_listed_count</th>\n",
       "      <th>User_media_count</th>\n",
       "      <th>User_location</th>\n",
       "      <th>User_protected</th>\n",
       "      <th>User_profile_link</th>\n",
       "      <th>User_profile_image_url</th>\n",
       "      <th>User_profile_banner_url</th>\n",
       "      <th>User_label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022-09-01 00:00:00+00:00</td>\n",
       "      <td>https://twitter.com/InformazioneA/status/15651...</td>\n",
       "      <td>1565127293961248773</td>\n",
       "      <td>#Report #31August</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1565127293961248773</td>\n",
       "      <td>qht</td>\n",
       "      <td>...</td>\n",
       "      <td>18973</td>\n",
       "      <td>58</td>\n",
       "      <td>36</td>\n",
       "      <td>12139</td>\n",
       "      <td></td>\n",
       "      <td>False</td>\n",
       "      <td>TextLink(text='informazionea.altervista.org', ...</td>\n",
       "      <td>https://pbs.twimg.com/profile_images/123559427...</td>\n",
       "      <td>https://pbs.twimg.com/profile_banners/12352445...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   Datetime  \\\n",
       "0 2022-09-01 00:00:00+00:00   \n",
       "\n",
       "                                           Tweet_url             Tweet_id  \\\n",
       "0  https://twitter.com/InformazioneA/status/15651...  1565127293961248773   \n",
       "\n",
       "       Tweet_content  Tweet_reply_count  Tweet_retweet_count  \\\n",
       "0  #Report #31August                  0                    0   \n",
       "\n",
       "   Tweet_like_count  Tweet_quote_count  Tweet_conversation_id Tweet_language  \\\n",
       "0                 0                  0    1565127293961248773            qht   \n",
       "\n",
       "   ... User_statuses_count User_favourites_count User_listed_count  \\\n",
       "0  ...               18973                    58                36   \n",
       "\n",
       "  User_media_count User_location User_protected  \\\n",
       "0            12139                        False   \n",
       "\n",
       "                                   User_profile_link  \\\n",
       "0  TextLink(text='informazionea.altervista.org', ...   \n",
       "\n",
       "                              User_profile_image_url  \\\n",
       "0  https://pbs.twimg.com/profile_images/123559427...   \n",
       "\n",
       "                             User_profile_banner_url User_label  \n",
       "0  https://pbs.twimg.com/profile_banners/12352445...       None  \n",
       "\n",
       "[1 rows x 43 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16d9be18",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}