08-07-covid19-pew-survey.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "# Analysing the Pew Survey Data of COVID19\n",
    "\n",
    "\n",
    "\n",
    "![image.png](./images/author.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "阅读 https://www.journalism.org/dataset/election-news-pathways-april-2020-survey/\n",
    "\n",
    "1. 下载Pathways-April-2020-ATP-W66-1.zip 数据，该数据来自pewresearch针对2020年总统选举所做的panel study中的一次，其中加入了关于covid19的部分问题。\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/pew.png)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/pew2.png)\n",
    "\n",
    "https://www.journalism.org/2020/05/08/americans-views-of-the-news-media-during-the-covid-19-outbreak/"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![image.png](./images/pew3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:20:21.669695Z",
     "start_time": "2020-05-19T01:20:18.122032Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# the statsmodels.api uses numpy array notation\n",
    "# statsmodels.formula.api use formula notation (similar to R's formula notation)\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import statsmodels.api as sm\n",
    "import statsmodels.formula.api as smf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:20:52.007326Z",
     "start_time": "2020-05-19T01:20:51.998110Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/datalab/bigdata/Pathways April 2020 (ATP W66)\n"
     ]
    }
   ],
   "source": [
    "cd \"/Users/datalab/bigdata/Pathways April 2020 (ATP W66)/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:21:04.185920Z",
     "start_time": "2020-05-19T01:21:04.025337Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31mPathways April 2020 (ATP W66) Methodology.pdf\u001b[m\u001b[m*\r\n",
      "\u001b[31mPathways April 2020 (ATP W66) Questionnaire.pdf\u001b[m\u001b[m*\r\n",
      "\u001b[31mPathways April 2020 (ATP W66) Readme.txt\u001b[m\u001b[m*\r\n",
      "\u001b[31mPathways April 2020 (ATP W66) Topline.pdf\u001b[m\u001b[m*\r\n",
      "\u001b[31mPathways April 2020 (ATP W66).sav\u001b[m\u001b[m*\r\n"
     ]
    }
   ],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "First of all, we install the pyreadstat module, which allows us to import SPSS files as DataFrames "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T09:16:46.180692Z",
     "start_time": "2020-05-13T09:16:41.356418Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting pyreadstat\n",
      "  Downloading pyreadstat-0.3.3-cp37-cp37m-macosx_10_9_x86_64.whl (527 kB)\n",
      "\u001b[K     |████████████████████████████████| 527 kB 233 kB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: pandas>0.24.0 in /opt/anaconda3/lib/python3.7/site-packages (from pyreadstat) (1.0.1)\n",
      "Requirement already satisfied: numpy>=1.13.3 in /opt/anaconda3/lib/python3.7/site-packages (from pandas>0.24.0->pyreadstat) (1.18.1)\n",
      "Requirement already satisfied: pytz>=2017.2 in /opt/anaconda3/lib/python3.7/site-packages (from pandas>0.24.0->pyreadstat) (2019.3)\n",
      "Requirement already satisfied: python-dateutil>=2.6.1 in /opt/anaconda3/lib/python3.7/site-packages (from pandas>0.24.0->pyreadstat) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /opt/anaconda3/lib/python3.7/site-packages (from python-dateutil>=2.6.1->pandas>0.24.0->pyreadstat) (1.14.0)\n",
      "Installing collected packages: pyreadstat\n",
      "Successfully installed pyreadstat-0.3.3\n"
     ]
    }
   ],
   "source": [
    "!pip install pyreadstat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:18:58.442139Z",
     "start_time": "2020-05-13T16:18:58.439798Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# df = pd.read_spss('Pathways April 2020 (ATP W66).sav')\n",
    "# df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:21:50.999216Z",
     "start_time": "2020-05-19T01:21:50.189314Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QKEY</th>\n",
       "      <th>Device_Type_W66</th>\n",
       "      <th>LANG_W66</th>\n",
       "      <th>FORM_W66</th>\n",
       "      <th>COMATTACH_W66</th>\n",
       "      <th>MH_TRACK_a_W66</th>\n",
       "      <th>MH_TRACK_b_W66</th>\n",
       "      <th>MH_TRACK_c_W66</th>\n",
       "      <th>MH_TRACK_d_W66</th>\n",
       "      <th>MH_TRACK_e_W66</th>\n",
       "      <th>...</th>\n",
       "      <th>F_PARTYSUM_FINAL</th>\n",
       "      <th>F_INCOME</th>\n",
       "      <th>F_INCOME_RECODE</th>\n",
       "      <th>F_REG</th>\n",
       "      <th>F_IDEO</th>\n",
       "      <th>F_ACSWEB</th>\n",
       "      <th>F_VOLSUM</th>\n",
       "      <th>F_HISP_ORIGIN</th>\n",
       "      <th>F_YEARSINUS</th>\n",
       "      <th>WEIGHT_W66</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100197.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.294864</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100260.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.689713</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100314.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.442058</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100363.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.423189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100446.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.368074</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 119 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       QKEY  Device_Type_W66  LANG_W66  FORM_W66  COMATTACH_W66  \\\n",
       "0  100197.0              3.0       1.0       2.0            2.0   \n",
       "1  100260.0              3.0       1.0       1.0            2.0   \n",
       "2  100314.0              2.0       1.0       2.0            3.0   \n",
       "3  100363.0              2.0       1.0       2.0            2.0   \n",
       "4  100446.0              1.0       1.0       1.0            2.0   \n",
       "\n",
       "   MH_TRACK_a_W66  MH_TRACK_b_W66  MH_TRACK_c_W66  MH_TRACK_d_W66  \\\n",
       "0             1.0             1.0             1.0             2.0   \n",
       "1             1.0             1.0             1.0             3.0   \n",
       "2             1.0             2.0             2.0             3.0   \n",
       "3             1.0             2.0             3.0             2.0   \n",
       "4             2.0             3.0             2.0             2.0   \n",
       "\n",
       "   MH_TRACK_e_W66  ...  F_PARTYSUM_FINAL  F_INCOME  F_INCOME_RECODE  F_REG  \\\n",
       "0             2.0  ...               2.0       3.0              3.0    1.0   \n",
       "1             1.0  ...               1.0       6.0              2.0    1.0   \n",
       "2             1.0  ...               1.0       4.0              2.0    1.0   \n",
       "3             1.0  ...               2.0       9.0              1.0    1.0   \n",
       "4             1.0  ...               2.0       8.0              1.0    1.0   \n",
       "\n",
       "   F_IDEO  F_ACSWEB  F_VOLSUM  F_HISP_ORIGIN  F_YEARSINUS  WEIGHT_W66  \n",
       "0     3.0       1.0       1.0            6.0          5.0    0.294864  \n",
       "1     1.0       1.0       2.0            6.0          5.0    0.689713  \n",
       "2     3.0       1.0       1.0            6.0          5.0    0.442058  \n",
       "3     3.0       1.0       1.0            6.0          5.0    0.423189  \n",
       "4     4.0       1.0       1.0            6.0          5.0    0.368074  \n",
       "\n",
       "[5 rows x 119 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyreadstat\n",
    "df, meta = pyreadstat.read_sav('Pathways April 2020 (ATP W66).sav')\n",
    "df.head(5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:02:01.522119Z",
     "start_time": "2020-05-13T10:02:01.518653Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['QKEY', 'Device_Type_W66', 'LANG_W66', 'FORM_W66', 'COMATTACH_W66', 'MH_TRACK_a_W66', 'MH_TRACK_b_W66', 'MH_TRACK_c_W66', 'MH_TRACK_d_W66', 'MH_TRACK_e_W66', 'COVIDFOL_W66', 'COVIDCOVER1_W66', 'COVIDTONE_a_W66', 'COVIDTONE_b_W66', 'COVIDTONE_c_W66', 'COVIDTONE_d_W66', 'COVIDNEWSPLAT_a_W66', 'COVIDNEWSPLAT_b_W66', 'COVIDNEWSPLAT_c_W66', 'COVIDNEWSPLAT_d_W66', 'COVIDNEWSPLAT_e_W66', 'COVIDNEWSPLAT_f_W66', 'COVIDNEWSPLAT_g_W66', 'COVIDNEWSPLAT_h_W66', 'COVIDNEWSPLAT_i_W66', 'COVIDNEWSPLAT_j_W66', 'COVIDNEWSRELY_W66', 'COVIDDEAL_W66', 'COVIDNATLOC_W66', 'COVIDNEWSSWITCH_W66', 'COVIDDISC_W66', 'COVIDSM_W66', 'COVIDSMSHARE_W66', 'COVIDINFO_W66', 'COVIDREPORT_W66', 'COVIDNEWSFINAN_W66', 'COVIDNEWSPAIR_a_W66', 'COVIDNEWSPAIR_b_W66', 'COVIDNEWSPAIR_c_W66', 'COVIDNEWSPAIR_d_W66', 'COVIDNEWSPAIR_e_W66', 'COVIDINFODIFF_W66', 'COVIDWORN_W66', 'COVIDNEWSFEEL_W66', 'COVIDTOPICA_a_W66', 'COVIDTOPICA_b_W66', 'COVIDTOPICA_c_W66', 'COVIDTOPICA_d_W66', 'COVIDTOPICA_e_W66', 'COVIDTOPICA_f_W66', 'COVIDTOPICB_a_W66', 'COVIDTOPICB_b_W66', 'COVIDTOPICB_c_W66', 'COVIDTOPICB_d_W66', 'COVIDTOPICB_e_W66', 'COVIDTOPICB_f_W66', 'COVIDTOPICB_g_W66', 'COVIDINFOCHECK_W66', 'MH_TRACK_CV_W66', 'COVIDCUREWORK_W66', 'COVIDCUREHEARD_a_W66', 'COVIDCUREHEARD_b_W66', 'COVIDCUREHEARD_c_W66', 'COVIDCUREHEARD_d_W66', 'COVIDCUREHEARD_e_W66', 'COVIDCUREHEARD_f_W66', 'COVIDBILL_a_W66', 'COVIDBILL_b_W66', 'COVIDBILL_c_W66', 'COVIDBILL_d_W66', 'COVIDBILL_e_W66', 'COVIDBILLSIZE_W66', 'COVIDPOLCOOP_W66', 'COVIDMISINFO_W66', 'COVIDCONFUSE_W66', 'COVIDINFOWRONG_W66', 'COVIDAVOID_W66', 'CAMPNII_W66', 'CAMP20INT_W66', 'PELECT20_STATUS_W66', 'COVIDCONG1_W66', 'COVIDCONG2_W66', 'COVIDSPIR_W66', 'GROUP_TRUST_a_W57', 'NEWS_MOST_W57', 'TALKPOL_W57', 'FOLELECT_W57', 'CR_PARTYNEWSUSE_W57', 'CR_NEWS_INFLU_W57', 'CR_DIG_SAV_W57', 'CR_MAINSOPOL_GROUPS_W57', 'CR_KNOWLEDGE_INDEX_W57', 'F_METRO', 'F_CREGION', 'F_AGECAT', 'F_SEX', 'F_EDUCCAT', 'F_EDUCCAT2', 'F_HISP', 'F_RACECMB', 'F_RACETHN', 'F_CITIZEN', 'F_NATIVITY2', 'F_MARITAL', 'F_RELIG', 'F_BORN', 'F_ATTEND', 'F_PARTY_FINAL', 'F_PARTYLN_FINAL', 'F_PARTYSUM_FINAL', 'F_INCOME', 'F_INCOME_RECODE', 'F_REG', 'F_IDEO', 'F_ACSWEB', 'F_VOLSUM', 'F_HISP_ORIGIN', 'F_YEARSINUS', 'WEIGHT_W66'])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta_dict = dict(zip(meta.column_names, meta.column_labels))\n",
    "meta_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:23:25.090719Z",
     "start_time": "2020-05-13T10:23:25.086920Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Family income',\n",
       " 'COVIDCOVER1_W66. How well do you think the news media have covered the coronavirus outbreak?',\n",
       " 'COVIDFOL_W66. How closely have you been following news about the outbreak of the coronavirus strain known as COVID-19?')"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta_dict['F_INCOME'], meta_dict['COVIDCOVER1_W66'], meta_dict['COVIDFOL_W66']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 162,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T15:42:20.841147Z",
     "start_time": "2020-05-13T15:42:20.740402Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "QKEY Unique ID\n",
      "Device_Type_W66 The device used in the latest access of the survey link\n",
      "LANG_W66 Preferred language\n",
      "FORM_W66 Form number\n",
      "COMATTACH_W66 COMATTACH_W66. In general, how attached do you feel to your local community?\n",
      "MH_TRACK_a_W66 MH_TRACK_a_W66. In the past 7 days, how often have you…      Felt nervous, anxious, or on edge?\n",
      "MH_TRACK_b_W66 MH_TRACK_b_W66. In the past 7 days, how often have you…      Felt depressed?\n",
      "MH_TRACK_c_W66 MH_TRACK_c_W66. In the past 7 days, how often have you…      Felt lonely?\n",
      "MH_TRACK_d_W66 MH_TRACK_d_W66. In the past 7 days, how often have you…      Felt hopeful about the future?\n",
      "MH_TRACK_e_W66 MH_TRACK_e_W66. In the past 7 days, how often have you…      Had trouble sleeping?\n",
      "COVIDFOL_W66 COVIDFOL_W66. How closely have you been following news about the outbreak of the coronavirus strain known as COVID-19?\n",
      "COVIDCOVER1_W66 COVIDCOVER1_W66. How well do you think the news media have covered the coronavirus outbreak?\n",
      "COVIDTONE_a_W66 COVIDTONE_a_W66. Thinking about the coronavirus outbreak, how in general do you think each of the following has responded?       The news media\n",
      "COVIDTONE_b_W66 COVIDTONE_b_W66. Thinking about the coronavirus outbreak, how in general do you think each of the following has responded?        Donald Trump\n",
      "COVIDTONE_c_W66 COVIDTONE_c_W66. Thinking about the coronavirus outbreak, how in general do you think each of the following has responded?        Democratic leaders in Congress\n",
      "COVIDTONE_d_W66 COVIDTONE_d_W66. Thinking about the coronavirus outbreak, how in general do you think each of the following has responded? Public health officials such as those at the CDC (Centers for Disease Control and Prevention)\n",
      "COVIDNEWSPLAT_a_W66 COVIDNEWSPLAT_a_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.          International news outlets\n",
      "COVIDNEWSPLAT_b_W66 COVIDNEWSPLAT_b_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.           National news outlets\n",
      "COVIDNEWSPLAT_c_W66 COVIDNEWSPLAT_c_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.            Local news outlets\n",
      "COVIDNEWSPLAT_d_W66 COVIDNEWSPLAT_d_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.           Donald Trump and his coronavirus task force\n",
      "COVIDNEWSPLAT_e_W66 COVIDNEWSPLAT_e_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.           Joe Biden and his campaign\n",
      "COVIDNEWSPLAT_f_W66 COVIDNEWSPLAT_f_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.            State and local elected officials and their offices\n",
      "COVIDNEWSPLAT_g_W66 COVIDNEWSPLAT_g_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.             Public health organizations and officials\n",
      "COVIDNEWSPLAT_h_W66 COVIDNEWSPLAT_h_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.              Friends, family and neighbors\n",
      "COVIDNEWSPLAT_i_W66 COVIDNEWSPLAT_i_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.               Community or neighborhood newsletter or Listserv\n",
      "COVIDNEWSPLAT_j_W66 COVIDNEWSPLAT_j_W66. Please indicate whether each of the following is a source of news for you about the coronavirus outbreak.                Online forums or discussion groups\n",
      "COVIDNEWSRELY_W66 COVIDNEWSRELY_W66. And which of these sources do you RELY ON MOST for news about the coronavirus outbreak?\n",
      "COVIDDEAL_W66 COVIDDEAL_W66. All in all, do you think that the coronavirus outbreak has been…\n",
      "COVIDNATLOC_W66 COVIDNATLOC_W66. Do you pay more attention to news about the coronavirus outbreak at...\n",
      "COVIDNEWSSWITCH_W66 COVIDNEWSSWITCH_W66. In keeping up with the coronavirus outbreak, have you turned to any different sources for news than you usually do?\n",
      "COVIDDISC_W66 COVIDDISC_W66. How often, if at all, are you discussing the coronavirus outbreak with others whether online, in person or over the phone?\n",
      "COVIDSM_W66 COVIDSM_W66. How often, if ever, do you get news and information about the coronavirus outbreak on social media (such as Facebook, Twitter, or Instagram)?\n",
      "COVIDSMSHARE_W66 COVIDSMSHARE_W66. Have you posted or shared any news and information about the coronavirus outbreak on social media?\n",
      "COVIDINFO_W66 COVIDINFO_W66. In general, do you feel like you have a handle on the issues and developments surrounding the coronavirus outbreak?\n",
      "COVIDREPORT_W66 COVIDREPORT_W66. How much, if at all, do you think journalists have had to change the way they report the news due to the coronavirus outbreak?\n",
      "COVIDNEWSFINAN_W66 COVIDNEWSFINAN_W66. To the best of your knowledge, what impact has the coronavirus outbreak had on the finances of most news organizations?\n",
      "COVIDNEWSPAIR_a_W66 COVIDNEWSPAIR_a_W66. For each pair of opposite phrases, which one do you feel better describes the news media’s coverage of the COVID-19 outbreak? If you think neither phrase applies, please indicate so.Largely accurate/ inaccurate\n",
      "COVIDNEWSPAIR_b_W66 COVIDNEWSPAIR_b_W66. For each pair of opposite phrases, which one do you feel better describes the news media’s coverage of the COVID-19 outbreak? If you think neither phrase applies, please indicate so.Helping/ Hurting the country\n",
      "COVIDNEWSPAIR_c_W66 COVIDNEWSPAIR_c_W66. For each pair of opposite phrases, which one do you feel better describes the news media’s coverage of the COVID-19 outbreak? If you think neither phrase applies, please indicate so. Working for the benefit of the public/ to benefit\n",
      "COVIDNEWSPAIR_d_W66 COVIDNEWSPAIR_d_W66. For each pair of opposite phrases, which one do you feel better describes the news media’s coverage of the COVID-19 outbreak? If you think neither phrase applies, please indicate so.Gets me the information I need/ Is not getting me t\n",
      "COVIDNEWSPAIR_e_W66 COVIDNEWSPAIR_e_W66. For each pair of opposite phrases, which one do you feel better describes the news media’s coverage of the COVID-19 outbreak? If you think neither phrase applies, please indicate so. More positive/ negative than it should be\n",
      "COVIDINFODIFF_W66 COVIDINFODIFF_W66. When you get news and information about the coronavirus outbreak, do you generally find it…\n",
      "COVIDWORN_W66 COVIDWORN_W66. Which of the following statements comes closer to how you feel?\n",
      "COVIDNEWSFEEL_W66 COVIDNEWSFEEL_W66. Does keeping up with news about the coronavirus outbreak make you feel…\n",
      "COVIDTOPICA_a_W66 COVIDTOPICA_a_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak?      The economic impact of the outbreak\n",
      "COVIDTOPICA_b_W66 COVIDTOPICA_b_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak?       Actions and statements by the federal government\n",
      "COVIDTOPICA_c_W66 COVIDTOPICA_c_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak?         The number of confirmed cases and deaths in the U.S.\n",
      "COVIDTOPICA_d_W66 COVIDTOPICA_d_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak? Advice from national health organizations, such as the CDC (Centers for Disease Control and Prevention)\n",
      "COVIDTOPICA_e_W66 COVIDTOPICA_e_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak?          The ability of hospitals across the country to treat patients\n",
      "COVIDTOPICA_f_W66 COVIDTOPICA_f_W66. How closely are you following NATIONAL news and information about each of these topics related to the coronavirus outbreak?            The health impact of the coronavirus on people like me\n",
      "COVIDTOPICB_a_W66 COVIDTOPICB_a_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?        The status of nearby businesses\n",
      "COVIDTOPICB_b_W66 COVIDTOPICB_b_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?         The status of nearby schools\n",
      "COVIDTOPICB_c_W66 COVIDTOPICB_c_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?The availability of food and other essentials in stores\n",
      "COVIDTOPICB_d_W66 COVIDTOPICB_d_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?Actions and statements by your state and local government\n",
      "COVIDTOPICB_e_W66 COVIDTOPICB_e_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?          The availability of testing for COVID-19\n",
      "COVIDTOPICB_f_W66 COVIDTOPICB_f_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?            The status of nearby hospitals\n",
      "COVIDTOPICB_g_W66 COVIDTOPICB_g_W66. And how closely are you following news and information in YOUR LOCAL AREA about each of these topics related to the coronavirus outbreak?Availability of unemployment benefits and other aid\n",
      "COVIDINFOCHECK_W66 COVIDINFOCHECK_W66. If you wanted to check the accuracy of news and information about the coronavirus, how confident are you that you would know what steps to take?\n",
      "MH_TRACK_CV_W66 MH_TRACK_CV_W66. In the past 7 days, how often have you had physical reactions, such as sweating, trouble breathing, nausea, or a pounding heart, when THINKING about your experience with the coronavirus outbreak?\n",
      "COVIDCUREWORK_W66 COVIDCUREWORK_W66. As far as you know, how effective are at-home treatments for serious cases of the coronavirus?\n",
      "COVIDCUREHEARD_a_W66 COVIDCUREHEARD_a_W66. How much, if anything, have you heard about each of the following?      Using the medicine known as hydroxychloroquine to treat the coronavirus\n",
      "COVIDCUREHEARD_b_W66 COVIDCUREHEARD_b_W66. How much, if anything, have you heard about each of the following?       Drinking a form of diluted bleach sometimes called “miracle mineral solution” (MMS) to treat the coronavirus\n",
      "COVIDCUREHEARD_c_W66 COVIDCUREHEARD_c_W66. How much, if anything, have you heard about each of the following?       Taking vitamin C to prevent getting the coronavirus\n",
      "COVIDCUREHEARD_d_W66 COVIDCUREHEARD_d_W66. How much, if anything, have you heard about each of the following?          Treating COVID-19 patients with blood plasma transfusions from recovered patients\n",
      "COVIDCUREHEARD_e_W66 COVIDCUREHEARD_e_W66. How much, if anything, have you heard about each of the following The coronavirus going away in warmer weather\n",
      "COVIDCUREHEARD_f_W66 COVIDCUREHEARD_f_W66. How much, if anything, have you heard about each of the following?          A connection between 5G mobile-phone technology and the coronavirus\n",
      "COVIDBILL_a_W66 COVIDBILL_a_W66. From what you’ve seen or heard, does the economic aid law passed by Congress and signed by Donald Trump include each of the following or not?        $1,200 in stimulus money for many American adults\n",
      "COVIDBILL_b_W66 COVIDBILL_b_W66. From what you’ve seen or heard, does the economic aid law passed by Congress and signed by Donald Trump include each of the following or not?          Pay raises for members of Congress\n",
      "COVIDBILL_c_W66 COVIDBILL_c_W66. From what you’ve seen or heard, does the economic aid law passed by Congress and signed by Donald Trump include each of the following or not?Restrictions on businesses connected to Donald Trump and other administration officials from rec\n",
      "COVIDBILL_d_W66 COVIDBILL_d_W66. From what you’ve seen or heard, does the economic aid law passed by Congress and signed by Donald Trump include each of the following or not?Increased unemployment benefits for the next four months\n",
      "COVIDBILL_e_W66 COVIDBILL_e_W66. From what you’ve seen or heard, does the economic aid law passed by Congress and signed by Donald Trump include each of the following or not?Federal aid and benefits for undocumented immigrants\n",
      "COVIDBILLSIZE_W66 COVIDBILLSIZE_W66. From what you have seen or heard, about how much economic aid was in the bill passed by Congress and Donald Trump in March?\n",
      "COVIDPOLCOOP_W66 COVIDPOLCOOP_W66. How well are elected officials working together to address the coronavirus outbreak?\n",
      "COVIDMISINFO_W66 COVIDMISINFO_W66. How much, if any, news and information have you seen or heard about the coronavirus outbreak that seemed completely made up?\n",
      "COVIDCONFUSE_W66 COVIDCONFUSE_W66. How much do you think made-up news and information leaves Americans confused about the basic facts of the coronavirus outbreak?\n",
      "COVIDINFOWRONG_W66 COVIDINFOWRONG_W66. Did you initially think any of the made-up news and information you have seen or heard was true before realizing it was made-up?\n",
      "COVIDAVOID_W66 COVIDAVOID_W66. Have you stopped going to a specific source because you thought that they were putting out made-up news and information about the coronavirus outbreak?\n",
      "CAMPNII_W66 CAMPNII_W66. How closely have you been following news about candidates for the 2020 presidential election?\n",
      "CAMP20INT_W66 CAMP20INT_W66. How important, if at all, is it to you personally to get messages and statements about the upcoming election and other important issues from the presidential campaigns of Donald Trump and Joe Biden?\n",
      "PELECT20_STATUS_W66 PELECT20_STATUS_W66. As far as you know, are states still holding primaries for the Democratic presidential nomination…\n",
      "COVIDCONG1_W66 COVIDCONG1_W66. Has the congregation or house of worship you most often attend closed its regular religious services to the public in response to the coronavirus outbreak, or are people still gathering there in person?\n",
      "COVIDCONG2_W66 COVIDCONG2_W66. Is the congregation or house of worship you most often attend streaming or recording its services so that people can watch them online or on TV?\n",
      "COVIDSPIR_W66 COVIDSPIR_W66. As a result of the coronavirus outbreak, has your own religious faith become\n",
      "GROUP_TRUST_a_W57 GROUP_TRUST_a_W57. How much, if at all, do you trust the information you get from… National news organizations\n",
      "NEWS_MOST_W57 NEWS_MOST_W57. What is the most common way you get political and election news?\n",
      "TALKPOL_W57 TALKPOL_W57. Now, thinking about the people you talk with, whether in person, over the phone, or online…How often do you discuss government and politics with others?\n",
      "FOLELECT_W57 FOLELECT_W57. How closely do you follow political and election news?\n",
      "CR_PARTYNEWSUSE_W57 Index of news consumption by leaned party and outlet audience ideology - CREATED\n",
      "CR_NEWS_INFLU_W57 News influencers, combo of FOLELECT and LEADPOL - Includes follow somewhat in News influencer - CREATED\n",
      "CR_DIG_SAV_W57 Digital savviness combo of INTFREQ and DIGCONF - CREATED\n",
      "CR_MAINSOPOL_GROUPS_W57 Main source of election news, recoded - CREATED\n",
      "CR_KNOWLEDGE_INDEX_W57 Political knowledge index - CREATED\n",
      "F_METRO Metropolitan area indicator\n",
      "F_CREGION Census region based on self-reported zipcode\n",
      "F_AGECAT Age category\n",
      "F_SEX Sex\n",
      "F_EDUCCAT Education level category\n",
      "F_EDUCCAT2 Education level category 2\n",
      "F_HISP HISP.Are you of Hispanic, Latino, or Spanish origin, such as Mexican, Puerto Rican or Cuban? - Includes RACE backcodes\n",
      "F_RACECMB Combining race\n",
      "F_RACETHN Race-Ethnicity\n",
      "F_CITIZEN Citizenship\n",
      "F_NATIVITY2 NATIVITY. Where were you born? (with Cuba and Mexico broken out)\n",
      "F_MARITAL Marital status\n",
      "F_RELIG Religion\n",
      "F_BORN BORN. Would you describe yourself as a born-again or evangelical Christian, or not?\n",
      "F_ATTEND Religious service attendance\n",
      "F_PARTY_FINAL Party\n",
      "F_PARTYLN_FINAL Party lean\n",
      "F_PARTYSUM_FINAL Party summary\n",
      "F_INCOME Family income\n",
      "F_INCOME_RECODE Family income recoded\n",
      "F_REG Registered voter\n",
      "F_IDEO Ideology\n",
      "F_ACSWEB Household internet status\n",
      "F_VOLSUM Volunteerism status\n",
      "F_HISP_ORIGIN Hispanic Origin\n",
      "F_YEARSINUS Foreign-born Hispanic Years in U.S.\n",
      "WEIGHT_W66 Wave 66 weight\n"
     ]
    }
   ],
   "source": [
    "for i, j in meta_dict.items():\n",
    "    print(i, j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:24:21.100189Z",
     "start_time": "2020-05-19T01:24:21.081241Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    10139.000000\n",
       "mean         1.000000\n",
       "std          1.213926\n",
       "min          0.065288\n",
       "25%          0.351129\n",
       "50%          0.572091\n",
       "75%          1.107659\n",
       "max          7.487729\n",
       "Name: WEIGHT_W66, dtype: float64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['WEIGHT_W66'].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:11:24.055930Z",
     "start_time": "2020-05-13T16:11:24.046885Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    0.107506\n",
       "2.0    0.316698\n",
       "3.0    0.310090\n",
       "4.0    0.265707\n",
       "Name: F_AGECAT, dtype: float64"
      ]
     },
     "execution_count": 183,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['F_AGECAT'].value_counts(normalize=True).sort_index()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:43:26.673231Z",
     "start_time": "2020-05-13T16:43:26.666586Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    0.564750\n",
       "2.0    0.208798\n",
       "3.0    0.226452\n",
       "Name: COVIDNEWSPAIR_a_W66, dtype: float64"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['COVIDNEWSPAIR_a_W66'].value_counts(normalize=True).sort_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 205,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:43:32.512685Z",
     "start_time": "2020-05-13T16:43:32.504603Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Largely accurate      0.564750\n",
       "Neither applies       0.226452\n",
       "Largely inaccurate    0.208798\n",
       "Name: COVIDNEWSPAIR_a_W66, dtype: float64"
      ]
     },
     "execution_count": 205,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['COVIDNEWSPAIR_a_W66'].map(meta.variable_value_labels['COVIDNEWSPAIR_a_W66']).value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:27:40.283462Z",
     "start_time": "2020-05-19T01:27:39.761310Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def clean_missing_data(var):\n",
    "    # raw data using 99 as missing values\n",
    "    df[var][df[var]==99] =np.nan \n",
    "    df[var] = df[var].fillna(df[var].median())\n",
    "    return df[var]\n",
    "\n",
    "for i in df.columns:\n",
    "    df[i] = clean_missing_data(i)\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-19T01:30:28.168337Z",
     "start_time": "2020-05-19T01:30:28.133436Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# dummy coding\n",
    "df['republic'] = [1 if i==1 else 0 for i in df['F_PARTYLN_FINAL'] ]\n",
    "df['edu'] = [7-i for i in df['F_EDUCCAT2']]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weighting\n",
    "\n",
    "![image.png](./images/pew4.png)\n",
    "\n",
    "按总体人口数据，总体平均收入应为：(5000 * 1000+4000 * 2000+3000 * 3000)/6000 = 3666.67 元。\n",
    "\n",
    "- 按样本数据（不加权）：全部人口的样本平均收入 = (5000 * 200+4000 * 100+3000 * 100)/400 = 4250 元。\n",
    "- 采用总体/样本倍数加权，总体平均收入= (5000 * 200 * 5+4000 * 100 * 20+3000 * 100 * 30)/(200 * 5+100 * 20+100 * 30) = 3666.67元。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**A weighting adjustment technique** can only be carried of proper auxiliary variables are available. \n",
    "- Such variables must have been measured in the survey, and there population distribution must be available. \n",
    "- Typical auxiliary variables are gender, age, marital status and region of the country. \n",
    "- The population distribution of such variables can usually be obtained from national statistical institutes.\n",
    "\n",
    "|            | Young | Middle | Old  |\n",
    "| ---------- | ----- | ------ | ---- |\n",
    "| Population | 30%   | 40%    | 30%  |\n",
    "| Sample     | 60%   | 30%    | 10%  |\n",
    "| Weight     | 0.5   | 1.33   |  3   |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T15:38:12.941037Z",
     "start_time": "2020-05-13T15:38:12.744902Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD6CAYAAABApefCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAcxklEQVR4nO3de3CV9b3v8fenXJpUgiKko9wM5SJgt7WS04oFuZTQ00639TJuhFqpR0TbjjhY69ieqZczapmjHm2t24I3tpWz3bYK2oqloFDxQlsQ3SjKRRRJjkUIuitCrJjv+WM9SVfSQC5rJSvh+bxm1pj1e27ftYKf/NbvedbzU0RgZmaHt08UugAzM2t/DnszsxRw2JuZpYDD3swsBRz2ZmYp0L3QBRxMv379oqysrNBlmJl1KevWrdsdEaWN2ztt2JeVlbF27dpCl2Fm1qVI2t5Uu4dxzMxSwGFvZpYCDnszsxTotGP2Tfnoo4+orKykpqam0KVYM4qKihg4cCA9evQodClmRhcL+8rKSkpKSigrK0NSocuxg4gIqqurqaysZMiQIYUux8zoYsM4NTU19O3b10HfyUmib9++/gRm1ol0qbAHHPRdhH9PZp1Llwt7MzNrPYd9Hq1bt44rr7yyyWUTJ07kmWee6eCKWubaa69l9erVh1xn4cKFzJo1q4MqMrN861InaBu7dfnmvO5vbsWInLYfM2YMY8aMATInkx966CEuv/zyfJTWwOLFi+nXrx/jx4/Py/6uvfbanLa/4YYbuPjii+nXr19e6rGDy/e/+ZbK9f8NKzz37NvJ1q1bWbp0abvs+9FHH2X79ia/EV0Qd911l0/GmnVyDvtWmDx5Mk8//TQAc+bM4dJLLwVg2bJlnHfeeaxatYopU6awZcsWvvnNb/Lcc88xbNgw3n//fQCWL1/O5z73OQYNGsRTTz0FwP79+7nkkksYPnw4w4cPZ/78+QD1+6rz7W9/mwceeIB58+bxyCOPcMUVV/Cd73ynVfUB3HTTTYwePZpRo0axZMmSBvsGWLNmDSeeeCKjR4/m+9//PsOGDavf/3vvvcfXv/51jj32WC677DIAzjzzTKqqqhg/fjwLFy7M0zttZvnWpYdxOtrUqVN5+umnOe2003jppZf429/+BsAzzzxDRUVF/XrDhw9n0aJFXH/99axYsaK+vbKykpdeeol7772XG2+8kcmTJzNv3jwOHDjA5s2b2bNnD2PHjmXs2LEHreGqq67itddeY8qUKfUB3tL6nnrqKdavX8+GDRvYs2cPY8aM4fTTT6/f/sCBA0yfPp3777+f8ePHc8011zTY/3PPPceLL75I9+7dGTp0KFdccQWLFy+mrKyM1atXM3DgwLa/uWbWrtyzb4WKigqefvppdu7cSWlpKb179+add97h2WefbdALP5jp06cDMH78eHbs2AHA0qVLmTNnTv216WeffTarVq1ql/qWLl3KqlWrOOGEExg/fjwffvghO3furN9+8+bNHHXUUfXnAhr/MZk8eTKf/vSnOfrooxk9ejSVlZVtqtPMOp579q1w8skns2XLFpYtW0ZFRQXvvvsuy5cvZ+/evQwYMIAtW7YccvuioiIAevTowccffwxketOf+MTf/+ZGBN26daN79+7160DmVhG51nfgwAF++MMf1g/vNLZv3z569ux50GPW1d/4NZhZ5+eefStI4otf/CI///nPmTp1KpMmTeL222/nS1/60j+sW1xczJ49e6itrT3kPisqKrj99tuJCPbs2cOjjz7KlClTOO6449i0aRMffvghNTU1PP/88w32vXv3biKiVfWNGzeO+++/n7179xIR9eP7dUaOHMnWrVtZv349AAsWLGjR+3Kwesys8+jSPftCXA5WUVHBvHnzGDJkCIMHD+bVV1/l6quv/of1TjrpJHr27MlnPvMZNmzYcND9XX311Xz3u99l6NChFBcX8+Mf/5jjjz8eyAyjnHrqqYwePZoRI/7+WqdNm8Y555zDG2+8wU9/+tMW13f22Wfz7LPPMnLkSIqLi5k9ezannXZa/ba9evVi/vz5nHHGGRxxxBHMmDGDbt26NfueXHDBBUydOpVbbrmFb33rW82ub2YdT521N1ZeXh6NZ6p69dVXGTVqVIEqSp/Vq1dzzTXX1F851Fr+feWfr7O35khaFxHljdu7dM/e8m/VqlVMmDCBDz/8kJtuuomvfe1rhS7JspzyVsuG1vLv5gId1/Kl2TF7SUWSFkjaJGm7pLlJ+2WS3krav5q1/jxJlZI2SBqTtHWXtFBSlaQ1knzf207qrrvuon///owcOZIhQ4Yc9GSumXUtLenZHwEsAy4G+gKvSHoB+B5wAjAIWCHpOGA8MA4oAyYB9wAnAecDRcBAYBZwK3BGPl+I5ceiRYsKXYKZtYNme/YRUR0RD0fGbmAHcBrwUES8HxEbgTeBMcBZwMKIOBARy4FSScck7XdH5gTBIqD5i9LNzCxvWnXppaTPkumh9wOyb85SCRxLppef3V7VuD0i9gH7JPVpYv+zJa2VtHbXrl2tKc3MzA6hxWEvqR/wS+ACoCeQfQF5LfBxG9obiIgFEVEeEeWlpaUtLc3MzJrRorBPeuG/AX4UEX8G3gYGZK0ykMzwTuP2/mR6/fXtkoqB7hHx15yrNzOzFmnJ1Ti9gceAGyLiiaT5ceBcSZ+SNAo4GngxaZ8pqZukCmBzROxJ2i9Itj0PWJLn19EpFGLykv3793P66ac3+03dzjx5ipm1v5ZcjTMHOBm4TdJtSdtU4AHgFaAGmBURIWkxMAHYBlQDM5L17wDuk7QjWTYtL9Wv/EledlNv0g9z2ryjJi/JVlxczGOPPdbm7TuqTjMrrJZcjXN9RBwREcOyHtsi4saIGBIRoyLi2WTd2oiYExHHRcTJEfFa0l4TEdMjYlBETIiIv7T3Cyu09py8JJ+6Sp1mlhvfCK0V2mPykjpvvfUWQ4cOBaC2tpY+ffqwbt06IHNr5BUrVlBdXc03vvENjj/+eL785S/zl79k/mZ27/73D2jXXXcdZWVljB07lunTp3P99dfXL2t8/IPVaWaHH4d9K9RNDgLw0ksvUXfvnoNNXnLqqaeydetWSkpKgL9PXnLddddx4403Ntj34MGD6dGjB5WVlbzwwgscddRRrFy5EsicCxg3bhxz587lkksuYdOmTZxzzjn85CcNh7F+//vf8+ijj/Lyyy+zbNkyXn755QbLGx//YHWa2eHHYd8K7TF5SVP7f/LJJ7nqqqtYuXIl27dvp6ysjKKiIp544gnmzp3LyJEjufnmm6mqqmqw/YoVK5g5cya9evWid+/eDWahasnxzezw5RuhtUJ7TF6SraKigqVLl1JZWcmDDz7I3XffzcqVK5k6dSqQmUzkT3/6E717925y/01NPlJcXNzi45vZ4cs9+1Zoj8lLsk2aNIk//vGP1NbW0qtXL8rLy/nFL35RP0Q0btw47rzzTgDeeecdNm7c2GD7L3zhCzzwwAPU1NSwe/duHn744WaP2ZY6zazr6do9+xwvlWyLfE9ekq2kpIRevXrV//GYPHkyjzzyCCeeeCIAP/vZz5g5cyZ33nknffr04b777muw/YwZM1i1ahVDhw5l2LBhnHbaac1OPtK4To/bmx2ePHnJYezCCy9kwoQJnH/++QU5vn9f+ff8PVcU5LhjL/T97LuKg01e4mGcw0hVVVX9eYONGzfy+OOPM3ny5AJXZWadQdcexrEGPvjgA04//XQ++OADjjzySObPn8/AgQMLXZaZdQIO+8PIiBEjeO211wpdhpl1Ql1uGKeznmOwhvx7MutculTYFxUVUV1d7SDp5CKC6urq+uv6zazwutQwzsCBA6msrMSzWHV+RUVFPl9g1ol0qbDv0aMHQ4YMKXQZZmZdTpcaxjEzs7ZpzRy0xZJGtGcxZmbWPlo0LaGkJcBO4Mqk7XZJW7MeH0saLalM0v6s9puT9btLWiipStIaSR6LMTPrQC0Zs68Fbgd+C5wCEBGX1i2UNBz494jYKKkM+GNETGy0j/OBIjITk88CbgXOyLF2MzNroZZMS7g3Ip4EDhxklYuAe5vZzVnA3ZG5ZnIR0OTN3yXNlrRW0lpfcWNmlj85naCV1AP4F+D/Jk0BnCjpdUm/lTQsaR8EbAeIiH3APkl9Gu8vIhZERHlElJeWluZSmpmZZcn1apzTgWcj4j2AiNgeEUcDw4GVwMJkvZ5khoPq1AKePcPMrIPkGvYXAfc0boyIWmA+8Nmk6W1gAGSu6gG6R8Rfczy2mZm1UJvDXtJxwDAyPfi6tgFJmAOcB/wp+flx4IKs9iVtPa6ZmbVes1fjSCoB1gMlQJGkiWR69JOAf4uGN6oZDdwj6SNgS7IewB3AfZJ2ANuAaXl7BWZm1qxmwz4i3ifTg29sZRPrLgcGN9FeA0xvS4FmZpa7LnVvHDNLl1uXby7YsedWHF43DPC9cczMUsBhb2aWAg57M7MUcNibmaWAw97MLAUc9mZmKeCwNzNLAYe9mVkKOOzNzFLAYW9mlgIOezOzFHDYm5mlgMPezCwFWhz2koolHV63gTMzS4lmw15Sb0lLgJ3AlVntH0namjwezGqfJ6lS0gZJY5K27pIWSqqStEbSkPZ4MWZm1rSW3M++Frgd+C1wSlZ7VUQ0mNRE0mRgHFBGZiare4CTgPOBImAgMAu4FTgjx9rNzKyFmu3ZR8TeiHgSONCC/Z0FLIyIA8msVaWSjkna706mMFwETGlqY0mzJa2VtHbXrl0tfxVmZnZIuZyg7SvpdUkrJZUnbYOA7VnrVAHHZrdHxD5gn6Q+jXcYEQsiojwiyktLS3MozczMsrV5WsKIKAGQdA6wmEyg9yQz7FOnFvj4EO1mZtYBcr70MiJ+BRRLOgp4GxiQtbg/UJndLqkY6B4Rf8312GZm1jJtCntJ/ZJwR9JXgT0R8R7wODBTUjdJFcDmiNiTtF+QbH4esCT30s3MrKWaHcaRVAKsB0qAIkkTgZ8Cl0uqJdNrPydZfTEwAdgGVAMzkvY7gPsk7UiWTcvjazCzdnbr8s2FLqHDFeo1z61on68zNRv2EfE+MKyJRbc3sW4tMCd5ZLfXANPbWKOZmeXIt0swM0sBh72ZWQo47M3MUsBhb2aWAg57M7MUcNibmaWAw97MLAUc9mZmKeCwNzNLAYe9mVkKOOzNzFLAYW9mlgIOezOzFHDYm5mlQIvDXlKxpPa50bKZmbWrZsNeUm9JS4CdwJVJW19J/yFpSzLp+LlJe5mk/ZK2Jo+bk/bukhZKqpK0RtKQ9nxRZmbWUEsmHK8lM1HJb4FTkrZS4M6IWCVpGLBW0sPJsj9GxMRG+zgfKAIGArOAW4EzcqzdzMxaqNmefUTsjYgngQNZba9FxKrk563AR0DxIXZzFnB3RASwCJiSS9FmZtY6OZ+gTSYcfyEi/goEcGIytPPbpNcPMAjYDhAR+4B9kvo0sa/ZktZKWrtr165cSzMzs0ROYZ+E+U3AxQARsT0ijgaGAyuBhcmqPckMB9WpBT5uvL+IWBAR5RFRXlpamktpZmaWpc1hL+k44NfA+RHxZvayZOLx+cBnk6a3gQHJdsVA9+STgJmZdYA2hb2kAcAjwEUR8UJ2exLmAOcBf0p+fhy4IKt9SdvKNTOztmj2ahxJJcB6oAQokjQRENAP+HdJdauOTh73SPoI2AJclCy7A7hP0g5gGzAtj6/BzMya0WzYR8T7wLDm1kssBwY3sY8aYHrrSjMzs3zx7RLMzFLAYW9mlgIt+QatmaXcKW8tKHQJHW7N4NmFLiGv3LM3M0sBh72ZWQo47M3MUsBhb2aWAg57M7MUcNibmaWAw97MLAUc9mZmKeCwNzNLAYe9mVkK+HYJh4uVPynMcSf9sDDHNbNWaXHPXlKxpBHtWYyZmbWPZsNeUm9JS4CdwJVZ7ZdJekvSpmTS8br2eZIqJW2QNCZp6y5poaQqSWskDWmPF2NmZk1rSc++FrgduLyuQdJQ4HvACcCZZGan6iFpMjAOKEvWvyfZ5HygCBiYtN2ap/rNzKwFmg37iNgbEU8CB7KazwQeioj3I2Ij8CYwBjgLWBgRByJiOVAq6Zik/e6ICGARMCXPr8PMzA6hrSdoBwEvZz2vBI5N2h/Naq/Kat8OEBH7JO2T1Cci3s3eqaTZwGyAwYP/YXbDzq9QJ0nNzJrR1ksve5IZ3qlTC3zchvYGImJBRJRHRHlpaWkbSzMzs8baGvZvAwOyng8EdjTR3p9Mr7++XVIx0D0i/trGY5uZWSu1NewfB86V9ClJo4CjgReT9pmSukmqADZHxJ6k/YJk2/OAJTnWbWZmrdDsmL2kEmA9UAIUSZoIXAQ8ALwC1ACzIiIkLQYmANuAamBGsps7gPsk7UiWTcvz67BCKeR5ikJ9ocvnZqwLajbsI+J9YFgTi1YCNzZatxaYkzyy22uA6W0v08zMcuF745iZpYDD3swsBRz2ZmYp4Ltempk14ZS3FhToyDe3y14Pz7D31RJmZg14GMfMLAUc9mZmKeCwNzNLAYe9mVkKOOzNzFLg8Lwax6wdPb+tutAlmLWae/ZmZingsDczSwGHvZlZCjjszcxSoM0naCVdBczKahoInEVmwvHtSdvaiDg3WX8emVmq3gW+HRHr2nps6zwKebJy7KSCHdqsy2lz2EfEPGAegKQjycxm9XugKiIaTHYiaTIwDigDJgH3ACe19dhmZtY6+RrG+Sbw64g4cJDlZwELI+JARCwHSiUdk6djm5lZM/J1nf2FZAIfoK+k14G3gB9ExFpgEJnhnTpVwLHAX7J3Imk2MBtg8ODBeSotHXztt5kdSs49e0ljgJqIeA0gIkoiYijwr8DiZLWeQG3WZrXAx433FRELIqI8IspLS0tzLc3MzBL5GMa5iMwYfAMR8SugWNJRwNvAgKzF/YHKPBzbzMxaIKewl3QE8M/AQ8nzfkm4I+mrwJ6IeA94HJgpqZukCmBzROzJrXQzM2upXMfspwG/i4i9yfNjgcck1ZLpzZ+TtC8GJgDbgGpgRo7HNTOzVsgp7CPiXuDerOcbgCFNrFcLzEkeZmbWwfwNWjOzFHDYm5mlgMPezCwFHPZmZingsDczSwGHvZlZCjjszcxSwBOO55FvRmZmnZV79mZmKeCwNzNLAYe9mVkKOOzNzFLAYW9mlgIOezOzFHDYm5mlgMPezCwFcp2W8E1JW5PH6qTtMklvSdqUTE1Yt+48SZWSNiSTlJuZWQfJ+Ru0ETGs7mdJQ4HvAScAg4AVko4DxgPjgDJgEpkJyk/K9dhmZtYy+R7GORN4KCLej4iNwJvAGOAsYGFEHIiI5UCppGMabyxptqS1ktbu2rUrz6WZmaVXrmG/X9LrktZI+gqZ3vz2rOWVZCYhb9xelbQ3EBELIqI8IspLS0tzLM3MzOrkOuH4KABJ44HFwK+A2qxVaoGPgZ4HaTczsw6Ql2GciFhNZsjmbWBA1qKBwI4m2vuT6fWbmVkHaHPYSzpC0rHJz58nMyzzJHCupE9JGgUcDbwIPA7MlNRNUgWwOSL25F6+mZm1RC7DOJ8C/iCpG/BfwHkR8aykB4BXgBpgVkSEpMXABGAbUA3MyLFuMzNrhTaHfUTsAkY00X4jcGOjtlpgTvIwM7MO5m/QmpmlwGE5LaGnBzQza8g9ezOzFHDYm5mlgMPezCwFHPZmZingsDczSwGHvZlZCjjszcxSwGFvZpYCDnszsxRw2JuZpYDD3swsBRz2ZmYp4LA3M0uBXGaqKpK0QNImSdslzU3aP5K0NXk8mLX+PEmVkjZIGpOP4s3MrGVyucXxEcAy4GKgL/CKpF8DVRExLHtFSZOBcUAZMAm4Bzgph2ObmVkrtLlnHxHVEfFwZOwmM7H4UQdZ/SxgYUQciIjlQKmkY9p6bDMza528jNlL+ixQBLwM9JX0uqSVksqTVQYB27M2qSIzQXnj/cyWtFbS2l27duWjNDMzIw9hL6kf8EvggqSXXxIRQ4F/BRYnq/UEarM2qwU+bryviFgQEeURUV5aWppraWZmlsgp7CX1AX4D/Cgi/py9LCJ+BRRLOgp4GxiQtbg/UJnLsc3MrOVyuRqnN/AYcENEPJG09UvCHUlfBfZExHvA48BMSd0kVQCbI2JP7uWbmVlL5HI1zhzgZOA2SbclbWcDSyTVkunNn5O0LwYmANuAamBGDsc1M7NWanPYR8T1wPVNLBrSxLq1ZP44zGnr8czMrO38DVozsxRw2JuZpYDD3swsBRz2ZmYp4LA3M0sBh72ZWQo47M3MUsBhb2aWAg57M7MUcNibmaWAw97MLAUc9mZmKeCwNzNLAYe9mVkKOOzNzFKgw8Je0r9IekPSVkn/o6OOa2Zmuc1U1WKSSoBbgFPITDT+oqTfRMSujji+mVnadVTP/ivAHyKiKiL+AjwFfLmDjm1mlnod0rMHBgHbs55XAsc2XknSbGB28nSvpE1tPF4/YHcbt21Prqt1Dl3XrFs6rpKGuub7VTiuqzVm3ZJrXcc11dhRYd8TqM16XktmOKeBiFgALMj1YJLWRkR5rvvJN9fVOq6rdVxX66Stro4axnkbGJD1fCCwo4OObWaWeh0V9suAr0j6tKRjgFOB33fQsc3MUq9DhnEiYqek/wk8nzR9PyI+aMdD5jwU1E5cV+u4rtZxXa2TqroUEe2xXzMz60T8DVozsxRw2JuZpcBhGfaSiiWNKHQdZmadxWEV9pJ6S1oC7ASuLHQ9dSQVSVogaZOk7ZLmFromAEmfkLRc0uaktq8UuqY6knpK2ijp7kLXkk3Sm8n9nbZKWl3oeupIOlLSg5KqJL0uqWcnqOmqrPdqq6QaSV8rdF0Aki6XtCW5X9f3Cl1PHUk/yvr/8Rt53ffhdIJWUi/gi8AQ4JSImFXgkgCQ1BeYCDwC9AVeAcojoqDfNZAk4JiIeFvSfweu7yxfMpF0LfAF4P91lt8jZMI+IsoKXUdjku4HNgM3AJ8EPoxO9D+3pCOB9cCIiDhQ4FrKgFXACUARsA3o385XCDZL0iRgHpms6Ac8A5wQEXvzsf/DqmcfEXsj4kmgoP+YGouI6oh4ODJ2k/lC2VGdoK6IiLeTp8cBLxWynjqSRgH/DXio0LV0BVnfXbkx+Z3WdKagT3wT+HWhgz7xUfLfWjKXn+8F/la4cuqVAysiYn/SEfxPMp3XvDiswr4rkPRZMr2JlwtdC4CkKyVVA3OB/9UJ6hHwM+CyQtdyEPuTYZI1nWjY6wTgDeDh5OP/zcn72JlcCNxb6CIAIqIKuBZYA6wApkfER4fcqGO8AkyV1EvSscDngdJ87dxh34Ek9QN+CVzQWXpeEfG/I6Iv8CNgWScIiUuAVRGxtcB1NCkiRkXEUOAHwCJJBf+EBnwaGA1cCpwMfAn454JWlEXSGKAmIl4rdC2QObcHzCDTofg/wBWSOuo+YQcVEUuB3wFrgTvI9Oyr87V/h30HkdQH+A3wo4j4c6HraSwiHgF6kTmnUEjfAs6V9CKZTxpnSvpBgWv6BxGxGngTKCtsJQC8A6yLiMpk3Hk5cHyBa8p2EXBPoYvIch7wnxGxKiLuS9oqCllQnYj4cUSMjIizyNxDLG9/IB32HSDpSTwG3BARTxS6njqSPpOM9yJpLJneV0Fv+RoRp0bEP0XEScDVwOKIuKmQNdWRdETy8RpJnydzm+4tha0KyAxHjJbUX9IngSlkeocFJ+kIMp8yOtP5lxrgJEk9komVRgDvFrgmJHVP3q+6272/kc+LOAr+0SWfkl/ceqAEKJI0EbgoIlYWtDCYQ+bj9W2SbkvapkbEtgLWBJmTxL+T1I1M73Bagevp7D4F/CF5v/4LOK/QV3AARMQHki4l06P/JLCwE/ybrzMN+F2+rijJkweAyWSuwtkP/FtErClsSUDm39c6ScVkLpbI6/Sth9Wll2Zm1jQP45iZpYDD3swsBRz2ZmYp4LA3M0sBh72ZWQo47M3MUsBhb2aWAg57M7MU+P8NCdB1AhpivwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(df[\"F_INCOME\"], alpha =0.5, label = 'without weight')\n",
    "plt.hist(df[\"F_INCOME\"], weights=df[\"WEIGHT_W66\"], alpha = 0.5, label = 'with weight')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 161,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T15:38:40.318163Z",
     "start_time": "2020-05-13T15:38:40.144117Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXsAAAD6CAYAAABApefCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAXGElEQVR4nO3de3DV5Z3H8fdXApNUoNziiAQJchHQtY6hFyhBiQRrx6Gi4yJUQVaKty0syjrqbr10EZmKYyt1LIg1q7DjsiMXW3ARFAoqtgWFQVEuVYFkECGxKxdjhXz3j/MjHk5zOSQ5OTl5Pq+ZjJznd3meJ498zsNzfuf3M3dHRERatzPS3QAREUk9hb2ISAAU9iIiAVDYi4gEQGEvIhKArHQ3oDbdunXz/Pz8dDdDRCSjbN68+ZC75yaWt9iwz8/PZ9OmTeluhohIRjGzPTWVaxlHRCQACnsRkQAo7EVEAtBi1+xFpHl99dVXlJaWUllZme6mSBKys7PJy8ujbdu2Se2vsBcRAEpLS+nQoQP5+fmYWbqbI3Vwd8rLyyktLaV3795JHaNlHBEBoLKykq5duyroM4CZ0bVr19P6V5jCXkSqKegzx+mOlcJeRCQACnsRyQibN2/m7rvvrnHbZZddxuuvv97MLUrOgw8+yIYNG+rcp6SkhMmTJ6e0HfqAVhpn7SPpq3vEvemrOwCPr97ZpOebXty/UccXFBRQUFAAxD5MXrx4MXfeeWdTNO0US5cupVu3bhQWFjbJ+R588MFGHf/www9zyy230K1bt0adRzN7Eck4u3fvZuXKlSk59/Lly9mzp8Y7DqTF008/3SSXwyrsRaRFKCoqYv369QBMnTqVn/70pwCsWrWKG264gXXr1jFy5Eh27drFj3/8Y95880369u3L4cOHAVi9ejXf+ta36NmzJ6+99hoAX3zxBbfeeiv9+vWjX79+zJs3D6D6XCfddNNNLFy4kNmzZ7NkyRJmzJjBbbfddlrtA3j00UcZNGgQAwcOZNmyZaecG+Ctt97ioosuYtCgQdx111307du3+vx//etfueqqq+jevTvTpk0DYMyYMZSVlVFYWEhJSUmjfr9axhGRFmHUqFGsX7+e4cOHs3XrVv72t78B8Prrr1NcXFy9X79+/Vi0aBEzZ85kzZo11eWlpaVs3bqV3/72t8yaNYuioiJmz57N8ePH2blzJxUVFQwZMoQhQ4bU2oZ77rmHDz74gJEjR1YHeLLte+2113jnnXfYtm0bFRUVFBQUMHr06Orjjx8/zrhx43juuecoLCzkgQceOOX8b775Jlu2bCErK4s+ffowY8YMli5dSn5+Phs2bCAvL6/hv1w0sxeRFqK4uJj169dz4MABcnNz6dixI59++ilvvPHGKbPw2owbNw6AwsJC9u3bB8DKlSuZOnVq9XXp1157LevWrUtJ+1auXMm6deu44IILKCws5Msvv+TAgQPVx+/cuZNOnTpVfxaQ+GZSVFTEWWedRZcuXRg0aBClpaUNamdtNLOXRtn4YXna6h4yIm1VSwpccskl7Nq1i1WrVlFcXMxnn33G6tWrOXLkCD169GDXrl11Hp+dnQ1A27ZtOXHiBBCbTZ9xxtdzWnenTZs2ZGVlVe8DsVtFNLZ9x48f5957761e3kl07Ngx2rVrV2udJ9uf2Iemopm9iLQIZsZ3v/tdfv3rXzNq1ChGjBjB3Llz+f73v/93++bk5FBRUUFVVVWd5ywuLmbu3Lm4OxUVFSxfvpyRI0fSq1cvduzYwZdffkllZSUbN2485dyHDh3C3U+rfcOGDeO5557jyJEjuHv1+v5JAwYMYPfu3bzzzjsAzJ8/P6nfS23tOV2a2YtIjRp7qWRDFBcXM3v2bHr37s25557L+++/z/333/93+1188cW0a9eO8847j23bttV6vvvvv5/bb7+dPn36kJOTw89+9jPOP/98ILaMMnToUAYNGkT//l/3dezYsVx33XV89NFH/OpXv0q6fddeey1vvPEGAwYMICcnhylTpjB8+PDqY9u3b8+8efO4+uqrOfPMMxk/fjxt2rSp93cyadIkRo0axWOPPcaNN95Y7/61sca+W6TK4MGDXU+qavk2PjMjbXUPuXlO2upujd5//30GDhyY7mYEY8OGDTzwwAPVVw41RE1jZmab3X1w4r5axhERaSbr1q3D3amsrOTRRx/lhz/8YbPVrbAXEWkmTz/9NOeccw4DBgygd+/etX6YmwpasxcRaSaLFi1KW92a2YuIBEBhLyISAIW9iEgAFPYiIgFQ2ItIRkjHw0u++OILRo8eXe83dVvyw1NO0tU4IlKzpn4wTSMfNtNcDy+Jl5OTw0svvdTg45urncnQzF5EMk4qH17SlFpSOxX2ItIipOLhJSft3buXPn36AFBVVUXnzp3ZvHkzELs18po1aygvL+dHP/oR559/PpdffjmffPIJAFlZXy+APPTQQ+Tn5zNkyBDGjRvHzJkzq7cl1l9bO9NFYS8iLcLJh4MAbN26lZP3xqrt4SVDhw5l9+7ddOjQAfj64SUPPfQQs2bNOuXc5557Lm3btqW0tJS3336bTp06sXbtWiD2WcCwYcOYPn06t956Kzt27OC6667jkUdOXcZ65ZVXWL58Oe+++y6rVq3i3XffPWV7Yv21tTNdkgp7M2tnZtvNbEH0epqZ7TWzHWZ2Zdx+s82s1My2mVlBVJZlZiVmVmZmb5lZ79R0RUQyWSoeXlLT+V999VXuuece1q5dy549e8jPzyc7O5uXX36Z6dOnM2DAAObMmUNZWdkpx69Zs4aJEyfSvn17OnbseMpTqJKpP92S/YD2PuBjADPrA9wBXAD0BNaYWS+gEBgG5AMjgGeAi4EJQDaQB0wGHgeubqoOiEjrkIqHl8QrLi5m5cqVlJaW8sILL7BgwQLWrl3LqFGjgNjDRP70pz/RsWPHGs9f08NHcnJykq4/3eqd2ZvZQODbwOKoaAyw2N0Pu/t2Ym8CBcA1QIm7H3f31UCumZ0dlS/w2L2UFwH1v0WLSHBS8fCSeCNGjOCPf/wjVVVVtG/fnsGDB/Ob3/ymeolo2LBhPPXUUwB8+umnbN++/ZTjv/Od77Bw4UIqKys5dOgQL774Yr11NqSdqVLnzN7MDHgCuI3YrB1is/n4xapSoHtUvjyuvCyufA+Aux8zs2Nm1tndP6uhvinAFIitsYlIGjXyUsmGaOqHl8Tr0KED7du3r37zKCoqYsmSJVx00UUAPPHEE0ycOJGnnnqKzp078+yzz55y/Pjx41m3bh19+vShb9++DB8+vN6HjyS2M53r9nU+vMTMbgO6uPvDZnYTscD/Ctjk7s9E+7wA/BexN4Q57v5qVP4WseD+b+Aqd/9LVP4J0N/dP6+rYXp4SWbQw0taDz285PTcfPPNXHrppUyYMCFtbWjKh5fcCFxvZluAnxNbwvkE6BG3Tx6wD9ifUH4OsVl/dbmZ5QBZ9QW9iEhLU1ZWVv25wfbt21mxYgVFRUVpblXy6lzGcfehJ/8cN7P/PfC8mc0BegFdgC3ACuB2M1sEFAE73b3CzFYAk4D1wA3AshT0Q0QkpY4ePcro0aM5evQo3/zmN5k3bx55eXnpblbSTvt2Ce6+2cwWAu8BlcBkd3czWwpcCnwIlAPjo0OeBJ41s33RtrFN0nIRkWbUv39/Pvjgg3Q3o8GSDnt3LwFKoj/PAmYlbK8CpkY/8eWVwLhGtlNEmoG7E7suQ1q6uj5vrYm+QSsiQOw68fLy8tMOEWl+7k55eXn1tf3J0F0vRQSAvLw8SktLOXjwYLqbIknIzs4+rc8MFPYiAsS++dm7t+5m0lppGUdEJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAAKexGRACjsRUQCoLAXEQlAVrobkAqPr96ZlnqnF/dPS70iIvXRzF5EJAD1hr2ZnWFmq81sp5ntMLMrovJpZrY3Krsybv/ZZlZqZtvMrCAqyzKzEjMrM7O3zKx36rokIiKJklnGcWCCu+83sx8AD5vZbuAO4AKgJ7DGzHoBhcAwIB8YATwDXAxMALKBPGAy8DhwddN2RUREalPvzN5j9kcvewFbgTHAYnc/7O7bgY+BAuAaoMTdj7v7aiDXzM6Oyhe4uwOLgJFN3xUREalNUmv2Zna3mZUD04GfE5vN74nbpRToXkN5WWK5ux8DjplZ5xrqmWJmm8xs08GDBxvQHRERqUlSYe/uv3D3rsB9wCqgHVAVt0sVcKIB5Yn1zHf3we4+ODc393T6ISIidTitq3HcfQnQHtgP9IjblAfsq6H8HGKz/upyM8sBstz984Y3W0RETkcyV+OcF627Y2ZDgEpgBXC9mX3DzAYCXYAtUflEM2tjZsXATneviMonRae8AVjW9F0REZHaJHM1Tifgf82sDfApMNbdN5vZQuA9YuE/2d3dzJYClwIfAuXA+OgcTwLPmtm+aNvYJu6HiIjUod6wd/e3gb/7aqi7zwJmJZRVAVOjn/jySmBco1oqIiINpm/QiogEQGEvIhIAhb2ISAAU9iIiAVDYi4gEQGEvIhIAhb2ISAAU9iIiAWiVjyX83t75aap5TprqFRGpm2b2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIAOoNezPLNrP5ZrbDzPaY2fSofJqZ7Y3Kr4zbf7aZlZrZNjMriMqyzKzEzMrM7C0z6526LomISKKsJPY5E1gF3AJ0Bd4zs7eBO4ALgJ7AGjPrBRQCw4B8YATwDHAxMAHIBvKAycDjwNVN2REREaldvTN7dy939xc95hCwDxgOLHb3w+6+HfgYKACuAUrc/bi7rwZyzezsqHyBuzuwCBhZU11mNsXMNpnZpoMHDzZJB0VE5DTX7M3sQmIz9G7AnrhNpUB3YrP8+PKyxHJ3PwYcM7POied39/nuPtjdB+fm5p5O00REpA5Jh72ZdQOeByYB7YCquM1VwIkGlIuISDNIKuyjWfjvgPvc/c/AfqBH3C55xJZ3EsvPITbrry43sxwgy90/b3TrRUQkKclcjdMReAl42N1fjopXANeb2TfMbCDQBdgSlU80szZmVgzsdPeKqHxSdOwNwLIm7oeIiNQhmatxpgKXAL80s19GZaOAhcB7QCUw2d3dzJYClwIfAuXA+Gj/J4FnzWxftG1s03VBRETqU2/Yu/tMYGYNm2ZFP/H7VhF7c5iaUF4JjGt4M0VEpDGSmdmLSEux9pH01Dvi3vTUK01Gt0sQEQmAwl5EJAAKexGRACjsRUQCoLAXEQmAwl5EJAC69FIkg2z8sDwt9Q4ZkZZqpQlpZi8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBSDrszSzHzPqnsjEiIpIaWfXtYGYdgeeAImAxMDkqnwbcBXwB/Iu7vxyVzwZuAD4DbnL3zWaWBSwAioF9wDh3/6jpuyMi0kTWPpKeekfcm5LTJjOzrwLmAneeLDCzPsAdwAXAGOAZM2trZkXAMCA/2v+Z6JAJQDaQF5U93kTtFxGRJNQb9u5+xN1fBY7HFY8BFrv7YXffDnwMFADXACXuftzdVwO5ZnZ2VL7A3R1YBIxs4n6IiEgdGvoBbU9gT9zrUqB7DeVlieXufgw4ZmadE09qZlPMbJOZbTp48GADmyYiIokaGvbtiC3vnFQFnGhA+Sncfb67D3b3wbm5uQ1smoiIJGpo2O8HesS9ziP2wWti+TnEZv3V5WaWA2S5++cNrFtERE5TvVfj1GIF8LyZzQF6AV2ALVH57Wa2iNjVOzvdvcLMVgCTgPXErtRZ1uiWi4ik0MYPy9NS75ARqTlvMpdedgDeAToA2WZ2GfATYCHwHlAJTHZ3N7OlwKXAh0A5MD46zZPAs2a2L9o2ton7ISIidag37N39MNC3hk1rgVkJ+1YBU6Of+PJKYFzDmykiIo2h2yWIiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiARAYS8iEgCFvYhIABT2IiIBUNiLiASg2cLezP7RzD4ys91m9k/NVa+IiEBWc1RiZh2Ax4DvASeALWb2O3c/2Bz1i4iErrlm9lcAf3D3Mnf/BHgNuLyZ6hYRCZ65e+orMZsOdHP3f4te/wLY7+6PJ+w3BZgSvTwf2NHAKrsBhxp4bEvTWvrSWvoB6ktL1Vr60th+9HL33MTCZlnGAdoBVXGvq4gt55zC3ecD8xtbmZltcvfBjT1PS9Ba+tJa+gHqS0vVWvqSqn401zLOfqBH3Os8YF8z1S0iErzmCvtVwBVmdpaZnQ0MBV5pprpFRILXLMs47n7AzP4N2BgV3eXuR1NYZaOXglqQ1tKX1tIPUF9aqtbSl5T0o1k+oBURkfTSN2hFRAKgsBcRCUCrCHszyzGz/uluR2O1ln6ISMuT0WFvZh3NbBlwALi7hu0XmtlWM9tjZnPNrEX2N4l+lJhZWXRfod1mdm7ztzI5ZpZtZvPNbEf0e5+esD0jxgSS6ktGjIuZnWFmq81sZ9SXKxK2Z9KY1NeXjBiTk8ysnZltN7MFCeVNPybunrE/QHtit12YDCyoYft64EqgDfAH4Op0t7mB/SgBLkt3O5PsS1fgWsCIfRPwANAz08Ykyb5kxLhE7e8e/fkHwKaE7Zk0JvX1JSPGJK69DwIrE//ep2JMWuw7eDLc/Yi7vwocT9xmZrlAb3d/2d1PAIuI/c/R4tTVj0zj7uXu/qLHHCL25blOkFljAnX3JZNE7d8fvewFbD25LQPHpNa+ZBozGwh8G1icUJ6SMcnosK9HHrA37nUp0D1NbWmsr4D/NLP3zOyudDcmWWZ2IZANvBsVZeyY1NAXyKBxMbO7zawcmA78PG5Txo1JHX2BDBkTMzPgCWBaDZtTMiatOeyTuh9PJnD3n7h7L2Lv7j8xs5HpblN9zKwb8DwwyaN/l5KhY1JLXzJqXNz9F+7eFbgPWBWFDWTgmNTRl0wak1uBde6+u4ZtKRmT1hz2re5+PO6+D/g9cGG621IXM+sM/A64z93/HLcp48akjr5Uy5RxAXD3JcQ+I+oaFWXcmJxUQ1/it7X0MbkRuN7MthD718kYM/vXaFtKxqTVhr277wWOmtllZtaG2C/3f9LcrAYxs77Rf7sSm7HUGDotgZl1BF4CHnb3l+O3ZdqY1NWXaHtGjIuZnRfdkwozGwJURp9BZOKY1NqXqCwjxsTdh7r7P7j7xcD9wFJ3fzTalpoxSfen0Y35AToAu4ldJfF/0Z/HADOi7ZcA24i9K/5HutvbiH6sBD4mdn//f053e+vpy78DR6M+nPy5K9PGJMm+ZMS4RL/zncBfiN2fqiAT/54k2ZeMGJOEPt0ELEj1mOjeOCIiAWi1yzgiIvI1hb2ISAAU9iIiAVDYi4gEQGEvIhIAhb2ISAAU9iIiAVDYi4gE4P8BAbd2ffB+Wr0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(df[\"COVIDCOVER1_W66\"], alpha =0.5, label = 'without weight')\n",
    "plt.hist(df[\"COVIDCOVER1_W66\"], weights=df[\"WEIGHT_W66\"], alpha = 0.5, label = 'with weight')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sampling with weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:32:16.415270Z",
     "start_time": "2020-05-13T10:32:16.382893Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>QKEY</th>\n",
       "      <th>Device_Type_W66</th>\n",
       "      <th>LANG_W66</th>\n",
       "      <th>FORM_W66</th>\n",
       "      <th>COMATTACH_W66</th>\n",
       "      <th>MH_TRACK_a_W66</th>\n",
       "      <th>MH_TRACK_b_W66</th>\n",
       "      <th>MH_TRACK_c_W66</th>\n",
       "      <th>MH_TRACK_d_W66</th>\n",
       "      <th>MH_TRACK_e_W66</th>\n",
       "      <th>...</th>\n",
       "      <th>F_PARTYSUM_FINAL</th>\n",
       "      <th>F_INCOME</th>\n",
       "      <th>F_INCOME_RECODE</th>\n",
       "      <th>F_REG</th>\n",
       "      <th>F_IDEO</th>\n",
       "      <th>F_ACSWEB</th>\n",
       "      <th>F_VOLSUM</th>\n",
       "      <th>F_HISP_ORIGIN</th>\n",
       "      <th>F_YEARSINUS</th>\n",
       "      <th>WEIGHT_W66</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>2.019011e+11</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.284252</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8833</th>\n",
       "      <td>2.019010e+11</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.476299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4964</th>\n",
       "      <td>2.018011e+11</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>7.476299</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2919</th>\n",
       "      <td>2.017012e+11</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.243243</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3452</th>\n",
       "      <td>2.018010e+11</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.504054</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 119 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              QKEY  Device_Type_W66  LANG_W66  FORM_W66  COMATTACH_W66  \\\n",
       "9997  2.019011e+11              1.0       1.0       1.0            2.0   \n",
       "8833  2.019010e+11              3.0       1.0       1.0            3.0   \n",
       "4964  2.018011e+11              2.0       1.0       1.0            1.0   \n",
       "2919  2.017012e+11              3.0       1.0       2.0            1.0   \n",
       "3452  2.018010e+11              2.0       1.0       2.0            4.0   \n",
       "\n",
       "      MH_TRACK_a_W66  MH_TRACK_b_W66  MH_TRACK_c_W66  MH_TRACK_d_W66  \\\n",
       "9997             3.0             2.0             3.0             4.0   \n",
       "8833             1.0             1.0             1.0             2.0   \n",
       "4964             1.0             1.0             1.0             3.0   \n",
       "2919             1.0             1.0             1.0             4.0   \n",
       "3452             2.0             1.0             2.0             1.0   \n",
       "\n",
       "      MH_TRACK_e_W66  ...  F_PARTYSUM_FINAL  F_INCOME  F_INCOME_RECODE  F_REG  \\\n",
       "9997             3.0  ...               2.0       5.0              2.0    1.0   \n",
       "8833             1.0  ...               2.0       1.0              3.0    1.0   \n",
       "4964             1.0  ...               1.0       3.0              3.0    3.0   \n",
       "2919             1.0  ...               1.0       2.0              3.0    1.0   \n",
       "3452             3.0  ...               2.0       4.0              2.0    1.0   \n",
       "\n",
       "      F_IDEO  F_ACSWEB  F_VOLSUM  F_HISP_ORIGIN  F_YEARSINUS  WEIGHT_W66  \n",
       "9997     5.0       1.0       1.0            2.0          4.0    0.284252  \n",
       "8833     3.0       2.0       2.0            6.0          5.0    7.476299  \n",
       "4964     3.0       1.0       2.0            6.0          5.0    7.476299  \n",
       "2919     3.0       1.0       2.0            6.0          5.0    4.243243  \n",
       "3452     3.0       1.0       2.0            1.0          1.0    2.504054  \n",
       "\n",
       "[5 rows x 119 columns]"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df2 = df.sample(frac=0.66, weights=df['WEIGHT_W66'], random_state = 2020) \n",
    "df2.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:32:16.990039Z",
     "start_time": "2020-05-13T10:32:16.978774Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F_INCOME    9.632119\n",
      "dtype: float64 \n",
      " F_INCOME    9.897722\n",
      "dtype: float64 \n",
      " F_INCOME    9.887926\n",
      "dtype: float64\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "print(df[['F_INCOME']].apply(np.average, weights=df['WEIGHT_W66']),'\\n',\n",
    "df[['F_INCOME']].apply(np.average),'\\n',\n",
    "df2[['F_INCOME']].apply(np.average)    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:15:38.414051Z",
     "start_time": "2020-05-13T10:15:38.406957Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10139,\n",
       " 2028,\n",
       " F_INCOME    9.970414\n",
       " dtype: float64)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df), len(df2), df2[['F_INCOME']].apply(np.average)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T10:28:27.247530Z",
     "start_time": "2020-05-13T10:28:27.243432Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Weighted Regression: WLS vs. GLM\n",
    "\n",
    "\n",
    "The weights are presumed to be (proportional to) the inverse of the variance of the observations. That is, if the variables are to be transformed by 1/sqrt(W) you must supply weights = 1/W.\n",
    "\n",
    "> sm.WLS?\n",
    "\n",
    "https://www.coursera.org/learn/fitting-statistical-models-data-python\n",
    "\n",
    "https://www.coursera.org/lecture/fitting-statistical-models-data-python/should-we-use-survey-weights-when-fitting-models-Qzt5p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "freq_weights : array_like\n",
    "\n",
    "    1d array of frequency weights. The default is None. If None is selected\n",
    "    or a blank value, then the algorithm will replace with an array of 1's\n",
    "    with length equal to the endog.\n",
    "    WARNING: Using weights is not verified yet for all possible options\n",
    "    and results, see Notes.\n",
    "    \n",
    "var_weights : array_like\n",
    "\n",
    "    1d array of variance (analytic) weights. The default is None. If None\n",
    "    is selected or a blank value, then the algorithm will replace with an\n",
    "    array of 1's with length equal to the endog.\n",
    "    WARNING: Using weights is not verified yet for all possible options\n",
    "    and results, see Notes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:08:50.649767Z",
     "start_time": "2020-05-13T16:08:50.638618Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "without weight, the mean value is:  50.5\n",
      "with weight, the mean value is:  67.0\n",
      "23.687784005919827\n",
      "p\n",
      "0.25    50\n",
      "0.50    71\n",
      "0.75    87\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df1=pd.DataFrame({ 'x':range(1,101), 'wt':range(1,101) })\n",
    "\n",
    "from statsmodels.stats.weightstats import DescrStatsW\n",
    "wdf = DescrStatsW(df1.x, weights=df1.wt, ddof=1) \n",
    "print('without weight, the mean value is: ', np.mean(df1.x))\n",
    "print( 'with weight, the mean value is: ', wdf.mean )\n",
    "print( wdf.std )\n",
    "print( wdf.quantile([0.25,0.50,0.75]) )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:34:56.125458Z",
     "start_time": "2020-05-13T16:34:56.058487Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            OLS Regression Results                            \n",
      "==============================================================================\n",
      "Dep. Variable:           COVIDFOL_W66   R-squared:                       0.123\n",
      "Model:                            OLS   Adj. R-squared:                  0.123\n",
      "Method:                 Least Squares   F-statistic:                     158.4\n",
      "Date:                Thu, 14 May 2020   Prob (F-statistic):          1.33e-281\n",
      "Time:                        00:34:56   Log-Likelihood:                -9776.3\n",
      "No. Observations:               10139   AIC:                         1.957e+04\n",
      "Df Residuals:                   10129   BIC:                         1.964e+04\n",
      "Df Model:                           9                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===================================================================================\n",
      "                      coef    std err          t      P>|t|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------\n",
      "const               1.8951      0.046     41.361      0.000       1.805       1.985\n",
      "F_INCOME           -0.0269      0.003     -8.709      0.000      -0.033      -0.021\n",
      "F_AGECAT           -0.1573      0.007    -23.487      0.000      -0.170      -0.144\n",
      "edu                 0.0226      0.005      4.746      0.000       0.013       0.032\n",
      "republic            0.0203      0.018      1.150      0.250      -0.014       0.055\n",
      "COVIDCOVER1_W66     0.1493      0.007     20.698      0.000       0.135       0.163\n",
      "MH_TRACK_a_W66     -0.0657      0.009     -7.725      0.000      -0.082      -0.049\n",
      "MH_TRACK_b_W66      0.0125      0.009      1.325      0.185      -0.006       0.031\n",
      "MH_TRACK_d_W66     -0.0012      0.003     -0.360      0.719      -0.008       0.005\n",
      "MH_TRACK_d_W66     -0.0012      0.003     -0.360      0.719      -0.008       0.005\n",
      "MH_TRACK_e_W66      0.0038      0.007      0.550      0.583      -0.010       0.017\n",
      "==============================================================================\n",
      "Omnibus:                      985.955   Durbin-Watson:                   2.025\n",
      "Prob(Omnibus):                  0.000   Jarque-Bera (JB):             1306.676\n",
      "Skew:                           0.824   Prob(JB):                    1.81e-284\n",
      "Kurtosis:                       3.613   Cond. No.                     4.31e+16\n",
      "==============================================================================\n",
      "\n",
      "Warnings:\n",
      "[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.\n",
      "[2] The smallest eigenvalue is 4.57e-28. This might indicate that there are\n",
      "strong multicollinearity problems or that the design matrix is singular.\n"
     ]
    }
   ],
   "source": [
    "# 'COVIDFOL_W66. How closely have you been following news about the outbreak of the coronavirus\n",
    "import statsmodels.api as sm\n",
    "X = sm.add_constant(df[['F_INCOME','F_AGECAT', 'edu', 'republic','COVIDCOVER1_W66', 'MH_TRACK_a_W66', \n",
    "                        'MH_TRACK_b_W66', 'MH_TRACK_d_W66', 'MH_TRACK_d_W66', 'MH_TRACK_e_W66']])\n",
    "y = df['COVIDFOL_W66']\n",
    "reg = sm.OLS(y,X, freq_weights=df['WEIGHT_W66'])\n",
    "results = reg.fit()\n",
    "reg1 = sm.GLM(y,X)\n",
    "results1 = reg1.fit()\n",
    "reg2 = sm.GLM(y,X, freq_weights=df['WEIGHT_W66'])\n",
    "results2 = reg2.fit()\n",
    "print(results.summary()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:35:03.663599Z",
     "start_time": "2020-05-13T16:35:03.650838Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 Generalized Linear Model Regression Results                  \n",
      "==============================================================================\n",
      "Dep. Variable:           COVIDFOL_W66   No. Observations:                10139\n",
      "Model:                            GLM   Df Residuals:                    10129\n",
      "Model Family:                Gaussian   Df Model:                            9\n",
      "Link Function:               identity   Scale:                         0.40316\n",
      "Method:                          IRLS   Log-Likelihood:                -9776.3\n",
      "Date:                Thu, 14 May 2020   Deviance:                       4083.6\n",
      "Time:                        00:35:03   Pearson chi2:                 4.08e+03\n",
      "No. Iterations:                     3                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===================================================================================\n",
      "                      coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------\n",
      "const               1.8951      0.046     41.361      0.000       1.805       1.985\n",
      "F_INCOME           -0.0269      0.003     -8.709      0.000      -0.033      -0.021\n",
      "F_AGECAT           -0.1573      0.007    -23.487      0.000      -0.170      -0.144\n",
      "edu                 0.0226      0.005      4.746      0.000       0.013       0.032\n",
      "republic            0.0203      0.018      1.150      0.250      -0.014       0.055\n",
      "COVIDCOVER1_W66     0.1493      0.007     20.698      0.000       0.135       0.163\n",
      "MH_TRACK_a_W66     -0.0657      0.009     -7.725      0.000      -0.082      -0.049\n",
      "MH_TRACK_b_W66      0.0125      0.009      1.325      0.185      -0.006       0.031\n",
      "MH_TRACK_d_W66     -0.0012      0.003     -0.360      0.719      -0.008       0.005\n",
      "MH_TRACK_d_W66     -0.0012      0.003     -0.360      0.719      -0.008       0.005\n",
      "MH_TRACK_e_W66      0.0038      0.007      0.550      0.583      -0.010       0.017\n",
      "===================================================================================\n"
     ]
    }
   ],
   "source": [
    "print(results1.summary()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-13T16:35:04.499072Z",
     "start_time": "2020-05-13T16:35:04.488162Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 Generalized Linear Model Regression Results                  \n",
      "==============================================================================\n",
      "Dep. Variable:           COVIDFOL_W66   No. Observations:                10139\n",
      "Model:                            GLM   Df Residuals:                 10129.00\n",
      "Model Family:                Gaussian   Df Model:                            9\n",
      "Link Function:               identity   Scale:                         0.47043\n",
      "Method:                          IRLS   Log-Likelihood:                -10559.\n",
      "Date:                Thu, 14 May 2020   Deviance:                       4765.0\n",
      "Time:                        00:35:04   Pearson chi2:                 4.76e+03\n",
      "No. Iterations:                     3                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===================================================================================\n",
      "                      coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------\n",
      "const               1.9164      0.047     40.841      0.000       1.824       2.008\n",
      "F_INCOME           -0.0221      0.003     -6.976      0.000      -0.028      -0.016\n",
      "F_AGECAT           -0.1773      0.007    -26.078      0.000      -0.191      -0.164\n",
      "edu                 0.0369      0.005      7.619      0.000       0.027       0.046\n",
      "republic           -0.0042      0.018     -0.229      0.819      -0.040       0.032\n",
      "COVIDCOVER1_W66     0.1803      0.008     23.747      0.000       0.165       0.195\n",
      "MH_TRACK_a_W66     -0.0709      0.009     -7.862      0.000      -0.089      -0.053\n",
      "MH_TRACK_b_W66      0.0104      0.010      1.073      0.283      -0.009       0.030\n",
      "MH_TRACK_d_W66     -0.0140      0.004     -3.912      0.000      -0.021      -0.007\n",
      "MH_TRACK_d_W66     -0.0140      0.004     -3.912      0.000      -0.021      -0.007\n",
      "MH_TRACK_e_W66      0.0072      0.007      0.993      0.321      -0.007       0.022\n",
      "===================================================================================\n"
     ]
    }
   ],
   "source": [
    "print(results2.summary()) "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## GLM Binomial Regression\n",
    "\n",
    "https://www.statsmodels.org/stable/glm.html\n",
    "\n",
    "![image.png](./images/pew5.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-14T05:10:17.236021Z",
     "start_time": "2020-05-14T05:10:17.165763Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 Generalized Linear Model Regression Results                  \n",
      "==============================================================================\n",
      "Dep. Variable:                      y   No. Observations:                 8022\n",
      "Model:                            GLM   Df Residuals:                  7694.52\n",
      "Model Family:                Binomial   Df Model:                            9\n",
      "Link Function:                  logit   Scale:                          1.0000\n",
      "Method:                          IRLS   Log-Likelihood:                -4259.8\n",
      "Date:                Thu, 14 May 2020   Deviance:                       8519.6\n",
      "Time:                        13:10:17   Pearson chi2:                 7.66e+03\n",
      "No. Iterations:                     4                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===================================================================================\n",
      "                      coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------\n",
      "const               2.8958      0.188     15.397      0.000       2.527       3.264\n",
      "F_INCOME            0.0275      0.012      2.200      0.028       0.003       0.052\n",
      "F_AGECAT            0.0798      0.026      3.051      0.002       0.029       0.131\n",
      "edu                -0.1463      0.019     -7.696      0.000      -0.184      -0.109\n",
      "republic           -0.5598      0.073     -7.710      0.000      -0.702      -0.417\n",
      "COVIDCOVER1_W66    -1.1353      0.039    -28.945      0.000      -1.212      -1.058\n",
      "MH_TRACK_a_W66      0.1216      0.035      3.432      0.001       0.052       0.191\n",
      "MH_TRACK_b_W66      0.0327      0.038      0.858      0.391      -0.042       0.107\n",
      "MH_TRACK_d_W66     -0.0242      0.014     -1.721      0.085      -0.052       0.003\n",
      "MH_TRACK_d_W66     -0.0242      0.014     -1.721      0.085      -0.052       0.003\n",
      "MH_TRACK_e_W66      0.0248      0.028      0.872      0.383      -0.031       0.081\n",
      "===================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Largely accurate \n",
    "from statsmodels.genmod import families\n",
    "dfs = df[df['COVIDNEWSPAIR_a_W66']!=2]\n",
    "X = sm.add_constant(dfs[['F_INCOME','F_AGECAT', 'edu', 'republic','COVIDCOVER1_W66', 'MH_TRACK_a_W66', \n",
    "                        'MH_TRACK_b_W66', 'MH_TRACK_d_W66', 'MH_TRACK_d_W66', 'MH_TRACK_e_W66']])\n",
    "y =[1 if i==1 else 0 for i in  dfs['COVIDNEWSPAIR_a_W66']]\n",
    "reg1 = sm.GLM(y,X, family=families.Binomial())\n",
    "results1 = reg1.fit()\n",
    "reg2 = sm.GLM(y,X, family=families.Binomial(), freq_weights=dfs['WEIGHT_W66'])\n",
    "results2 = reg2.fit()\n",
    "print(results2.summary()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-05-14T05:10:31.946742Z",
     "start_time": "2020-05-14T05:10:31.894609Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 Generalized Linear Model Regression Results                  \n",
      "==============================================================================\n",
      "Dep. Variable:                      y   No. Observations:                 8022\n",
      "Model:                            GLM   Df Residuals:                  7694.52\n",
      "Model Family:                Binomial   Df Model:                            9\n",
      "Link Function:                  logit   Scale:                          1.0000\n",
      "Method:                          IRLS   Log-Likelihood:                -4259.8\n",
      "Date:                Thu, 14 May 2020   Deviance:                       8519.6\n",
      "Time:                        13:10:31   Pearson chi2:                 7.66e+03\n",
      "No. Iterations:                     4                                         \n",
      "Covariance Type:            nonrobust                                         \n",
      "===================================================================================\n",
      "                      coef    std err          z      P>|z|      [0.025      0.975]\n",
      "-----------------------------------------------------------------------------------\n",
      "const              -2.8958      0.188    -15.397      0.000      -3.264      -2.527\n",
      "F_INCOME           -0.0275      0.012     -2.200      0.028      -0.052      -0.003\n",
      "F_AGECAT           -0.0798      0.026     -3.051      0.002      -0.131      -0.029\n",
      "edu                 0.1463      0.019      7.696      0.000       0.109       0.184\n",
      "republic            0.5598      0.073      7.710      0.000       0.417       0.702\n",
      "COVIDCOVER1_W66     1.1353      0.039     28.945      0.000       1.058       1.212\n",
      "MH_TRACK_a_W66     -0.1216      0.035     -3.432      0.001      -0.191      -0.052\n",
      "MH_TRACK_b_W66     -0.0327      0.038     -0.858      0.391      -0.107       0.042\n",
      "MH_TRACK_d_W66      0.0242      0.014      1.721      0.085      -0.003       0.052\n",
      "MH_TRACK_d_W66      0.0242      0.014      1.721      0.085      -0.003       0.052\n",
      "MH_TRACK_e_W66     -0.0248      0.028     -0.872      0.383      -0.081       0.031\n",
      "===================================================================================\n"
     ]
    }
   ],
   "source": [
    "# Largely inaccurate \n",
    "y =[1 if i==3 else 0 for i in  dfs['COVIDNEWSPAIR_a_W66']]\n",
    "reg1 = sm.GLM(y,X, family=families.Binomial())\n",
    "results1 = reg1.fit()\n",
    "reg2 = sm.GLM(y,X, family=families.Binomial(), freq_weights=dfs['WEIGHT_W66'])\n",
    "results2 = reg2.fit()\n",
    "print(results2.summary()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](./images/end.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}