01-Phonemes.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install -U -q folium\n",
    "import folium\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import geopy.distance\n",
    "from collections import Counter\n",
    "import matplotlib.pyplot as plt\n",
    "from IPython.display import display\n",
    "from scripts.cogsci_module import *\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "sns.set_style('darkgrid')\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [COGSCI 1B] Typology\n",
    "---\n",
    "### Professor Terry Regier\n",
    "\n",
    "This module explores a central question in cognitive science and linguistics: how do languages vary from one another? We will explore datasets of linguistic features ([PHOIBLE](http://phoible.org/) and [WALS](http://wals.info/)) to come to tentative answers to this question in a data-driven way. Example problems include visualizing the distribution of phonemes, the relationship between geography and the development of languages, and the genetic relationships of languages.\n",
    "\n",
    "---\n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "0 - [PHOIBLE Data](#PHOIBLE Data)\n",
    "\n",
    "1 - [Phoneme Distributions](#phoneme dist)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1.1 - [Consonants](#consonants)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1.2 - [Vowels](#vowels)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1.3 - [Phonemes](#phonemes)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 1.4 - [Consonants vs Vowels](#cons vs vows)<br>\n",
    "\n",
    "2 - [Phoneme Metadata](#metadata)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.1 - [Family](#phoneme fam)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.2 - [Continent](#phoneme cont)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.3 - [Latitude and Longitude](#lat lons)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.4 - [Population Size vs Phoneme Inventory Size](#pop v foam)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 2.5 - [Distance from Africa](#africa distance)\n",
    "\n",
    "3 - [Common Phonemes](#common)\n",
    "\n",
    "4 - [Tone](#tone)\n",
    "\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; 4.1 - [Altitude](#altitude)\n",
    "\n",
    "5 - [Morphological Complexity](#morph complex)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## PHOIBLE Data<a id='PHOIBLE Data'></a>\n",
    "\n",
    "We will start by familiarizing ourselves with the data and at the same time learn some common ways of analyzing data in the Python programming language. In order to do that, we need to load them into our notebook. \n",
    "\n",
    "First, we'll start with our data from [PHOIBLE](http://phoible.org/). In the code cell below, we create a variable called `file_name` that we assign to the name of our file in quotations. Note that we have `phoible_data/` in front of the file name, which means that our file `phoible_elevation.csv` is in the `phoible_data` directory (folder). We turn this file into what is called a [**DataFrame**](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html), which can be thought of as a slightly more rigid Excel sheet. It allows us to easily access, manipulate, and visualize our data.\n",
    "\n",
    "A code cell will display what is written in the last line of the cell (if it is not a variable assignment statement). So in the cell below, the last line says `phoible_data.head()`, which means that it will display our dataframe, but adding `.head()` at the end of it allows us to show only the first 5 rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = 'phoible_data/phoible_elevation.csv'  # assign the file path to a variable\n",
    "phoible_data = pd.read_csv(file_name)  # read in the CSV to pandas\n",
    "phoible_data.head()  # print the first five rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our dataframe, the column `Population` was stored as *strings*, not numbers, because some values in the column are words. As *string* is essentially text for the computer, and it can't perform all of the same mathematical operations on characters of text. The possible text entries we find in the `Population` column for those rows are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted(list(set(phoible_data['Population'])))[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use the numerical values of `Population` for further analysis, we are going to drop rows where the values are words, and convert the numbers to be represented as `ints`, or the Python representation of integers, and create a new dataframe called `phoib` with this new data. Reasons like this emphasize the importance of being aware of how your data is represented and how you store data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# phoib contains rows where population is a number\n",
    "phoib = phoible_data.copy()\n",
    "phoib[\"Population\"] = pd.to_numeric(phoib['Population'], errors='coerce')\n",
    "phoib = phoib.dropna(subset=['Population'])  # drop non integer rows\n",
    "length_difference = len(phoible_data) - len(phoib)\n",
    "print(\"When we remove those rows with text, we lose {} rows.\".format(length_difference))\n",
    "phoib.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next thing that we'll notice is that there are multiple rows for some of the languages. The cell below counts how many times a value in the column `LanguageCode` occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoible_data['LanguageCode'].value_counts().head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's weird, we might expect each language code to have a unique language name. Let's look at all of the rows where the `LanguageCode` is equal to `gwn`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# same language code, but different language name.\n",
    "cond = phoible_data['LanguageCode'] == 'gwn'\n",
    "phoible_data[cond]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that the six rows for `gwn` are the same language, just different dialects. There are cases like this, and also cases where the same language has an entry from more than one source (Korean is an example of one). Below is the number of different possible language codes, and below that, the total number of rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('There are {} unique languages in phoible_data'.format(len(list(set(phoible_data['LanguageCode'])))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('There are {} total rows in phoible_data'.format(len(phoible_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the following cell, we are going to map each of the rows from the `phoible_data` table based on their values in the `Latitude` and `Longitude` columns in our DataFrame. Don't worry if it takes a while, there are over 2000 points that have to be plotted!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mp = folium.Map(zoom_start=12)\n",
    "phoib_coords = phoible_data.dropna(subset=['Latitude', 'Longitude'])\n",
    "for coords in list(zip(phoib_coords['Latitude'], phoib_coords['Longitude'])):\n",
    "    folium.Circle(\n",
    "        radius=100,\n",
    "        location=coords,\n",
    "        color='crimson',\n",
    "        fill=False,).add_to(mp)\n",
    "mp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the distribution of languages in our data? Are these languages evenly spread throughout the world?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Phonemic Inventories\n",
    "\n",
    "Phonemes are the individual distinctive sounds in a language. For example, /a/, /f/, and /k/ are all phonemes in English. Phonemes come from the [IPA](https://en.wikipedia.org/wiki/International_Phonetic_Alphabet), which is the phonetic alphabet. Phonemes can either be consonants or vowels. Languages have a fixed number of phonemes. The phonemic inventory is the set of phonemes in a language. Languages differ in their phonemic inventories. The best data source on phonemic inventories is PHOIBLE, which we loaded above.\n",
    "\n",
    "One big question in cognitive science is determining the relationship between linguistic features (e.g. number of vowels, word order and number of tense categories) and non-linguistic features (e.g. population size, altitude and climate). In particular, a lot of attention has been paid to the relationship between population size and various linguistic features. For example, people have looked at the relationship between population size and the size of the phonemic inventory, and population size and morphological complexity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Phoneme Distributions <a id='phoneme dist'></a>\n",
    "\n",
    "Below we are going to plot the numerical distributions of the number of phonemes a language has (broken into consonants, vowels, and total phonemes), as well as the geographic distributions of those values. We'll be using histograms to visualize the numerical distributions and plot on a map for the geographic spread.\n",
    "\n",
    "### Consonants <a id='consonants'></a>\n",
    "\n",
    "We are going to first average together the values that each dialect has to get a row for each language code. Therefore, we'll use the `groupby` method followed by the `mean` to average the dialects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoib_avg = phoib.groupby('LanguageCode').mean().iloc[:,-7:].reset_index()\n",
    "phoib_avg.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.distplot(phoib_avg['Consonants'])\n",
    "plt.ylabel('proportion per consonant');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the distribution of consonants? (For reference, dialects of English have around 24 consonant phonemes)"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function splits the values that it gets for consonants into five different color categories and plots them on a map. Again, this might take a few seconds!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "map_with_bins('Consonants', phoib_avg.dropna())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that most of the dots are dark blue, except for some in southern Africa. The bins for this method are determined by taking the range from the smallest value to the largest value, then dividing that range into five evenly-sized sections. Looking back at our histogram, you'll see that almost all of our data falls into the two blue bins (values less than 55.6). So in this situation, it is hard to get much out of the visualization.\n",
    "\n",
    "To combat that, we'll divide the bins based on quantiles. That is, we'll split the bins up so that each bin contains the same number of points. The lowest fifth of values in the dark-blue bin, the second lowest fifth in the light-blue, and so on. By passing the phrase `quantiles=True` into the function, we are able to do that. Take notice of the ranges of values in each bin, and compare those to the map above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# plotting with bins set on quantiles instead\n",
    "map_with_bins('Consonants', phoib_avg.dropna(), quantiles=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Vowels <a id='vowels'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can repeat the above for `Vowels`, go ahead and write some code to bin and map the vowel distributions.\n",
    "\n",
    "First use `sns.distplot` for our `phoib_avg` dataframe for `Vowels`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does this distribution differ from the Consonant distribution?\n",
    "\n",
    "Now use the `map_with_bins` function to bin and map:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are there any areas where the colors seem significantly different between the maps for vowels and consonants?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Phonemes <a id='phonemes'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the same for total `Phonemes`.\n",
    "\n",
    "Get the distribution with `sns.distplot`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now `map_with_bins`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Consonants vs Vowels <a id='cons vs vows'></a>\n",
    "\n",
    "Next, we will visualize the relationship between number of consonants and number of vowels. The following graph plots a line-of-best-fit through the data. If you want to read more about how that line is determined, look [here](https://www.inferentialthinking.com/chapters/14/1/regression-model.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "overlay_hex(phoib_avg[\"Consonants\"], phoib_avg[\"Vowels\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What can you infer about the relationsihp between the number of consonants and the number of vowels, based purely on the line? How accurate do you think the line is? Can we trust it?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we calculate the average number of consonants per vowel that each **dialect** (note that we are using `phoib` instead of `phoib_avg`) has for a continent / geographical region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pho_cont = phoib[[\"Area\",\"Consonants\", \"Vowels\"]].copy()\n",
    "pho_cont = pho_cont.groupby(by=\"Area\").mean()\n",
    "pho_cont['Ratio'] = pho_cont['Consonants'] / pho_cont['Vowels']\n",
    "pho_cont[['Ratio']].plot.bar(figsize = (12,8), rot='horizontal')\n",
    "plt.title('Average Consonants per Vowel');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What effect could the choice to calculate these values using dialects instead of the grouped language codes have on the outcomes?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Phoneme Metadata <a id='metadata'></a>\n",
    "\n",
    "### By Family  <a id='phoneme fam'></a>\n",
    "\n",
    "We are now going to take a look at the effect that some of those non-linguistic features have on the linguistic features.\n",
    "\n",
    "In the next cell, we import our [WALS](http://wals.info/) data. There aren't any immediate problems with the data set that we have to address."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wals = pd.read_csv('wals_data/language.csv')\n",
    "wals.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Currently, we have genetic affilitation data in the column `family` in WALS, and we have the `area` column we saw previously in PHOIBLE, as well as many of the linguistic features. To do some of our analysis, we are first going to need to join our two dataframes together. WALS and PHOIBLE both identify languages with ISO 639-3, so we are able to pair rows together with that information. We will call our new dataframe `combined`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined = wals.dropna(subset=['iso_code']).merge(phoib.dropna(), left_on='iso_code', right_on='LanguageCode', how='inner')\n",
    "combined.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first visualization is a bar plot of the average number of total phonemes, consonants, and vowels for each genetic family."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('There are {} different families.'.format(len(combined.family.value_counts())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# double click on the image to zoom in (then you can scroll left or right)\n",
    "combined.groupby(by=\"family\")[['Phonemes', 'Consonants', 'Vowels']].mean().plot.bar(figsize=(50,8))\n",
    "plt.title('Mean Number of __ for Each Family')\n",
    "plt.ylabel('Count');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of different families so to get a zoomed in version, double click on the plot."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By Continent  <a id='phoneme cont'></a>\n",
    "\n",
    "We will recreate the same chart as above, except this time we will group together by `area` instead of `family`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "phoib.groupby(by=\"Area\")[['Phonemes', 'Consonants', 'Vowels']].mean().plot.bar(figsize=(12,6))\n",
    "plt.title('Mean Number of __ for Each Area')\n",
    "plt.ylabel('Count');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Latitude and Longitude <a id='lat lons'></a>\n",
    "\n",
    "Next, we'll look at the number of phonemes plotted against latitude and longitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sns.jointplot('Phonemes', 'Latitude', data=phoib, kind='hex')\n",
    "plt.title('Phonemes vs Latitude');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the syntax in the cell above, plot the `Phonemes` and `Longitude` below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are you able to make any meaningful conclusions from these two graphs?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Population Size vs. Phoneme Inventory Size <a id='pop v foam'></a>\n",
    "\n",
    "Below we visualize the relationship between population size and phoneme inventory size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "overlay_hex(phoib[\"Phonemes\"], phoib[\"Population\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get a better picture of what is going on, let's take the log of population. There are more in-depth reasons as to why you might want to take the log of a column before working with it, which you can read about [here](https://stats.stackexchange.com/questions/18844/when-and-why-should-you-take-the-log-of-a-distribution-of-numbers).\n",
    "\n",
    "Follow the syntax of the cell above, but use `np.log` around `phoib[\"Population\"]` to visualize the log population:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can you interpret the above graph?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distance from Africa <a id='africa distance'></a>\n",
    "\n",
    "It has been claimed that phoneme inventory size and distance from Africa are inversely related. We will test that. You can read the original article in [here](http://science.sciencemag.org/content/332/6027/346), and one of many responses [here](http://languagelog.ldc.upenn.edu/nll/?p=3090).\n",
    "\n",
    "We start by refering back to a subset of a graph we previously created, and look at the average number of phonemes per dialect for each area."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pho_pop_cont = phoib.loc[:,[\"Area\", \"Phonemes\"]]\n",
    "pho_pop_cont = pho_pop_cont.groupby(by = \"Area\").mean().sort_values('Phonemes', ascending=False)\n",
    "pho_pop_cont[['Phonemes']].plot.bar(figsize = (12,8))\n",
    "plt.ylabel('Average Number of Phonemes')\n",
    "pho_pop_cont"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on that bar chart, are you able to make any inferences about what will show up when we plot the distance to each individual dialect's distance to Africa and its relation to the number of phonemes?  "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate the distance from each point to Africa, we've chose the latitude/longitude pair `8.7832, 34.5085` to be our center point of the continent, and will calculate the distance to each dialect's listed location (in kilometers). We then plot the distance and phoneme values against each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "coordinates = list(zip(phoib.dropna()['Latitude'], phoib.dropna()['Longitude']))\n",
    "\n",
    "# chose this point b/c it comes up when\n",
    "# you google search 'africa coordinates'\n",
    "africa_center = (8.7832, 34.5085)\n",
    "\n",
    "# calculate the distance to each language's listed location\n",
    "distances = np.array([geopy.distance.vincenty(point, africa_center).km for point in coordinates])\n",
    "\n",
    "overlay_hex(distances, phoib.dropna()['Phonemes'])\n",
    "plt.xlabel('Kilometers from Africa');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Common Phonemes <a id='common'></a>\n",
    "\n",
    "What are the most common phonemes in the world? What is the distribution of frequency? There are a couple thousand phonemes in PHOIBLE, but only a handful are common and there's a long tail. \n",
    "\n",
    "Our previous Phoible data set contained mainly metadata and aggregated numbers. We will now pull in a second Phoible data set that contains a row for each phoneme that a language (and possibly a dialect) have."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "phonemes = pd.read_csv('phoible_data/phoible-by-phoneme.tsv', delimiter='\\t')\n",
    "phonemes.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Out of {} rows, there are {} unique phonemes.'.format(len(phonemes), len(list(set(phonemes['Phoneme'])))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the cell below, we construct a dataframe that counts the number of times that a phoneme occurs in `phonemes`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoneme_counts = pd.DataFrame.from_dict(Counter(phonemes['Phoneme']), orient='index').reset_index().sort_values(0, ascending=False)\n",
    "phoneme_counts.columns = ['Phoneme', 'Count']\n",
    "phoneme_counts.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We then plot the number of times that the top 200 phonemes occur in that table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoneme_counts.iloc[:200].plot.bar(figsize=(15, 5))\n",
    "plt.xticks([])\n",
    "plt.ylabel('Count')\n",
    "plt.xlabel('Phoneme')\n",
    "plt.title('Counts of 200 Most Common Phonemes');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can see that there is a quick drop-off in the frequency of different phonemes, and that there is a long right tail. Let's use a histogram to look at the distribution of these frequencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(15, 5))\n",
    "sns.distplot(phoneme_counts['Count'], ax=ax)\n",
    "plt.title('Distribution of Phoneme Frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does this the histogram tell us about the frequency of different phonemes?\n",
    "\n",
    "Phonemes can be described by a set of (mostly) binary features. PHOIBLE has this data too. Is the distribution of featue values evenly split for each feature? If not, which features are more prone to being either 0 or 1? Are some phonemes only present in some area of genetic affiliation?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "lc_to_area = dict(zip(phoib['LanguageCode'], phoib['Area']))\n",
    "\n",
    "def convert_code(code):\n",
    "    try:\n",
    "        return lc_to_area[code]\n",
    "    except:\n",
    "        return 'undefined'\n",
    "    \n",
    "phonemes['Area'] = [convert_code(code) for code in phonemes['LanguageCode']]\n",
    "\n",
    "pd.crosstab(phonemes['Phoneme'], phonemes['Area'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# normalizing by columns means that it accounts for the fact that\n",
    "# there are differing numbers of languages per country\n",
    "pd.crosstab(phonemes['Phoneme'], phonemes['Area'], margins=True, normalize='columns')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Tone <a id='tone'></a>\n",
    "\n",
    "One of the features of phonemes is `tone`. If a language has a phoneme with tone, it counts as a \"tone language\". Are most language tone? Where are the tone languages on the map?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tone_languages = phoib_avg['Tones'] > 0\n",
    "num_tone_languages = sum(tone_languages)\n",
    "total_languages = len(phoib_avg)\n",
    "\n",
    "print('There are {} tone languages out of our dataset of {} languages.'.format(num_tone_languages, total_languages))\n",
    "print(\"That's about {}%.\".format(np.round(num_tone_languages/total_languages*100, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to plot the tone languages, we create a table of just tone languages."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tone = phoib_avg[tone_languages]\n",
    "tone.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we plot them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "mappable_tone = tone.dropna(subset=['Latitude', 'Longitude'])\n",
    "\n",
    "mp = folium.Map(zoom_start=12)\n",
    "for coords in list(zip(mappable_tone['Latitude'], mappable_tone['Longitude'])):\n",
    "    folium.Circle(\n",
    "        radius=100,\n",
    "        location=coords,\n",
    "        color='crimson',\n",
    "        fill=False,).add_to(mp)\n",
    "mp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you notice about the locations of tone languages? How does it compare to the first map we created, where we mapped allow of the dialects?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Altitude <a id='altitude'></a>\n",
    "\n",
    "It has been claimed there is a relationship between being a tone language and the altitude. We've collected altitude from Google Maps Elevation based on the list coordinates for each language. You can read the original paper [here](http://pubman.mpdl.mpg.de/pubman/item/escidoc:2084217:2/component/escidoc:2084215/Everett%20et%20al_2015_Climate,%20vocal%20folds,%20and%20tonal%20languages.pdf).\n",
    "\n",
    "We'll ignore the rows that do not have an elevation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "phoib_avg['Tone Language?'] = phoib_avg['Tones'] > 0\n",
    "have_elevation = phoib_avg[['elevation', 'Tones', 'Tone Language?']].dropna()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will plot the histograms of elevation below for languages who are a tone language (in orange) and those that aren't (in blue)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f, ax = plt.subplots(figsize=(10, 8))\n",
    "sns.distplot(have_elevation[np.invert(have_elevation['Tone Language?'])]['elevation'], ax=ax, label='No tone')\n",
    "sns.distplot(have_elevation[have_elevation['Tone Language?']]['elevation'], ax=ax, label='Tone')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a difference between the elevation distributions for the two?"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are some pretty significant outliers that are affecting our data. Below, we'll recreate the above visualization, except this time, not including values that are three standard deviations outside of the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting rid of the 3 SD outliers to get a better picture\n",
    "no_out=have_elevation[((have_elevation['elevation'] - have_elevation['elevation'].mean()) / have_elevation['elevation'].std()).abs() < 3]\n",
    "\n",
    "f, ax = plt.subplots(figsize=(10, 8))\n",
    "sns.distplot(no_out[np.invert(no_out['Tone Language?'])]['elevation'], ax=ax, label='No tone')\n",
    "sns.distplot(no_out[no_out['Tone Language?']]['elevation'], ax=ax, label='Tone')\n",
    "plt.legend();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "## Relationship between population size and morphological complexity <a id='morph complex'></a>\n",
    "\n",
    "All the data for this part comes from WALS. Morphological complexity is a vague term, referring to how complicated the words in a language are. Below we look at some features in relation to phoneme inventory size.\n",
    "\n",
    "### Feature 30A: Number of Genders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "desired_columns = ['LanguageCode', 'Area', 'Latitude', 'Longitude', 'Population', 'Phonemes']\n",
    "\n",
    "gender_data = drop_and_subset('30A Number of Genders', combined, desired_columns)\n",
    "genders_dict = {'1 None':1, '2 Two':2, '3 Three':3, '4 Four':4, '5 Five or more':5}\n",
    "gender_data['Genders'] = [genders_dict[value] for value in gender_data['30A Number of Genders']]\n",
    "\n",
    "print('Rows with Gender data: {}'.format(len(gender_data)))\n",
    "gender_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "overlay_hex(gender_data['Genders'], gender_data['Phonemes']);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "overlay_hex(gender_data['Genders'], np.log(gender_data['Population']));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reduplication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "reduplication = drop_and_subset('27A Reduplication', combined, desired_columns)\n",
    "reduplication.groupby('27A Reduplication').mean()[['Phonemes']].plot.bar()\n",
    "plt.xticks(rotation=40);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 20A: Fusion of Selected Inflectional Formatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fusion = drop_and_subset('20A Fusion of Selected Inflectional Formatives', combined, desired_columns)\n",
    "fusion.groupby('20A Fusion of Selected Inflectional Formatives').mean()[['Phonemes']].plot.bar()\n",
    "plt.xticks(rotation=70);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 21A: Exponence of Selected Inflectional Formatives"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "exponence_a = drop_and_subset('21A Exponence of Selected Inflectional Formatives', combined, desired_columns)\n",
    "exponence_a.groupby('21A Exponence of Selected Inflectional Formatives').mean()[['Phonemes']].plot.bar()\n",
    "plt.xticks(rotation=70);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature 21B: Exponence of Tense-Aspect-Mood Inflection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exponence_b = drop_and_subset('21B Exponence of Tense-Aspect-Mood Inflection', combined, desired_columns)\n",
    "exponence_b.groupby('21B Exponence of Tense-Aspect-Mood Inflection').mean()[['Phonemes']].plot.bar()\n",
    "plt.xticks(rotation=70);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "***Please fill out our [module feedback form](https://goo.gl/forms/vncqym83o1XqMnQa2)!***"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}