02_goodness_of_embeddings_across_languages.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do you evaluate goodness of embeddings across languages?\n",
    "\n",
    "If you have embeddings for language A and another set of embeddings for language B, how can you tell how good they are? \n",
    "\n",
    "We might want to evaluate each set on a task specific to that language. But a much more interesting proposition is aligning the embeddings in an unsupervised way and evaluating the performance on translation!\n",
    "\n",
    "For this, we need 3 components. [The embeddings](https://fasttext.cc/docs/en/pretrained-vectors.html), [a way to align embeddings](https://github.com/artetxem/vecmap) and [dictionaries](https://github.com/facebookresearch/MUSE) we can use to evaluate the results.\n",
    "\n",
    "Let's begin by grabbing embeddings for English and Polish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-09-14 17:58:30--  https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en.vec\n",
      "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 172.67.9.4, 104.22.75.142, 104.22.74.142, ...\n",
      "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|172.67.9.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 6597238061 (6.1G) [binary/octet-stream]\n",
      "Saving to: ‘data/wiki.en.vec’\n",
      "\n",
      "wiki.en.vec         100%[===================>]   6.14G  55.4MB/s    in 1m 56s  \n",
      "\n",
      "2020-09-14 18:00:26 (54.5 MB/s) - ‘data/wiki.en.vec’ saved [6597238061/6597238061]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P data https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en.vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-09-14 18:00:26--  https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.pl.vec\n",
      "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.22.74.142, 172.67.9.4, 104.22.75.142, ...\n",
      "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.22.74.142|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 2707436342 (2.5G) [binary/octet-stream]\n",
      "Saving to: ‘data/wiki.pl.vec’\n",
      "\n",
      "wiki.pl.vec         100%[===================>]   2.52G  31.6MB/s    in 71s     \n",
      "\n",
      "2020-09-14 18:01:37 (36.6 MB/s) - ‘data/wiki.pl.vec’ saved [2707436342/2707436342]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P data https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.pl.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Time to align the embeddings. Let's translate from Polish to English. In order to do that, let's align our Polish embeddings with our English embeddings.\n",
    "\n",
    "Bear in mind that this is a challenging task - Polish and English come from distinct and very different language families (West Slavic languages of the Lechitic group and West Germanic language of the Indo-European language family respectively).\n",
    "\n",
    "Let us clone [vecmap](https://github.com/artetxem/vecmap) to a directory adjacent to this one and perform the alignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 17.8 s, sys: 5.69 s, total: 23.5 s\n",
      "Wall time: 26min 45s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "!python ../vecmap/map_embeddings.py --cuda --unsupervised data/wiki.pl.vec data/wiki.en.vec data/wiki.pl.aligned.vec data/wiki.en.aligned.vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We now have the English and Polish embeddings aligned to live in the same embedding space!\n",
    "\n",
    "Let's grab a Polish to English dictionary from the MUSE repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-09-14 18:28:23--  https://dl.fbaipublicfiles.com/arrival/dictionaries/pl-en.txt\n",
      "Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.22.74.142, 104.22.75.142, 172.67.9.4, ...\n",
      "Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.22.74.142|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1216386 (1.2M) [text/plain]\n",
      "Saving to: ‘data/pl-en.txt’\n",
      "\n",
      "pl-en.txt           100%[===================>]   1.16M  --.-KB/s    in 0.03s   \n",
      "\n",
      "2020-09-14 18:28:24 (34.8 MB/s) - ‘data/pl-en.txt’ saved [1216386/1216386]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P data https://dl.fbaipublicfiles.com/arrival/dictionaries/pl-en.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dictionary is just a text file with a source and target word per line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "roku\tyear\r\n",
      "jest\tis\r\n",
      "nie\tnot\r\n",
      "przez\tthrough\r\n",
      "przez\tby\r\n",
      "jako\tas\r\n",
      "oraz\tand\r\n",
      "był\twas\r\n",
      "jego\this\r\n",
      "jego\tits\r\n"
     ]
    }
   ],
   "source": [
    "!head data/pl-en.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's wrap it in a Python class to make usage of it easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "#export core\n",
    "\n",
    "class Dictionary():\n",
    "    def __init__(self, path_to_dict):\n",
    "        self.read_words(path_to_dict)\n",
    "        self.create_dict()\n",
    "        \n",
    "    def read_words(self, path_to_dict):\n",
    "        source_words, target_words = [], []\n",
    "        with open(path_to_dict) as f:\n",
    "            for line in f.readlines():\n",
    "                src, target = line.strip().split()\n",
    "                source_words.append(src)\n",
    "                target_words.append(target)\n",
    "        self.source_words = source_words\n",
    "        self.target_words = target_words\n",
    "        \n",
    "    def create_dict(self):\n",
    "        self.dict = {}\n",
    "        for src, target in zip(self.source_words, self.target_words):\n",
    "            self.dict[src] = target\n",
    "            \n",
    "    def __getitem__(self, source_word):\n",
    "        return self.dict[source_word]\n",
    "    \n",
    "    def __len__(self):\n",
    "        return len(self.source_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'queen'"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pl_en_dict = Dictionary('data/pl-en.txt')\n",
    "pl_en_dict['królowa']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "73901"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(pl_en_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thanks to [nbdev](https://github.com/fastai/nbdev), we can conveniently import the Embeddings class we defined in the earlier notebook and use it here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from embedding_gym.core import Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's limit ourselves to the most common 50 000 words from each set of embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.8 s, sys: 368 ms, total: 12.2 s\n",
      "Wall time: 11.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "embeddings_pl = Embeddings.from_txt_file('data/wiki.pl.aligned.vec', limit=50_000)\n",
    "embeddings_en = Embeddings.from_txt_file('data/wiki.en.aligned.vec', limit=50_000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see if our mechanism for translation works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['is', 'makes', 'are', 'becomes', 'has']"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "embeddings_en.nn_words_to(embeddings_pl['jest'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That it does - 'jest' means 'is' in Polish!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our translation results will be adversaly affected by synonyms. What we can however do is limit the task to words where the source and the target word exist in source and target embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 32min 53s, sys: 25.5 s, total: 33min 18s\n",
      "Wall time: 4min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "topk = 1\n",
    "source_embeddings = embeddings_pl\n",
    "target_embeddings = embeddings_en\n",
    "\n",
    "correct, total = 0, 0\n",
    "\n",
    "for source_word in source_embeddings.i2w:\n",
    "    if source_word in pl_en_dict.source_words and pl_en_dict[source_word] in target_embeddings.i2w:\n",
    "        total += 1\n",
    "        if pl_en_dict[source_word] in embeddings_en.nn_words_to(embeddings_pl[source_word], n=topk):\n",
    "            correct += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.5911755937593163, 20126)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correct / total, total"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nearly 60% accuracy with topk@1 across 20 thousand words! That seems like a great result given that we are not making any accomodations for synonyms. Let's see what results we get with topk@5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 32min 58s, sys: 26.3 s, total: 33min 24s\n",
      "Wall time: 4min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "topk = 5\n",
    "source_embeddings = embeddings_pl\n",
    "target_embeddings = embeddings_en\n",
    "\n",
    "correct, total = 0, 0\n",
    "\n",
    "for source_word in source_embeddings.i2w:\n",
    "    if source_word in pl_en_dict.source_words and pl_en_dict[source_word] in target_embeddings.i2w:\n",
    "        total += 1\n",
    "        if pl_en_dict[source_word] in embeddings_en.nn_words_to(embeddings_pl[source_word], n=topk):\n",
    "            correct += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.7637384477789924, 20126)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "correct / total, total"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}