nli_01_task_and_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural language inference: task and datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Christopher Potts\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2022\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Our version of the task](#Our-version-of-the-task)\n",
    "1. [Primary resources](#Primary-resources)\n",
    "1. [Set-up](#Set-up)\n",
    "1. [SNLI](#SNLI)\n",
    "    1. [SNLI properties](#SNLI-properties)\n",
    "    1. [Working with SNLI](#Working-with-SNLI)\n",
    "1. [MultiNLI](#MultiNLI)\n",
    "    1. [MultiNLI properties](#MultiNLI-properties)\n",
    "    1. [Working with MultiNLI](#Working-with-MultiNLI)\n",
    "    1. [Annotated MultiNLI subsets](#Annotated-MultiNLI-subsets)\n",
    "1. [Adversarial NLI](#Adversarial-NLI)\n",
    "    1. [Adversarial NLI properties](#Adversarial-NLI-properties)\n",
    "    1. [Working with Adversarial NLI](#Working-with-Adversarial-NLI)\n",
    "1. [Other NLI datasets](#Other-NLI-datasets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Natural Language Inference (NLI) is the task of predicting the logical relationships between words, phrases, sentences, (paragraphs, documents, ...). Such relationships are crucial for all kinds of reasoning in natural language: arguing, debating, problem solving, summarization, and so forth.\n",
    "\n",
    "[Dagan et al. (2006)](https://u.cs.biu.ac.il/~nlp/RTE1/Proceedings/dagan_et_al.pdf), one of the foundational papers on NLI (also called Recognizing Textual Entailment; RTE), make a case for the generality of this task in NLU:\n",
    "\n",
    "> It seems that major inferences, as needed by multiple applications, can indeed be cast in terms of textual entailment. For example, __a QA system__ has to identify texts that entail a hypothesized answer. [...] Similarly, for certain __Information Retrieval__ queries the combination of semantic concepts and relations denoted by the query should be entailed from relevant retrieved documents. [...] In __multi-document summarization__ a redundant sentence, to be omitted from the summary, should be entailed from other sentences in the summary. And in __MT evaluation__ a correct translation should be semantically equivalent to the gold standard translation, and thus both translations should entail each other. Consequently, we hypothesize that textual entailment recognition is a suitable generic task for evaluating and comparing applied semantic inference models. Eventually, such efforts can promote the development of entailment recognition \"engines\" which may provide useful generic modules across applications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Our version of the task\n",
    "\n",
    "Our NLI data will look like this:\n",
    "\n",
    "| Premise | Relation        | Hypothesis |\n",
    "|:--------|:---------------:|:------------|\n",
    "| turtle  | contradiction | linguist   |\n",
    "| A turtled danced | entails | A turtle moved |\n",
    "| Every reptile danced | entails | Every turtle moved |\n",
    "| Some turtles walk | contradicts | No turtles move |\n",
    "| James Byron Dean refused to move without blue jeans | entails | James Dean didn't dance without pants |\n",
    "\n",
    "In the [word-entailment bakeoff](hw_wordentail.ipynb), we study a special case of this where the premise and hypothesis are single words. This notebook begins to introduce the problem of NLI more fully."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Primary resources\n",
    "\n",
    "We're going to focus on three NLI corpora:\n",
    "\n",
    "* [The Stanford Natural Language Inference corpus (SNLI)](https://nlp.stanford.edu/projects/snli/)\n",
    "* [The Multi-Genre NLI Corpus (MultiNLI)](https://www.nyu.edu/projects/bowman/multinli/)\n",
    "* [The Adversarial NLI Corpus (ANLI)](https://github.com/facebookresearch/anli)\n",
    "\n",
    "The first was collected by a group at Stanford, led by [Sam Bowman](https://www.nyu.edu/projects/bowman/), and the second was collected by a group at NYU, also led by [Sam Bowman](https://www.nyu.edu/projects/bowman/). Both have the same format and were crowdsourced using the same basic methods. However, SNLI is entirely focused on image captions, whereas MultiNLI includes a greater range of contexts.\n",
    "\n",
    "The third corpus was collected by a group at Facebook AI and UNC Chapel Hill. The team's goal was to address the fact that datasets like SNLI and MultiNLI seem to be artificially easy – models trained on them can often surpass stated human performance levels but still fail on examples that are simple and intuitive for people. The dataset is \"Adversarial\" because the annotators were asked to try to construct examples that fooled strong models but still passed muster with other human readers.\n",
    "\n",
    "This notebook presents tools for working with these corpora. The [second notebook in the unit](nli_02_models.ipynb) concerns models of NLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set-up\n",
    "\n",
    "* As usual, you need to be fully set up to work with [the CS224u repository](https://github.com/cgpotts/cs224u/).\n",
    "\n",
    "* If you haven't already, download [the course data](http://web.stanford.edu/class/cs224u/data/data.tgz), unpack it, and place it in the directory containing the course repository – the same directory as this notebook. (If you want to put it somewhere else, change `DATA_HOME` below.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nli\n",
    "import os\n",
    "import pandas as pd\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_HOME = os.path.join(\"data\", \"nlidata\")\n",
    "\n",
    "ANNOTATIONS_HOME = os.path.join(DATA_HOME, \"multinli_1.0_annotations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SNLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SNLI properties"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For SNLI (and MultiNLI), MTurk annotators were presented with premise sentences and asked to produce new sentences that entailed, contradicted, or were neutral with respect to the premise. A subset of the examples were then validated by an additional four MTurk annotators."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* All the premises are captions from the [Flickr30K corpus](http://shannon.cs.illinois.edu/DenotationGraph/).\n",
    "\n",
    "\n",
    "* Some of the sentences rather depressingly reflect stereotypes ([Rudinger et al. 2017](https://www.aclweb.org/anthology/W17-1609)).\n",
    "\n",
    "\n",
    "* 550,152 train examples; 10K dev; 10K test\n",
    "\n",
    "\n",
    "* Mean length in tokens:\n",
    "  * Premise: 14.1\n",
    "  * Hypothesis: 8.3\n",
    "\n",
    "* Clause-types\n",
    "  * Premise S-rooted: 74%\n",
    "  * Hypothesis S-rooted: 88.9%\n",
    "\n",
    "\n",
    "* Vocab size: 37,026\n",
    "\n",
    "\n",
    "* 56,951 examples validated by four additional annotators\n",
    "  * 58.3% examples with unanimous gold label\n",
    "  * 91.2% of gold labels match the author's label\n",
    "  * 0.70 overall Fleiss kappa\n",
    "\n",
    "\n",
    "* Top scores currently around 90%.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with SNLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset snli (/Users/cgpotts/.cache/huggingface/datasets/snli/plain_text/1.0.0/1f60b67533b65ae0275561ff7828aad5ee4282d0e6f844fd148d05d3c6ea251b)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b761a1daf52e4b80a4ef908ec541ef5c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "snli = load_dataset(\"snli\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset has three splits:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['test', 'train', 'validation'])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snli.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The class `nli.NLIReader` is used by all the readers discussed here.\n",
    "\n",
    "Because the datasets are so large, it is often useful to be able to randomly sample from them. This is supported with the keyword argument `samp_percentage`. For example, the following samples approximately 10% of the examples from the SNLI training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"NLIReader({'splits': (Dataset({\n",
       "    features: ['premise', 'hypothesis', 'label'],\n",
       "    num_rows: 550152\n",
       "}),), 'filter_unlabeled': True, 'samp_percentage': 0.1, 'random_state': 42})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nli.NLIReader(snli['train'], samp_percentage=0.10, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The precise number of examples will vary somewhat because of the way the sampling is done. (Here, we choose efficiency over precision in the number of cases we return; see the implementation for details.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of the readers have a `read` method that yields `NLIExample` example instances. For SNLI, these have the following attributes:\n",
    "\n",
    "* __label__: `str`\n",
    "* __premise__: `str`\n",
    "* __hypothesis__: `str`\n",
    "\n",
    "Note: the original SNLI distribution includes a number of other valuable fields, including identifiers for the original caption in the [Flickr 30k corpus](http://shannon.cs.illinois.edu/DenotationGraph/), parses for the examples, and annotation distributions for the validation set. Perhaps someone could update [the dataset on Hugging Face](https://huggingface.co/datasets/snli) to provide access to this information!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following creates the label distribution for the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "entailment       183416\n",
       "contradiction    183187\n",
       "neutral          182764\n",
       "-                   785\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "snli_labels = pd.Series(\n",
    "    [ex.label for ex in nli.NLIReader(\n",
    "        snli['train'], filter_unlabeled=False).read()])\n",
    "\n",
    "snli_labels.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `filter_unlabeled=True` (the default) to silently drop the examples for which `gold_label` is `-`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a specific example in some detail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "snli_iterator = iter(nli.NLIReader(snli['train']).read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "snli_ex = next(snli_iterator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"NLIExample({'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 'neutral'})\n"
     ]
    }
   ],
   "source": [
    "print(snli_ex)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MultiNLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### MultiNLI properties\n",
    "\n",
    "\n",
    "* Train premises drawn from five genres: \n",
    "  1. Fiction: works from 1912–2010 spanning many genres\n",
    "  1. Government: reports, letters, speeches, etc., from government websites\n",
    "  1. The _Slate_ website\n",
    "  1. Telephone: the Switchboard corpus\n",
    "  1. Travel: Berlitz travel guides\n",
    "\n",
    "\n",
    "* Additional genres just for dev and test (the __mismatched__ condition): \n",
    "  1. The 9/11 report\n",
    "  1. Face-to-face: The Charlotte Narrative and Conversation Collection\n",
    "  1. Fundraising letters\n",
    "  1. Non-fiction from Oxford University Press\n",
    "  1. _Verbatim_ articles about linguistics\n",
    "\n",
    "\n",
    "* 392,702 train examples; 20K dev; 20K test\n",
    "\n",
    "\n",
    "* 19,647 examples validated by four additional annotators\n",
    "  * 58.2% examples with unanimous gold label\n",
    "  * 92.6% of gold labels match the author's label\n",
    "\n",
    "\n",
    "* Test-set labels available as a Kaggle competition.  \n",
    "\n",
    "  * Top matched scores currently around 0.81.\n",
    "  * Top mismatched scores currently around 0.83."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with MultiNLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n",
      "Reusing dataset multi_nli (/Users/cgpotts/.cache/huggingface/datasets/multi_nli/default/0.0.0/591f72eb6263d1ab527561777936b199b714cda156d35716881158a2bd144f39)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b0538a6eec324cc18092fa3e6c4f9fd7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mnli = load_dataset(\"multi_nli\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For MultiNLI, we have the following splits: \n",
    "\n",
    "* `train`\n",
    "* `validation_matched`\n",
    "* `validation_mismatched`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['train', 'validation_matched', 'validation_mismatched'])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnli.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The MultiNLI test sets are available on Kaggle ([matched version](https://www.kaggle.com/c/multinli-matched-open-evaluation) and [mismatched version](https://www.kaggle.com/c/multinli-mismatched-open-evaluation))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The interface to these is the same as for the SNLI readers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"NLIReader({'splits': (Dataset({\n",
       "    features: ['promptID', 'pairID', 'premise', 'premise_binary_parse', 'premise_parse', 'hypothesis', 'hypothesis_binary_parse', 'hypothesis_parse', 'genre', 'label'],\n",
       "    num_rows: 392702\n",
       "}),), 'filter_unlabeled': True, 'samp_percentage': 0.1, 'random_state': 42})"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nli.NLIReader(mnli['train'], samp_percentage=0.10, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `NLIExample` instances for MultiNLI have nearly all the attributes that SNLI is supposed to have!\n",
    "\n",
    "* __promptID__: `str`\n",
    "* __label__: `str`\n",
    "* __pairID__: `str`\n",
    "* __premise__: `str`\n",
    "* __premise_binary_parse__: `nltk.tree.Tree`\n",
    "* __premise_parse__: `nltk.tree.Tree`\n",
    "* __hypothesis__: `str`\n",
    "* __hypothesis_binary_parse__: `nltk.tree.Tree`\n",
    "* __hypothesis_parse__: `nltk.tree.Tree`\n",
    "\n",
    "The only field that is unfortunately missing is  __annotator_labels__, which gives all five labels chosen by annotators for the two dev splits. Perhaps someone could [create a PR to bring these fields back in](https://huggingface.co/datasets/multi_nli)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The full label distribution for the train split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "entailment       3463\n",
       "contradiction    3240\n",
       "neutral          3129\n",
       "dtype: int64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "multinli_labels = pd.Series(\n",
    "    [ex.label for ex in nli.NLIReader(\n",
    "        mnli['validation_mismatched'], filter_unlabeled=False).read()])\n",
    "\n",
    "multinli_labels.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No examples in the MultiNLI train set lack a gold label. The original corpus distribution does contain some unlabeled examples in its dev-sets, but those seem to have been removed in the Hugging Face distribution. As a result, the value of the `filter_unlabeled` parameter has no effect for `mnli`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at a specific example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnli_iterator = iter(nli.NLIReader(mnli['train']).read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "mnli_ex = next(mnli_iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are three versions of the premise and hypothesis sentences:\n",
    "\n",
    "1. Regular string representations of the data\n",
    "2. Unlabeled binary parses\n",
    "3. Labeled parses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Conceptually cream skimming has two basic dimensions - product and geography.'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnli_ex.premise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The binary parses lack node labels; so that we can use `nltk.tree.Tree` with them, the label `X` is added to all of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvgAAAEnCAIAAABTwKDTAAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBAZokEykBKkDulARhAbsoYcIDfIGwqCQqFoKAnKgHKhnVARVApVQXVQE/QLdA66At2EBqGH0Dg0A/0NfYQRmATTYQVYA9aH2TAHdoV94fVwNJwK58D58F64Aq6HT8Id8BX4NjwMC+GX8BwCECLCQJQRXYSNcBEPJBiJQgTIVqQQKUfqkVakG+lD7iFCZBb5gMKgaCgmShdliXJG+aH4qFTUVlQxqgp1AtWB6kXdQ42jRKjPaDJaHq2DtkDz0IHoaHQWugBdjm5Et6OvoYfRk+h3GAyGgWFhzDDOmCBMHGYzphhzGNOGuYwZxExg5rBYrAxWB2uF9cCGYdOxBdhK7EnsJewQdhL7HkfEKeGMcI64YFwSLg9XjmvGXcQN4aZwC3hxvDreAu+Bj8BvwpfgG/Dd+Dv4SfwCQYLAIlgRfAlxhB2ECkIr4RphjPCGSCSqEM2JXsRY4nZiBfEU8QZxnPiBRCVpk7ikEFIGaS/pOOky6SHpDZlM1iDbkoPJ6eS95CbyVfJT8nsxmpieGE8sQmybWLVYh9iQ2CsKnqJO4VA2UHIo5ZQzlDuUWXG8uIY4VzxMfKt4tfg58VHxOQmahKGEh0SiRLFEs8RNiWkqlqpBdaBGUPOpx6hXqRM0hKZK49L4tJ20Bto12iQdQ2fRefQ4ehH9Z/oAXSRJlTSW9JfMlqyWvCApZCAMDQaPkcAoYZxmjDA+SilIcaQipfZItUoNSc1Ly0nbSkdKF0q3SQ9Lf5RhyjjIxMvsl+mUeSKLktWW9ZLNkj0ie012Vo4uZynHlyuUOy33SB6W15b3lt8sf0y+X35OQVHBSSFFoVLhqsKsIkPRVjFOsUzxouKMEk3JWilWqUzpktILpiSTw0xgVjB7mSJleWVn5QzlOuUB5QUVloqfSp5Km8oTVYIqWzVKtUy1R1WkpqTmrpar1qL2SB2vzlaPUT+k3qc+r8HSCNDYrdGpMc2SZvFYOawW1pgmWdNGM1WzXvO+FkaLrRWvdVjrrjasbaIdo12tfUcH1jHVidU5rDO4Cr3KfFXSqvpVo7okXY5upm6L7rgeQ89NL0+vU++Vvpp+sP5+/T79zwYmBgkGDQaPDamGLoZ5ht2GfxtpG/GNqo3uryavdly9bXXX6tfGOsaRxkeMH5jQTNxNdpv0mHwyNTMVmLaazpipmYWa1ZiNsulsT3Yx+4Y52tzOfJv5efMPFqYW6RanLf6y1LWMt2y2nF7DWhO5pmHNhJWKVZhVnZXQmmkdan3UWmijbBNmU2/zzFbVNsK20XaKo8WJ45zkvLIzsBPYtdvNcy24W7iX7RF7J/tC+wEHqoOfQ5XDU0cVx2jHFkeRk4nTZqfLzmhnV+f9zqM8BR6f18QTuZi5bHHpdSW5+rhWuT5z03YTuHW7w+4u7gfcx9aqr01a2+kBPHgeBzyeeLI8Uz1/9cJ4eXpVez33NvTO9e7zofls9Gn2eedr51vi+9hP0y/Dr8ef4h/i3+Q/H2AfUBogDNQP3BJ4O0g2KDaoKxgb7B/cGDy3zmHdwXWTISYhBSEj61nrs9ff3CC7IWHDhY2UjWEbz4SiQwNCm0MXwzzC6sPmwnnhNeEiPpd/iP8ywjaiLGIm0iqyNHIqyiqqNGo62ir6QPRMjE1MecxsLDe2KvZ1nHNcbdx8vEf88filhICEtkRcYmjiuSRqUnxSb7JicnbyYIpOSkGKMNUi9WCqSOAqaEyD0tandaXTlz/F/gzNjF0Z45nWmdWZ77P8s85kS2QnZfdv0t60Z9NUjmPOT5tRm/mbe3KVc3fkjm/hbKnbCm0N39qzTXVb/rbJ7U7bT+wg7Ijf8VueQV5p3tudATu78xXyt+dP7HLa1VIgViAoGN1tubv2B9QPsT8M7Fm9p3LP58KIwltFBkXlRYvF/OJbPxr+WPHj0t6ovQMlpiVH9mH2Je0b2W+z/0SpRGlO6cQB9wMdZcyywrK3BzcevFluXF57iHAo45Cwwq2iq1Ktcl/lYlVM1XC1XXVbjXzNnpr5wxGHh47YHmmtVagtqv14NPbogzqnuo56jfryY5hjmceeN/g39P3E/qmpUbaxqPHT8aTjwhPeJ3qbzJqamuWbS1rgloyWmZMhJ+/+bP9zV6tua10bo63oFDiVcerFL6G/jJx2Pd1zhn2m9az62Zp2WnthB9SxqUPUGdMp7ArqGjzncq6n27K7/Ve9X4+fVz5ffUHyQslFwsX8i0uXci7NXU65PHsl+spEz8aex1cDr97v9eoduOZ67cZ1x+tX+zh9l25Y3Th/0+LmuVvsW523TW939Jv0t/9m8lv7gOlAxx2zO113ze92D64ZvDhkM3Tlnv296/d5928Prx0eHPEbeTAaMip8EPFg+mHCw9ePMh8tPN4+hh4rfCL+pPyp/NP637V+bxOaCi+M24/3P/N59niCP/Hyj7Q/Fifzn5Ofl08pTTVNG02fn3Gcufti3YvJlykvF2YL/pT4s+aV5quzf9n+1S8KFE2+Frxe+rv4jcyb42+N3/bMec49fZf4bmG+8L3M+xMf2B/6PgZ8nFrIWsQuVnzS+tT92fXz2FLi0tI/QiyQvpTNDAsAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAfdEVYdFNvZnR3YXJlAEdQTCBHaG9zdHNjcmlwdCA5LjUzLjNvnKwnAAAgAElEQVR4nO3dT2zjWGLn8Ve9Hey0a5I1G5F3MxvAFp2TPSfRbmCRAaoAUYeuRh8WMH3snosoYCaHXEbUsedGFXIJkBzIufQ09kQOsKeuOYgNuLAJArTFwQILGdgAYsnZbCFrI2YhO3ZmgyTew+vicChblq0/lKjv5yRR5OOjTIs/vT/Uo+vrawEAAFBE7+RdAQAAgFkh6AAAgMIi6AAAgMIi6AAAgMIi6AAAgMJ6N+8KAMhfEARCCE3TFEURQkRRFEWRqqqqquZdNQCYCC06wKqLosg0TcMwZNwRQti2XavVLMvKt2IAMDmCDrDqVFUNwzDdfrO9vV2v133fz7diADC5R9wwEIAQIgxD0zSDIAjD0LKsMAzzrhEATAFBB8A3XNf1PC+KoiAIGJ0DoBgIOgB+TVVVXddd1827IgAwHYzRAfAN0zQbjUYURYzOAVAYTC8HIIQQshXHsizTNHVdV1VV07S8KwUAk6JFB4AIw9DzvHa7LYRQFMV1XdM04zjOu14AMCmCDrDqfN/XdT2KItu25RLLsqIo0jQtiqJ86wYAE2IwMgAAKCxadAAAQGERdAAAQGERdAAAQGExvRxYdeFgEF9eCiGi8/P+2dmbqyt1Y0Pb3JSv6ru7udYOACZC0AEKK768DAcD+bjT68kH4enpNw8GgzdXV+OXVt3ZkQ+SDLRXLitra0IItVRSNzamUWUAmDJmXQFLKTo7i87PxdtmGCFEfHUll8SXl794m2aGVTY3lcePhRBqqSRjyvbGhloqCSGUx4+1ra104fHVVffVK7lhkpC+Ojm5rfByqfRNUWtr8sH73/62DEZJ4QAwTwQdYOEEb1tfkmaY6Pw8vroSdzXDDDe61N52PGlbWzLfzKKe4enpxS9/OUk9k8ahWdQTwCoj6ADzM9wMI962lIxuhklaSpJmmGVpKVnBQwawUAg6wHRMsRlmBZs3ePcAzAhBB7jDiAErtEnM0xQbh4aHJQEoKoIOVtqIUSbR+fmr8/PbNqQhYWENNw6NM9FsfW0tSTzzGeEEYD4IOiimWcwbEtxUpigytw4SU52zBmChEHSwfGbRDMOdYJDx4LsQ0TgELBSCDhZIcmmhGQbLYvLGoeRETRqHiN3AFBF0MCfJ9SBphkmuB6ObYbgeYNk9OMEnjUNJghepxiESPDAOgg4mxYc4MBV8GQBmgaCDUfjkBRYK3yuA+yLorKjJB1rycQksrAd/RWHAPoqHoFNATJ0FcCduwYAVQdBZJsx3BTBn3M0By46gsyiGb29PMwyApTCLxiE+vjAtBJ15mOIPFtIMA2BJjWgc4qdbMTsEnZmwPC84OeHnHgHgXib86dbgRz+aTz2xRN7NuwKFpTx+3PzwQ/mYLx8AMA51Y2Oc4TsjmsmBDFp0AABAYb2TdwUAAABmhaADAAAKi6ADAAAKi8HIUxMEgRBC0zRFUYQQURRFUaSqqqqqeVcNAIqJD17ciRad6YiiyDRNwzDkf50QwrbtWq1mWVa+FQOAouKDF+Mg6EyHqqphGKa/Rmxvb9frdd/3860YABQVH7wYB9PLpykMQ9M0gyAIw9CyrDAM864RABQcH7wYjaAzZa7rep4XRVEQBHQSA8Ac8MGLEQg606eqqq7rruvmXREAWBV88OI2jNGZMtM0G41GFEV0EgPAfPDBixGYXj5N8suEZVmmaeq6rqqqpml5VwoAiowPXoxGi87UhGHoeV673RZCKIriuq5pmnEc510vACgsPnhxJ4LOdPi+r+t6FEW2bcsllmVFUaRpWhRF+dYNAAqJD16Mg8HIAACgsGjRAQAAhUXQAQAAhUXQAQAAhcX08ikLer1Or/c//+7v/tMf/IG+s6NtbeVdIwAopqDXE0KEp6cXv/xlfHX13//mb37nvfe0zU0hxPbGhloqKY8f8yEMBiNPQXx56R8fdwcD//j4zdWVEOLf/87v/J9/+AchRLlUMvb29splY38/72oCwPLJpJno/Dy+vPzF6enwmvvl8r/867/e+FJ1Z0cIQQZaTQSdhwsHg+DkxPv6a/l/tb62pu/s1HZ3jf195fHj6OzM73Y7vd5XJyfDr+ZddwBYFPHlZTgYCCE6vZ4QIjo/l4Hm1fl5Zs31tTWZTmRk2SuXlbU1tVRSNzYya4aDQXx5GZ2f98/OhBDh6akQQn4aZ6Qz0Pvf/rZ8oO/uTvUQkSeCzr35x8fdV6/8blf+E8o2m9ru7m3/GMPtPZXNzcMPPqBjC8DqiM7OovNzMUaaKZdKaqkkxkgzDzN+BqpsbiqPH6ulkrK2RgZaXgSdsURnZ8HJSafX+1m3K5dUZfPM3t69/vfkCJ5MSKJjC0AxyDQTX111X70SbzNEOBjI73hpMs3IBCOEqO3uCiG0ra18G7xvzEA31p8MtEQIOqOEg4H39dfByUnSOWXs7+9tbU3e/UTHFoAlJdPAOGlGpoFFSzMPs+wZbpURdLLiy0vZeBOcnMh2l8rmpr6zc/jBB7PoaaJjC8ACon9nfGSgBUfQ+YZsYum+epV0Th3ITqV7dk5Ngo4tAPOUntAkGLE7A+OPs85koKmPTFplqx50ZLZIOqfKpZLsQtJ3dnKM2HRsAZiW8adnMwd7nmYx1ww3WsWgc2Nv0ew6pyZBxxaAO2UumSPSTOaSSZpZTA/+g5KBbrRCQee2ZhJ9Z2cpzgk6toBVRgMAJJro7qv4QefG294sdT6gYwsoqhuHtY4zpEMOayXNrDgy0I2KGXRkj48cfFPgHh86toBlxCQdzN8q3yq6UEFn+DcZpnXbm8VHxxawUCa89RxpBnNT+FsJFCHoJI03Y/4mQ7HRsQXMTeGvEFhxxbhV9LIGnWn9JkOB0bEFTAXjHoBhS9QDu2RB58bfZMj9tjeLj44tYDTSDDAti5aBliDoJL/JsPi3vVl8dGxhZTE9G8hdLreKXtygc+NvMizRbW8WHB1bKCT5VVKM/TEqSDPAYpjdV5HFDTrKD3/45uoq+U0GOllmJ9Ox1W+3+azHMrI87/nPf55ekvvgAACTG/NW0dWdneBHPxrefHGDjnt0pG1t0bowT3KIt/n0ad4VAR5CRnbSDLBSkgF22ubmjfO8FjfoAAAATOidvCsAAAAwKwQdAABQWO8IIeI4zrsaOTBNU9d1XdfDMLzvq8UTBEEQBMmZEEVREARRFOVbK+BOnLoARnvHMAzbtvOuxii6rs+iWNd1gyDQNO3GnDf61YKJosg0TcMwgiCQS2zbrtVqlmXlWzFgNE5dAHei6wpCVdUwDFVVVVVVLtne3q7X677v51sxYDROXQB3eqfVau3t7SXPgyDQdV3TNFVVDcNItwDrum5ZVrvd1jRtuFsnDEPDMLS3LMtKf9b4vq+qqq7rqqqappk0k1iWpeu667qWZSUrJBvKyoRhqKfIbeVLyfc213Xlq+ljsyxLVjWz0wdLatJut9P1n1Gb0zwpiuK6rnyXgiDwPM913bwrBdyNUxfAHa5Tut1upVLp9/vyaafTKZfLFxcXyQpCiGazKR97nndwcCAf9/v9crnc7XaTp5VKJVnTcZx6vZ6U43letVpNymw2m+vr647jpLf1PC9ZIb1yRualzOF0Op3ksW3btm0Pl9BsNtOr3flqp9Op1+uZOiTv2LJzHKdarZbL5cIcEVYEpy6A2/xG15Vt2+12O2kE1nW91Wqlvx5VKpWkMSPdL+44TqvV0jRNPlVVtdVq1Wq1dLGKoiQbapqWLtYwDNM0k21933ccZ/IMJ1uD5FhFTdMuLi6mUmYURUlDVxAE6WbzZWeaZhRFsg0s77oA98CpC+A2vxF0ZH9Qeomqqp1OJ3mahBXpzZs38oHsJk+/ZBhGUlQcx/JpIjOVKVOsqqpT6WZSVdWyrE6n0+l0pjg4sdVqJcO3bdtutVrTKjl3pmk2Go0oihjigOXCqQvgNu8KIeI4llFD07QoitKRJY7jpJ1mhNHRRDbSZNJMWib3yGaScWo/gmmavu8nlQ+CIJ3YJqHrum3bsjWrSM05so3Nsiw5tV5V1XH+9EDuOHUBjPBOu91O2icajUa65SOOY8dxGo3GnaUcHh7atp3OOkEQJEUdHh5mGlRkd1LyNAzDZLBwEASmaaabSRRFSSeh9Pjo9OOkTy2pfBJB5IHceRTjk406juNkdrq8wjD0PE8eTnp0Z971Au7AqQtACBGGoaIoNza7vHtxcZEediObcOTkpjAMW62WjAvyfhVyapVsHDYMQwgh52Hput5oNOSGiqLIMTHpqUnJBCjxtsEmPUbHNM3t7W1ZoKqqmRadVqtlGIZ8NdlWtg/JHjFN05JON9M0ZclyzJA8Ijlbyvf9OI7lq67rep4n3t5eTJYmJ5Td+ap426ijquqIZqol4vu+aZqKosjRVEIIy7KiKJJvbGGarFA8nLoApDiO5XCapJMqccOPespkIB50pz7ZTqNp2o0J4MZXZWPP6KaRpErDJYdhKMPZ8B7lVoqizKId2zCMJG8BAIB8JY06meX5/3r5OEFn0fi+3+12l6vOAACsoHfz3b1lWc+fPxdCPH/+3PM82T+1sGSTeDLXTCxbPgMAYNXk36IDAAAwI/zWFQAAKKycu66wUIJez37x4m/+/u//c6XS+ugj5fHjvGsE3Cq+vAwHg+j8vH92Fp2fx1dXX52cCCGqOzvK2ppaKm1vbKilkra1xZkMrDK6riDE24jz1cnJb3/rW1u/+7v/42//dn1tzXzyhLiDRXBbpkmUSyW1VNoulYQQ/fPz6Pz81fl5egXSD7CyCDqrLok46WRz48K8a4qVMGam0TY3hRC13V3l8WNta+vGosLBIL687PR6Qojw9JT0A6wmgs7qujPNEHcwU1PMNOMj/QCrhqCziu6VYIg7mFwumWZ8pB+gwAg6q+XBqSXZUAhRf/Kk9eyZurEx48piKS14phkf6QcoBoLOqojOzizf/1m3K4RofvjhwxpmiDvICHq9+Oqq++rVjZlmfW1N29pSSyVlbW2vXFbW1vTd3byqOhWkH2DpEHSKLzo7s1+8+MnLl2JK6STo9ZyjI5mZiDurI8k08dVVdH4eDgZvrq6SV4uXacZH+gEWGUGnyKYeceZWOPJFppkc6QdYEASdYppbCiHuLDsyzTyRfoD5I+gUTS7Jg7izFMg0i4n0A8wUQac4ck8buVcACTLNsiP9ANNC0CmChUoYmcoc7u9zEZ0pMs3qIP0AD0DQWW7x5aXl+wsScdLScae6s9N69ozr6+RGZxqRus6RaVYH6QcYjaCzrOLLS/vLL92XL99cXS1skiDuPFjm6jUi03D1wjDSD5Ag6CyfpYg4adHZmXN0tEQVnjOuSZgPzjSsJoLOMlm6iJO21JWfFq40WDSckyg8gs5yKExKKMyB3InrB5YXZy+KhKCzBCzPK1gyKF7cCXo9rgootnulH31nZ0EmRgAEnSWgNptqqVSANJCRxB3zyZP24WHe1ZmI9tlnvzg9FWQarJjb0o/3gx8Y+/t51w4QgqCzFOLLywJfLOPLSyHEsh9gOBiopdKyHwUwFfw7YKEQdAAAQGG9k3cFAAAAZoWgAwAACougg3vQdV3X9XFWC4JgFnufRbG3CYJA13XLsua2R2B5BUEQBEEcx/JpFEVBEERRlG+tAEHQwb2MmTM0TVMUZep7n1Gxt5G5KgzDue0RWFJRFJmmaRhG8hFh23atVuN7AhbBu3lXAAXUbreXqFgAE1JVNQxDXddVVZVLtre36/W667r5VgwQBJ1ZC8PQtu2k/VbX9b29PcMw5Evy60673Y6iSD5utVqmacqVfd+3LEtV1SiKdF1vt9vp9gzLsoIgUBQl/appmlEUyYWqqtZqNcdx4jj2fV/TtPGrnew6jmPZjjIcMpI+rHQzj6yAPKj0HkdX7M5q31asrIamae+//77nefL9yaxjWZbv+/JtdF3Xtm0hhOu6ySfyneRbLYRQFMX3/Tv/Cvd6G4FiUBTFdV3TNGU7qOd5tIZiUVxjZvr9frlc7na7ydNKpdJsNtPrVKvVSqVSr9cvLi4uLi5s25bLHceRC+VTz/Oq1Wp6w06nkzy2bTvZUAjhOM719XW9Xpeb9Pv9g4ODe9W8XC73+/0bdy0fd7vdarXqed6NmzebzXT1xqnYONW+sVi5bfKuep6X3qrZbCZv48XFRaVSue85L4RI/1EyVbrtryCNeBuBQnIcp1qtps98IHcEnRlqNpvy4p3wPC9zqa5Wq5noI5XL5STl3FZat9vtvJUUklzI07HgvpfYcrnsOE7yUZVkNVlUp9OpVCrphRm3BZ0RFRun2rcFnUqlkn66vr6ePpD0S/1+/75BJ1P48Dt5418h2fttbyNQVOVyuV6v510L4NfoupqhMAxrtVp6iey0ysisI8VxPLzy9vZ2UrJhGKqqyj4aOT9oOpUWQhboOE6n04njOI7jVquV6QwSQozf9TNrmRHKb968SR5nKvmAOo8Y/nznX2H02wgUj2majUaj0+n4vn/jxx0wfwSdGZKDMx68bWY4SJppmulhN0EQdDqdB9ZyiKxzMppEjj7RdT2pTBiGrusahjGihgsiiqI4jtM1n2Lho/8Kd76NQMHIoceWZZmmKQcmk+yxCJhePkOHh4e2baezThAEY863PDw8zKwpb1MhH8dxnDROxHHsOM6UqiyEELZtpwuUF+ZMYjNNs91uG4bx4CQ3H+12W9d1OZB5/Dd/TKP/CuO8jUBhyAHIMtknA5M54bEQ8u47KzjHcWSPdbPZTAYdy5eazWa1Wl1fX69UKtVqdXhsr9yk2WzKBwcHB8m2slg50lauI/dSr9eFELKDXG4lx4Wsr6/fOBLoRrZtVyoVWawsPxkbJCssH3e73XK5nO6Pl+MQ5VDE5KBkBUZXbPSrI4rt9/uySskY4YODA5Eam3x9fd3pdA4ODqrVar1ev9cYHTngOin84uJCPk2G6dz2V7jzbQQKxvO89fV1+e8gl8h/FkYlYxHwo57zIFtiHna/u9u2jeM4DENFUWbUOCzLF6lp5MWgKMoUv2Xe+Vco6tsIAMuCoIMVkty4KO+KAADmhDE6KDh5K8JHjx7JJrFWq5V3jQAA80OLDgAAKCxadAAA0/RfwzAcDPKuBfAN7qMDAHi46OwsOj/v9HrR+Xl0fv6L09PkperOjloqbW9saJub2taW8vhxjvXEyqLrCgBwD+FgEA4G/bOz8PQ0HAzeXF3J5etra9rWlra5+R/+3b/7p3/5l4tf/jI8Pf3q5CTZsFwqaZubaqm0Vy5rm5vqxkZOR4DVQtABANwqvrwMB4Pw9LR/dhYOBukGm8rmploqqaVSbXd3RINNdHYWnp52X70KT0+j8/NX5+fJS9WdHW1zc3tjQy2V9N3dmR8MVhJBBwDwa0kuic7Pw9PTqeeSyZMTcC8EHQBYaUGvF52fy66oXHqa7uwL2yuX1VJJ29qaUQVQbAQdAFghiz92ePFriOVC0AGAIruzvWR7Y0Pb2lrk9pKg1wtPTxndjIch6ABAcazCCBhGN+NeCDoAsMRmPXZ48a1CtsMkCDoAsExyHzu8+BjdjDSCDgAsLkbmTo73cMURdABggRRg7PDiY3TzSiHoAEBuGF+yCBjdXGwEHQCYH8YOLz7SZ8EQdABghhg7XACMbl5qBB0AmBrGva4C/srLhaADAA/H2GEIRjcvNoIOAIyL0RsYB6ObFwpBBwDuEPR69osXXLHwMKPzsba11Xr2jMae2SHoAMAdLM/zu136IDAtmR5P/wc/ICvPDkEHAAAU1jt5VwAAAGBWCDoAAKCwCDoAAKCw3s27AgCwiIIgEEJomqYoihAiiqIoilRVVVU176qhCDjB5oYWHQDIiqLINE3DMOTVSAhh23atVrMsK9+KoRg4weaJoAMAWaqqhmGY/nq9vb1dr9d938+3YigGTrB5Yno5ANwsDEPTNIMgCMPQsqwwDPOuEQqFE2w+CDoAcCvXdT3Pi6IoCAIGT2DqOMHmgKADAKOoqqrruuu6eVcExcQJNmuM0QGAW5mm2Wg0oihi8ARmgRNsDpheDgA3k1+yLcsyTVPXdVVVNU3Lu1IoDk6w+aBFBwBuEIah53ntdlsIoSiK67qmacZxnHe9UBCcYHND0AGALN/3dV2Posi2bbnEsqwoijRNi6Io37qhADjB5onByAAAoLBo0QEAAIVF0AEAAIVF0AEAAIXF9HIAGCXo9cLBQN3Y0Hd2lMeP864OisY9Orr6p3/69A//kLNrRhiMDABZQa/X6fXC09OvTk6EEL/9rW/931/9SghR2dzUd3b2ymVCDyYUDgbO0ZF/fPzm6kouOdjbq+3umk+f5lqvAiLoAIAQQoSDQXBy0n31Kjg5kdeecqmk7+zsbW0Z+/vhYNDp9YKTk1+cnsr1CT14gOjszO92naOjV+fnQoiDvb3DDz5QS6Uk9KyvrRn7+42nT7WtrbwrWxAEHQCra0S40Xd21I2N4U3iy8tkk0zoqe3u6ru7cz0ALA/36KjT6/2s2xVCVDY3Dz/4wHzyJBOR0+uUS6XG06fG3t6N5yHGR9ABsFqis7Pg5KQ7GAQnJ/Jb9frammybue9F5cbQU93Z0TY3CT2Q0l1U62tr5pMnhx98MLq1Jr68dF++9L7+Wp5U1Z2dw/19Y3+fhsOHIegAKL748tI/Pr4x3Og7O1PpIyD0IO3GLipjf/9ehYSDgff11+7Ll0mX1uH+PufSfRF0ABRTjslD7lqO6ZHXubntGrkbp4vqvvzj406v95OXL4UQ5VLJ2NtrPH1Kl9aYCDoACkVOmFqcZpXhnjIhxMHe3hQbk7AIHtBFdV+yYdI5OpLndmVzs/H0KV1adyLoAFh6mdngYlGnRI0YHkToWVJT6aJ6wE6doyO/25U7rT95UtvdnfVOlxdBB8BSkhOmOr1eEm5kk/6ihZvbEHqW3Sy6qO4r6PW84+OZNiMVAEEHwNJ4wGzwpSBbBdLHRehZWHPoorov2aXlHR/L0J9X6lpYBB0ACy1p+UjuISunnyx7uLlNUcPcssuli+q+hivJrZYFQQfAAqJbRyL0LIJF6KK6r0yz04rfapmgA2AhJLPBkyGWQoiqvN3wKoWb2xB65mwBu6gegFstC4IOgBxxk72HGZ5llv5ZrgVvbFhwS9FFdV8rfqtlgg6AeRu+1Q2/FfVgyzK1fvEtYxfVfa3mrZYJOgDmgevxHPAmP0Axuqjua6VutUzQATArI4aVrE6zeV5uazYj9EiF7KK6rxW51TJBB8A0MWZ20dw4EGqV+wpXoYvqvop9q2WCDoDpiM7O9D/5k8xs8BWc4rHIbgw9zQ8/bB8e5lux+YjOzrQf/3iluqjuK3Or5fjP/zzvGk0BQQfA1Jiff769scFs8KWQhJ69crkw393vZH7+eZHaKmZEdmn1z86KkYAJOgAAoLDeybsCAAAAs0LQAQAAhUXQAQAAhUXQAfBwQRAEQRDHsXwaRVEQBFEU5VsrIMEpugiCINB13bKsXPZO0AHwQFEUmaZpGEYQBHKJbdu1Wi2vjzMgg1N0Qei6HgRBGIa57J2gA+CBVFUNw1BVVVVV5ZLt7e16ve77fr4VAyROUQiCDoBJKIriuq5pmnEcB0HgeZ7runlXCvi1ZT9FLctSVVXXdVVVZQeQruuy6833/eQleYDpDeXKmqapqmoYRqa3bkSxYRjKp2EYyl2oqpq8aZZlaZo2vFPLsnRdd103XfJwoJSbyxKSbZM9ttvtdGm6rk/nTbwGgMk4jlOtVsvlcr/fz7suwA2W9BRtNpv1ev3i4uL6+vri4qJSqSRXbcdxkpeur689z6tWq8mG3W63UqkkB9vpdMrlcrLyiGIT1Wq1UqnI1S4uLmzbTopK1rFtO1kui11fX3ccRz7t9/uVSsXzvGQFIUSyvuM4BwcHyUudTqder2cqMK0/FkEHwBSUy+XM5xQWVrlcHv7S22w2867XbC3jKVoul9NP+/1+kkjSwUVqNptJyDg4OEgnkuvra8dxkpAxothEtVq97ZTodrudt9LryPyUKTkdvyqVSmYXmafpZDbFP9a702kXArDCTNNsNBqdTsf3fcMw8q4O7rCCc46W9BRNhhYNP43jePhAtre35YMgCDLdRqqq2rYtR2GPKDatVqtlloRhaBiGqqqapom3vWPpFRRFyZSc7lDLvJrRarVs25Z9ZMmDqSDoAJiI/DyyLMs0TdkxLz8EgQWxvKdoFEVxHCf5ID1rSY6AuS06aJoWRVEmGCVHPaLY0UzT9H0/KScIgk6nk14hU1QQBLelqGG6rtu2LefHpcePT8G0moYArKBut1utVpMmdDkyINOiDuRoqU9Rz/OSoTadTqdarSZXbdu2M507si8p2TA9Aubi4iLdMTSi2ES1Ws10fl3/Zn/ZxcXFwcFBputqfX09Gf0jBwZ1u910mZldZMqXlTk4OJjuH4gf9QTwQL7vm6apKIphGHK6hJypoSjKvb7JATNSgFM0CALHceI4VlW11Wptb28nV23LspLOI3k4rusm7TSu6zqOIyc3hWHYarXSXV2jiw3DUE7Ll6U1Gg25reu6tm0bhiHL1HXd93052UpuKITY3t72PE8IIUuWb3IYhrJYuYnsdwvDUNO05BZHkpwplky/mgqCDgAAy0FRlOFp5EIITdOGu7FkHBFC3DlPe7jY28gyFUUZ7v6TQWfCjGIYRjquTQVjdAAAWAJymFFm4YgQoyjKOLeiubHYCct8GHnbnummHMENAwEAWFhRFCmK8ujRI3n5b7Vai1msZVnPnz9//vz5o0eP7nvj6aQyh4eHz58/n/oPdNB1BQAACouuKwCTii8v/ePj//bXf/3d3/9988kT5fHjvGsEZEVnZ+0XL9bX1tqHh3nXBXNFiw6Ah4vOzpyjI/flyzdXV/9RUf53HK+vrRn7+42nT7WtrbxrB3zD8jx5lgohyqWS++mn+u5u3pXCnBB0ADyEf3zsHB19dTaXQJAAAA2vSURBVHIihKju7DSePjX294Nezzk6+lm3Kxce7u+bT5/mXFGstnAwMD///Benp9WdnbZhhIOB5ftvrq7qT560DYPWx1VA0AFwD7KXyn7x4tX5uWy8aT17pm5spNdJN/Osr62ZT540nj7NrAPMWnx5aX/55fOf/3x9ba310UfWs2fJcsv3f/Ly5fraWtswyOKFR9ABMJZ0fCmXSo2nT+8cjuMeHXnHx7LV52Bvr/H0Kf0FmI+g1zN/+tNX5+cHe3ttwxjO2ckK1Z0d99NPCeIFRtABcIcbe6nG3zwcDJyjI//4ePyEBDxYfHlpfv75z7rd9bU19/vfH32uWp73/Oc/F0I0P/yQQcpFRdABcLNxeqnuVZr78qVzdJSUxoBlTJ1/fGx+/vmbq6vmhx+2PvponDwdnZ2ZP/3pVycnDFIuKoIOgKwH9FKNjwHLmIUJ84p7dMQg5aIi6AD4tQl7qcbHgGVMUfvFC/vLL+/VkDOMQcpFRdABMOVeqnthwDImkcwer2xuut///uSdoQxSLh6CDrDSZtpLNT4GLOO+bps9PhUMUi4Sgg6woubWSzU+BixjTHNod2GQcmEQdIDVkmMv1fgYsIzbpEfS3Dl7fHIMUi4Agg6wKhakl2p8DFhGRjJ7fJ6xg0HKy46gAxTfAvZS3QsDlhGdnVm+/7NuN6+OJAYpLy+CDlBYS9FLNT4GLK+sqcwenwoGKS8jgg5QQEvXSzU+BiyvFPlj41+dnExr9vjkGKS8dAg6QKEsey/V+BiwXHiy+UQOz1q05hMGKS8Rgg5QBAXrpRofA5YLaSkGxDBIeVkQdIAiaL940fL9gvVS3UsyYPlgb8//4Q/zrg4mojab8eXlUqSHJJP12+3FDGQg6ABFEJ2dhaenRe2lGl84GMSXlwybWHbhYKCWSkuU14Nej7NuYRF0AABAYb2TdwUAAABmhaADAAAKi6ADYLkFQRAEQRzH8mkURUEQRFGUb60wIdM0dV0PwzDvikxHwQ5nuRB0gIWm63reVVhoURSZpmkYRhAEcolt27VazbKsfCuGCbmuq2lakl8nsQj/RFM8HNwXQQfAElNVNQxDVVVVVZVLtre36/W67/v5VgzAgiDoAAsqCALZ1q2nJN8IXdeVS5LG8MwSubmmaaqqGoZR4K4cRVFc1zVNM47jIAg8z3NdN+9KFZZlWbquu65rWZaqqrquq6qajpXJGRuGoe/7MoOm/yIjzky5vny13W5nNkla6ZJTPV2xMAwNw9DesizL9/3R/0RjHq+mafIw5TmWvCSr1G635QrDPVO3HQ7m7RrAAqtWqyNeuri4uL6+7vf78oHjOM1m8/r6utvtViqVfr8v1+x0OuVyWa5TVI7jVKvVcrmcHDVmpNlsrq+vO44jn/b7/Uql4nleep1qtVqpVOr1+sXFxcXFhW3bcvmIM9NxHLm+fKler5fL5U6nky4zvYv09avf75fL5W63m66S/F8Y3vBe0hWwbTs5kKQOyV48zzs4OEheuvNwMDcEHWChjfiMbjab8uqSfKY3m035SXpwcJD5SHUcJ/MZXTzlcrler+ddi+JrNpuZ97nf72dO1Gq1miSAtBFn5nAWr1QqYwadZrOZBC/J87xk20mCzvX1dbfb7byVOahKpZJ+ur6+njy+83AwN3RdAcuqVqt1u105QkUOxZUN9ekHCVVVO51OPhWdC9M0G41GFEWMzpkDRVHST1VVHe4PqtVqwxuOODNVVc0UO/4gYvlfkF5iGMbkY5BlsZZlyZQzPMI9U+E3b94kjyc5HEzXu3lXAMAD6bpu27Z4m3jSF3hN06IoSn/0x3GsaVoOtZwLOQTEsiw5iVdV1QIf7CLIDEYJgiCTM24z4swcHkYWBMGNaWnYjUlrcqZp+r6fnEtBEIz/bWGSw8F00aIDLDRFUdIXlcynZxzHvu8bhtFoNOQoUblcPk2v5jhOo9GYT53nLAxDz/PkYM/0wOS861VkYRgmb3IQBKZptlqtcTYccWYahpF+yXXdzNmefpoZ23t4eGjbdvqPHgRBUtrof6IR4jhOMpms6pgbijEOB3Pzbz777LO86wDgVtvb24ZhvH79OgiCVqsVhmGtVnvvvffkq1EUfec73zk8PPzOd77zp3/6p59++unu7q4QYnd3982bN6Zpyq6cdrv9x3/8x9/73vdyPZSZkDnvV7/6VRzH8uvyJ598cnJy8sUXX3z88ceZvgNMRRAEmqZ997vf/eyzz7744ovXr1+7rrv79ict5USkMAzlrKsvvvjivffeS14dcWbWajV5kkdR5LpuHMff+973XNd99OiRbFN5/fp1u93u9XqtVuv3fu/3/vIv//L169cff/yxEEJV1UePHn3yySe9Xu8v/uIvWq1WHMc//vGP5X/K6H+iEd57771PPvnk9evXsqqapvm+3+v1Pv744yiKDMOQh3l4eCiEMAzj5OTkH//xH+V5eOfhYG74UU9g0cVxLL+Papp2ryt3siGDAzBFsqFikvnSI87MKIpk39aNfWFhGMqurtv+EeRgteEVJvwnUhTlYelk9OFgPgg6AIB7mDzoAPNE0AEAjMuyrOfPn8vHnucZhpFvfYA7EXQAAEBhMesKAAAUFvfRAVAE8eWl/eWX/+Wv/urf/tZvtZ49M58+zbtGABYCXVcAlpuMOO7Ll2+urv5gY+P//fM//6+Li3Kp1Hr2zNjfVx4/zruCAPJE0AGwrNIRp7qz03r2TN/dFUK4R0f2ixevzs/X19bMJ09aH31E3AFWFkEHwPKJzs6co6PhiJPmHh15x8dfnZzIuNN4+lTd2MiltgByRNABsEyiszP7xYufvHwphLgt4qQFvZ794sVXJydCiPqTJ61nz4g7wEoh6ABYDumIU3/y5HB/f3TESQsHA+fo6GHbAlhqBB0Aiy4TcR7cKnPf1iAABUDQAbC4gl7POTr6WbcrptfxlBnfc7i/z1x0oMAIOgAW0azH1qRnbMm56MQdoJAIOgAWSxJx5jNbKpmLXi6VjL095qIDBUPQAbAoMhFnnpmDW+8ARUXQAZC/BckZ6aRl7O8zFx0oAIIOgDwtSMRJm8UIaAB5IegAyEd6cMwCjgVmLjpQDAQdAHMVX176x8eLHHHSZNzxj49H/NYEgEVG0AEwJ8s7o3t5aw6AoANg5tJBobK52Xj6dBmDQnx56b586RwdJW1Rxv5+7iOKAIxG0AEwQ+mIU5iunwUcQA3gNgQdADNRyIiT5h4decfHudz1B8D4CDoAZiK+vHz/j/6okBEnLbn1zsWf/RlBB1hABB0AsxKdna3IHWhW50iBpUPQAQAAhfVO3hUAAACYFYIOAAAoLIIOgN9gWZau667rzm4Xuq4HQTDFAk3T1HU9DMNZ72hCi1YfYBUQdAD8hna7rWlav9+f3S40TVMUZYoFuq6raVocx7Pe0YQWrT7AKng37woAWDntdrtgOxrTotUHWAUEHQA3sywr6Wdpt9u6rqeXK4oSRZGu6+12O91K4fu+ZVmqqsZxLBsw0ld30zSjKBJv243SuwvD0LZt+aoQQtf1vb09wzBuq156R4eHh5lXb9yRXChrrqpqrVZzHCeOY9/3k3WSYoePTtd1TdPef/99z/PkwsxRjDj2EQcuhAiCwLbtOI7lhu12W1XVyXcKQAghrgHgNzWbzfX1ddu25dN+v1+pVDqdjnyaPLi+vrZtO1lNKpfL/X5fPvY8r1qt3lh+uhC5i3K53O1203tsNpu31dBxnHq9fnFxIZ/W6/VyuZwp88YdCSEcx5GbyLr1+/2Dg4Mbix2uvxAiqZXnecmGYx77cH2ur6+73W6lUkk27HQ65XI5qcPkOwVWHEEHQFaz2azX6+klnU4nfX3tdrudtzJxpFwuO46TXHqT7JIpP3O9bzabMn8kPM8bzgTpvaSjwPX1dTqKjdhR8u0u/VISDoaLzVSsUqmkX11fX8/UavSx3xh0Dg4OMgsdx0nHxwl3Cqw4BiMDuEFmzKyu67LnJQxDVVUty5Ipx7KszIZBEPT7fTl1S9O0pCtqNFlseolhGEln2TBVVYdrOM6ORovjWO43kZnJldnpmzdv0k8fduxBEGQqr6pqp9OZ6U6B1cEYHQB3k0NbhBCmaaZHtARBkL4ky3lPyRgROcxF1/U7pxrJISb3qk9mSRAEtVpt/BJuq4bv+w+bGPXgY5fpJJ3z5Gibme4UWB206AC4geu66cYMy7IajYYQIo7j5JIcx7HjOOmtbNtOL5GX23ESzOHhoRyNmywJgmC4uShhGEb6Vdd1p9KScXh4mNlpEARj3vnmwcfeaDTSO5Xvqny3Z7dTYHXQogPgN1iW5fu+aZqWZSmKIhs5Go2GnADVarU0TTMMI47jMAx1Xfd9P45jeYNBOTlI0zRd1+UKrVYrCUau63qeJ4SIokjO2xJv5xDput5oNOQDRVGCIJCTj26rZLvdtixLri9bm2SFG42GaZq37UhmArmCPFJ5A54wDC3LkmUmxQohgiBQVVWuHEWRaZphGBqG4fu+EEK+IXLD0cc+4sBlObIJZ3jDSXYKQOJHPQHcKgzDZNJyslBeUBVFua17Ra4gHjRuRjafjHljvSiKZKfP1C/t96pG2oOPfZI3bZJtgcIj6AAAgMJijA4AACgsgg4AACgsgg4AACgsgg4AACgsgg4AACgsgg4AACis/w+FzRPXsLdIngAAAABJRU5ErkJggg==",
      "text/plain": [
       "Tree('X', [Tree('X', ['Conceptually', Tree('X', ['cream', 'skimming'])]), Tree('X', [Tree('X', ['has', Tree('X', [Tree('X', [Tree('X', ['two', Tree('X', ['basic', 'dimensions'])]), '-']), Tree('X', [Tree('X', ['product', 'and']), 'geography'])])]), '.'])])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnli_ex.premise_binary_parse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's the full parse tree with syntactic categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvwAAAEACAIAAABAgjUnAAAJMmlDQ1BkZWZhdWx0X3JnYi5pY2MAAEiJlZVnUJNZF8fv8zzphUASQodQQ5EqJYCUEFoo0quoQOidUEVsiLgCK4qINEWQRQEXXJUia0UUC4uCAhZ0gywCyrpxFVFBWXDfGZ33HT+8/5l7z2/+c+bec8/5cAEgiINlwct7YlK6wNvJjhkYFMwE3yiMn5bC8fR0A9/VuxEArcR7ut/P+a4IEZFp/OW4uLxy+SmCdACg7GXWzEpPWeGjy0wPj//CZ1dYsFzgMt9Y4eh/eexLzr8s+pLj681dfhUKABwp+hsO/4b/c++KVDiC9NioyGymT3JUelaYIJKZttIJHpfL9BQkR8UmRH5T8P+V/B2lR2anr0RucsomQWx0TDrzfw41MjA0BF9n8cbrS48hRv9/z2dFX73kegDYcwAg+7564ZUAdO4CQPrRV09tua+UfAA67vAzBJn/eqiVDQ0IgALoQAYoAlWgCXSBETADlsAWOAAX4AF8QRDYAPggBiQCAcgCuWAHKABFYB84CKpALWgATaAVnAad4Dy4Aq6D2+AuGAaPgRBMgpdABN6BBQiCsBAZokEykBKkDulARhAbsoYcIDfIGwqCQqFoKAnKgHKhnVARVApVQXVQE/QLdA66At2EBqGH0Dg0A/0NfYQRmATTYQVYA9aH2TAHdoV94fVwNJwK58D58F64Aq6HT8Id8BX4NjwMC+GX8BwCECLCQJQRXYSNcBEPJBiJQgTIVqQQKUfqkVakG+lD7iFCZBb5gMKgaCgmShdliXJG+aH4qFTUVlQxqgp1AtWB6kXdQ42jRKjPaDJaHq2DtkDz0IHoaHQWugBdjm5Et6OvoYfRk+h3GAyGgWFhzDDOmCBMHGYzphhzGNOGuYwZxExg5rBYrAxWB2uF9cCGYdOxBdhK7EnsJewQdhL7HkfEKeGMcI64YFwSLg9XjmvGXcQN4aZwC3hxvDreAu+Bj8BvwpfgG/Dd+Dv4SfwCQYLAIlgRfAlxhB2ECkIr4RphjPCGSCSqEM2JXsRY4nZiBfEU8QZxnPiBRCVpk7ikEFIGaS/pOOky6SHpDZlM1iDbkoPJ6eS95CbyVfJT8nsxmpieGE8sQmybWLVYh9iQ2CsKnqJO4VA2UHIo5ZQzlDuUWXG8uIY4VzxMfKt4tfg58VHxOQmahKGEh0SiRLFEs8RNiWkqlqpBdaBGUPOpx6hXqRM0hKZK49L4tJ20Bto12iQdQ2fRefQ4ehH9Z/oAXSRJlTSW9JfMlqyWvCApZCAMDQaPkcAoYZxmjDA+SilIcaQipfZItUoNSc1Ly0nbSkdKF0q3SQ9Lf5RhyjjIxMvsl+mUeSKLktWW9ZLNkj0ie012Vo4uZynHlyuUOy33SB6W15b3lt8sf0y+X35OQVHBSSFFoVLhqsKsIkPRVjFOsUzxouKMEk3JWilWqUzpktILpiSTw0xgVjB7mSJleWVn5QzlOuUB5QUVloqfSp5Km8oTVYIqWzVKtUy1R1WkpqTmrpar1qL2SB2vzlaPUT+k3qc+r8HSCNDYrdGpMc2SZvFYOawW1pgmWdNGM1WzXvO+FkaLrRWvdVjrrjasbaIdo12tfUcH1jHVidU5rDO4Cr3KfFXSqvpVo7okXY5upm6L7rgeQ89NL0+vU++Vvpp+sP5+/T79zwYmBgkGDQaPDamGLoZ5ht2GfxtpG/GNqo3uryavdly9bXXX6tfGOsaRxkeMH5jQTNxNdpv0mHwyNTMVmLaazpipmYWa1ZiNsulsT3Yx+4Y52tzOfJv5efMPFqYW6RanLf6y1LWMt2y2nF7DWhO5pmHNhJWKVZhVnZXQmmkdan3UWmijbBNmU2/zzFbVNsK20XaKo8WJ45zkvLIzsBPYtdvNcy24W7iX7RF7J/tC+wEHqoOfQ5XDU0cVx2jHFkeRk4nTZqfLzmhnV+f9zqM8BR6f18QTuZi5bHHpdSW5+rhWuT5z03YTuHW7w+4u7gfcx9aqr01a2+kBPHgeBzyeeLI8Uz1/9cJ4eXpVez33NvTO9e7zofls9Gn2eedr51vi+9hP0y/Dr8ef4h/i3+Q/H2AfUBogDNQP3BJ4O0g2KDaoKxgb7B/cGDy3zmHdwXWTISYhBSEj61nrs9ff3CC7IWHDhY2UjWEbz4SiQwNCm0MXwzzC6sPmwnnhNeEiPpd/iP8ywjaiLGIm0iqyNHIqyiqqNGo62ir6QPRMjE1MecxsLDe2KvZ1nHNcbdx8vEf88filhICEtkRcYmjiuSRqUnxSb7JicnbyYIpOSkGKMNUi9WCqSOAqaEyD0tandaXTlz/F/gzNjF0Z45nWmdWZ77P8s85kS2QnZfdv0t60Z9NUjmPOT5tRm/mbe3KVc3fkjm/hbKnbCm0N39qzTXVb/rbJ7U7bT+wg7Ijf8VueQV5p3tudATu78xXyt+dP7HLa1VIgViAoGN1tubv2B9QPsT8M7Fm9p3LP58KIwltFBkXlRYvF/OJbPxr+WPHj0t6ovQMlpiVH9mH2Je0b2W+z/0SpRGlO6cQB9wMdZcyywrK3BzcevFluXF57iHAo45Cwwq2iq1Ktcl/lYlVM1XC1XXVbjXzNnpr5wxGHh47YHmmtVagtqv14NPbogzqnuo56jfryY5hjmceeN/g39P3E/qmpUbaxqPHT8aTjwhPeJ3qbzJqamuWbS1rgloyWmZMhJ+/+bP9zV6tua10bo63oFDiVcerFL6G/jJx2Pd1zhn2m9az62Zp2WnthB9SxqUPUGdMp7ArqGjzncq6n27K7/Ve9X4+fVz5ffUHyQslFwsX8i0uXci7NXU65PHsl+spEz8aex1cDr97v9eoduOZ67cZ1x+tX+zh9l25Y3Th/0+LmuVvsW523TW939Jv0t/9m8lv7gOlAxx2zO113ze92D64ZvDhkM3Tlnv296/d5928Prx0eHPEbeTAaMip8EPFg+mHCw9ePMh8tPN4+hh4rfCL+pPyp/NP637V+bxOaCi+M24/3P/N59niCP/Hyj7Q/Fifzn5Ofl08pTTVNG02fn3Gcufti3YvJlykvF2YL/pT4s+aV5quzf9n+1S8KFE2+Frxe+rv4jcyb42+N3/bMec49fZf4bmG+8L3M+xMf2B/6PgZ8nFrIWsQuVnzS+tT92fXz2FLi0tI/QiyQvpTNDAsAAAAJcEhZcwAADdcAAA3XAUIom3gAAAAfdEVYdFNvZnR3YXJlAEdQTCBHaG9zdHNjcmlwdCA5LjUzLjNvnKwnAAAgAElEQVR4nO3dT4zj1p0n8FceD3ZT5YNoQAUMclCRfdmpugRilbEYGOheiHXo9m5ORR3XMTBiAfEhlwmpW/oolX0azBigfIh7cxMNDLALdx9EL7qxyR7copHDquaQEVt18C6mFIgdpKs2mz+oPTzXy2tKeqL+8t/3c5L0JPKRfCR/fP+0dXNzQwAAAACy7o24MwAAAACwCQh6AAAAIBcQ9AAAAEAuIOgBAACAXEDQAwAAALnwZtwZAABYDdd12WtFURRFGf9OEASe5xFCNE2LmMovlpEkSVXV5fMMAJu0hSHrAJABnudZluV5HotFfN93HIcPTVqtlm3bNKBxHCdi6tbWVqVSoQuUJEmSpCAIJEmaGAwBQJIh6AGA7NA0jcUinufpuu77Pn3rOE673XYch74NgkDTNMdxaIWQIHVr69vrpGVZx8fHdBWNRgNBD0DqoE8PAGSTqqqKorDQpNFotFotlipJUrPZbDQaM1P7/X5oyZqm8V8GgLRA0AMA2dRqtWiFDSHE8zzaMsV/gVULiVMn9g2a+CEAJBw6MgNAdniex6IcvqkrCIKJ/Y5p7CJOBYDMQNADANmhqioNdJrNZrvdZp8rikKHZYXQD8WpAJAZaN4CgAyyLOv09NQwDPpWURTf91mnZspxHF3XZ6YCQGYg6AGAbDIMIwgC1sLVbDZ1XQ+CgL6lQ9zr9XqUVADIhr94+PBh3HkAAFiW67qGYXie57rud77znYODA0LIu+++q2na1tbWu+++e3BwsLW19aMf/YjO39NqtX72s5/RrxFCxKnNZvPhw4e/+MUvfv7znzuOw5YPAOmCeXoAIF9c1xXMpyxOBYBUQ9ADAAAAuYA+PQAAAJALCHoAAAAgFxD0AAAAQC4g6AGAXHN7va0PPnB7vbgzAgBrh6AHAAAAcgFBDwAAAOQCgh4AAADIBQQ9AAAAkAsIegAAACAXEPQAAABALiDoAQAAgFxA0AMAAAC5gKAHAAAAcgFBDwAAAOQCgh4AAADIBQQ9AAAAkAsIegAAACAXEPQAAABALiDoAQAAgFxA0AMAAAC5gKAHAAAAcgFBDwDkmrSzU9nfl3Z24s4IAKzd1s3NTdx5AAAAAFg71PQAAABALiDoAQAAgFxA0AMAAAC58GbcGQAAiI3v+77vE0I0TYs7LwCwdqjpAYCcarVamqZ1Op1Op6MoiqIocecIANYLo7cAII9839c0zfM8SZIIIUEQvP3227geAmQbanoAII+CIFAUhUY8hBBJkmzbjjdLALBuqOkBgJxSVVXTtGq1qqpq3HkBgE1ATQ8A5JTneXfu3LFtW9M0VVUdx4k7RwCwXqjpAQD4touP67rozgyQYajpAYA8arVanuext4qi6LpOh68DQFYh6AGAPOr3+3zP5SAIHMdBNQ9AtmFyQgDIL9qXOQgC13Xr9TqCHoBsQ58eAMivIAhoIxdmZAbIAwQ9AAAAkAvo0wMAAAC5gKAHAAAAcgFBDwAAAOQCRm8BQDb5l5f+cPjt6+Gwf3nJXgfX1/T1V77/29/9rrC9/fL6urC9re7tsZ+rpRJ7fWd3VykW6WtpZ4f/GgCkCDoyA0A6BFdX3mDw7evr6+6LFyzJu7igL/zh8MVtoDONXCwqxeIf/vSni1//+v/85je//+Mf1VKpsr8fXF+zICm4uvr6dplilf199lopFqXtbfr67bfe4sMmdW9P2tmJskAAWB8EPQAQG7fXY6873GuPCzi+PD+fuRwWeUjb26xKhg87QtUzradPG48fvxgOK/v7TV2fWXPjDQbB1RV9zVcahbLqDQYvb+uQBGjUxd7ysdGhLLOwSSkWld3dmUsDgOgQ9ADAyvAtSt7FxejVq28/51qUokQG5VKJ1YvwMcHxwQF7rXGvo3N7vcbjx1+en8vFYlPX9aOjBRYSBb8ryOshHb83FqtS4vdJqEppsd0CkBMIegBgqmktSnxLUPQWJfqabwPiKzbW3QAUXF1ZjvPps2eF7W3j7t1mtbq+dS3Mfb26i0WNZP7aL/L6bufrwMjr4SOqlCA/EPQA5MvMFqUodQ98n99pLUqJupVa7Xbr2bOX19e1u3frDx4kJ2PL46uUQl2d+CqlKLEpef3Ikum9uQmqlCCdEPQApBvf3WRai9Jc3WLIqluU4uU8f245Du2+U3/wII2bsA7TqpT4OjwSuUqJb47ka/LI6+UHvbkhdgh6AJJl2kDrZVqUpnWVzfZNyBsMLMeh3XfqDx4Y9+7FnaN048PraaPnyKqrlDBBAKwWgh6AtWNP1dNuFcu0KPF3iES1KMUouLpqfPHF2ZMnhBDz/v36e+9lOLZLMr5PGHm9OXWxCQKmVSlhggCICEEPwNymtSgtPNCacE+6goHWEEXz8ePGF1+8vL4+OTxs6jqiwNSJZYIAnGs5gaAHINLUvcsMtOavrehTsj5ur2c8evRiOCyXSk1dx67OD0wQABEh6IEMWu3UvfQ1X5eOFqWk8S8vjUePvjw/L2xvN3Ud3XcgCvf12GhaldLyEwRgzsnkQNAD6RDX1L2QcOi+A5u02gkCyPQqJUwQsCYIeiAeyZ+6F5Kv9fSp5Tgvr68r+/ut99/HAzQk02rnnOQvephzcl4IemA1sjR1LySf2+tZjvP1xYVcLLbefx9xLWQMJghYEwQ9MFU+p+6FhPMvLy3H+bzbLWxv1997z3rwIO4cASRCaIKAaXNOLj9BAEnznJMIenIEU/dCqtHuO+zfJJq6nqJLLUAy5W2CAAQ9KYapeyE/Wk+fNh4/pv8m0dT1zFfCAySWYM7J5ScIEMw5uZJnaQQ9iUPLE6buBaD4f5No6rp+dBR3jgBgbhEnCFigSmniBAHT7nEIehLHv7y8Y1mhDzF1L+SW8/y58dlnxt27zWo17rwAwOYsM0FAQ9cndvhD0JNENCJGtxgAyr+8RMUkAERBb6DTumQg6AEAAIBceCPuDAAAAABsAoIeAAAAyIU3485ATgVB4HmeoiiKotBPXNclhEiSpKoqe0vxXwPIJN/3fd8nhKiqKklS6BPP89g3cToAwMLQpycelmU5jhMEge/7kiT5vm8YBn1NP7csy/M8GgARQnzfdxyHvQXIGF3XP//880qlcnp6qus6IcSyLNd1fd///ve//8033+B0AIDlIeiJDY17dF1vNpvsk+PjY03T6FtN01h9j+d5uq7TB1+ATDIMo1qtsvJPCGk2m5IkGYZBcDoAwCqgT0+cdF33PI9vyZpGVVVFUaJ8EyClqtVqu93mP2m327TWJwSnAwAsBkFPzFqtljU2FeE42hCG+nzIMFqXEwQBfeu6rqZptH9PCE4HAFgMOjLHTFEUTdOazeZ46OP7PvvQcRxa1b/xDAJszunpKXsMsG2btfwSnA4AsAoIeuJXr9c1TRuvxpck6fj4mL7mr/4AWaXruqZplmX5vh8EAT9KC6cDACwPQU/8JEmq1+uWZYUG4kqSxHfqBMg8RVHoAPV2u119/Z+2cDoAwPLQpycRaDUPPxkJQD5Vq1Xbtl3XpYO2AADm4nmeJEmKorAOgjwEPfFwXddxHMdx2MNrs9lkQQ/twul5nqZpmqY5jhNfTgE2Std1Wv7ZJzgdACC6IAhevnwZBMHEoAfz9AAAAEB2sMqe8SQEPQAAAJALaN4CAACAXEDQAwAAALmAoCdxvMHAGwzizgVAUvyT5+GMAICVwDw98QuurrzBoNPreRcX3mDw8vqaEFLY3tb295Vi8fjgQDs4iDuPAJtAzwXv4mL06pV3ceEPhy+GQ5ZaLpWknR21VLqzu6sUizgvAGAit9fr9HrN1+f6otCROR60Oqc7GHiDwdcXF/RDuVjU9vd733zzP//lX/7z3/zN//jVr9gVv7K/r5ZKh7Ks7e9LOzvxZRxgZUIhDov4qXKppBSLSrH47/7qr/7wpz/1Ly/HwyC5WFSKRRYGqXt7ODsAwGq3z548ufnpT8eTUNOzIROrcwghlf198/59Pppxe73jjz/+j9/73qNazb+8dM/P+5eX7vn52ZMn9CflUknd2zvc29P295Xd3dg2CWAeM0McWrV5KMs0fBEsyu31guvr7osX/nDoD4fs1CCEFLa31b09tVR6+6231FIJYRAA8FDTs0Y0ZJlYnXNnd1fb3592Zd/64IPa3butH/yA/zC4unLPz7svXngXF1+en7OlsRog8X0CYJP8y0t/OOz0esH1tT8cshJLVfb3lWJR2t4+PjhQisXlY3dvMPCHQxoGBdfX/OpoGLTa1QFAkglqehD0rBhtSvSHQ/f8nK/OmatxSvvoI3849M/OxCvyLi66L16wFbFnXHQDgg3bcIiz2ixJOzt4YADIEgQ9a8S3QPHVOTTKUUulBeKP5uPHdcfpN5sRbw+sh5B7fs56PND2AnQDgpVj1Srj8QSLvAkhSatWoWHQzC5EUdrXACDJ0KdnxaZV59DeOWqptOSFXtvfJ4S45+dGtOWoe3vq3h79e0b/8pLVALG+DrRNDd2AYAF8y5E/HLLIntyGOOb9+6noQKPs7iq7u/xDSLib0cXF590uS8VgMYDsQU1PJHwksarqHDHpww+1/X3nww+XWQjfe5o9jtPB8OgGBBPNDHEy30fY7fX84RCDxQDSCzU9i2CdZryLC77NaFXVOWLa/r77ei+EBUg7OxrXv8cbDGhXaPf8nD3R0v5GxwcHuILn0MwbvHn/ft7qOca3FIPFADIDNT1/xqpzxutFNj9JYOvp09NHj7o/+cmaKmPYyLKJ3YDWHdJBLKLXYeQnxFkMBosBJBk6Mk81rTqHjSqP64LlX17esayGrlsPHqx7XWwwfKjxjnYDoh2G1p0HWK2ZUxvTTrsIcVYFg8UAkgNBz59NnO0mmf/5oJimUiy6P/7xhtdLu2mP/yfGmnovwfIiTm2McUmbhMFiAHHJe58e2peFjirnq3Nqd+8meUCTtr//6bNnMax3UjcgfmAL/hMjXiuc2hjWB4PFABIom0HPtOocdW9PPzxMS6fd44ODT589c3u9GC9/fNsWP8E0/58YsTcFZtvMdhP96AjdR1IhNLCACnW04o8vBosBrFx2gh5xdU4aO6bQ2Xo6sQY9PGV319jdpbMB8WFlaDYg8T9sgNjMEKd29y5CnCzBYDGATUpxn56Jf+HJ/xVDBq4I6sOHhBDv4cOY8zHL+GyNyewmlTQpndoYNgyDxQDmkp2OzOz/Fub9C8+Uokdu9A//kKLobWKVG7oBEcz7B6uDwWIAAikOeiZW55Dc3ETdXu/444/bP/yhfnQUd14WMfGPycqlkrq3l+Qu5CuBEAc2CYPFAJhUBj3GZ5/lpDpHbOuDD2p377Z+8IO4M7Ksib3L6V95NHU9G9FP6+nT9vPnmPcPkmDmXE3Szk716Mi4dy++PAKsRSqHrHuDgbSzQ//zIdvVOWL2++9nI8iTdnb0oyNWZcWmhXTPzzNzcPuXl/5wqBSL+uEhQhyIV5TBYv3Ly7iyB7A+p/fuHU+59ia3pgcAAABghd6IOwMAAAAAm4CgBwAAAHIBQQ8AAADkQlI6Mruuy79VVVWSpIipGRAEged5iqIoikI/oZssSZKqquLUuPI8l/RuoO/7vu8TrtSFPuEL58QMh0pvEjYK8mPmycWXT/5rANl0kwD9fr9SqVQqlXK5LMtypVJpt9sRU7PBNE1ZlguFwmg0urndZFmWy+Vyv98Xp8ad90jSu4EnJyeEEL7UmaZZLpcLhYJpmt1ut1KpFAqFyi1Zlm3b5pfAp5bL5UqlEsd2QE6JT66JBbjb7cada4B1SUTQw3Q6HdM0F0tNO3pt4jfQNM1OpxMlNRXSu4G1Wi2Uk0ajwUc2fBwzGo1kWaY3mPHUTIbskHAzTy6+iHa7XVmWN5o/gA1Cn54E0XXd87xQa0jE1FRI6QZWq9V2u81/0m63dV2f+GVJkhRFoe1fIc1mU1XVaT8EWJ/oJ5eqqoqiJPA0BFiJpPTpAarVatHL0wKpqZDGDdQ0zTCMIAhonx7XdTVN43uVBUHAbhKdTifUa4cmeZ7XbreTtmmQHxFPriAIfN9HtzPIqphreiY+EEdMzSRFUTRNazabC6SmQko38PT0tNVq0de2bZ+envKpvu83brmuqyhKEAT8F4IgMAyDLQFg8wQnl+/71i1VVZvNZsZGigAwMQc9hmEInjzEqVlVr9fb7fa0gE+cmgpp3EBd123bJoT4vh8EQWiEi6qq7i3P8+7cuWNZFv8Fy7JOT0/p07PjOKGQCGAzpp1ckiQd3/J9Hy2wkGHx9+nhbwDjNwNxaiZJklSv10N3zYipqZDGDVQURVVVz/Ns265Wq+Ivh4L1VqtFa3roW9u2cxjKQxJMO7kkSdJuxZIxgI2JOeiRJIm/AbTb7ePj44ipGUaftKbdGsWpqZDGDaxWq7Ztu67LwpdpWq0W6xLh+75t22jYgoRI5skFsEKe59EBJZMrSuIdPDYajejkEHRQZaPRiJ6aJZ1OR5ZlOgsR/aTf7xcKBTqsVJyaCtnYwNC435ubm06nMz7NSa1WY0PWadGtcBK4XZBhM089vgBjPgXIgE6nQwgpFAoTp3lLxL+s0ylup82zLE4FAAAAYFhlz3hSIoIeAAAAgHWLvyMzAAAAwAYg6AEAAIBcwIzMAAD54g0GwdWV8/y5srurlkpKsajs7sadKYBNSGjQo330UfXoyLh3L+6MxMwbDCzHaeq6urcXd17WIksbaLXbhJDmrFl8ADbJv7z0h0Pv4mL06pV3ceEPhy+Gw4nfrOzvK8WitL19KMtKsZiBUxJgXEKDni/Pz9VSKe5cxC+4uvry/Dy4uoo7I+uSpQ30Li7izgLkndvr+cNh//LSHw6D6+svz8/51HKppJZK+uHhnd1dpVj8b7/85d+77n/527/9X998QwjxLi6c589fXl/z35d2dtRSiX5fOzjY9PYArFpCgx4AABCgTVSdXi+4vvaHQ28w4OMVuVhUikXz/v2333prWgOWtLPz9677v3/zG756Mri68gYDGjnRmiE+cipsb6t7e2qpJFgsQJIh6AEASDRxExUNRPSjo3lbptS9PblYbH/1lfXgAftQ2tkZr9GhYVD3xQtagXT25AmfinYxSBEEPQAACTJXE5W6tyft7Cy8Lv3w8OzJE//yUlxho+7t0biKfULjsE6vR9AuBqmCoAcAIB7LN1EtqfrOO2dPnjjdLl/ZE4Wyu6vs7vLRDNrFIBUQ9AAArN2amqiWNLGFazFoF4NUQNADALBim2yiWlLEFq7FoF0MkgZBDwDA4mJvolrSwi1ci0G7GMQLQQ8AQCTJbKJa0gpbuBaDdjHYJAQ9AAATpKiJaklrbeFaDNrFYE0Q9ABA3qW9iWpJG27hWgzaxWAlEPQAQI5ksolqSbG3cC0G7WKwAAQ9AJBZ+WmiWlICW7gWg3YxEEPQAwBZkPMmqiWlooVrMWgXAx6CHgBIGTRRrVxKW7gWg3axPEPQAwCJhiaqzchMC9di0C6WEwh6ACAp0EQVowy3cC0G7WKZhKAHAGLmDQb6J5+giSpetIWr0+sh6JlmgXYx8/79ZrW6wTzCDFs3Nzdx5wEAcs0bDBpffKEUi2iiipc3GCjFInb+kvh2MUIIgp5EQdADAAAAufBG3BkAAAAA2AQEPQAAAJALCHoAAAAgF94MgkCSpLizsWmGYfi+TwhpNpuqqs6VmmeaphFCXNed+bV6vU6/vNq1r2OxYq7rNhoNVVWbzeYm15s9QRB4nqcoiqIo9BNakCRJomcZX674rwEkHMp2irzZaDSSfDXXNG3mLXYBrVaLEGJZVhAE86bmmeu6UWIOVVXXEUmvabFi2q0Nrzd7Go2G4zhBEPi+L0mS7/uNRoO+pp83Gg3P89hjhu/7juPgqQOSD2U7RTBPD6zemsLoJEfnMBM9fI7j0ActRVFc17Us6/j4mD740pCaPeR4nqfrOq1zBUgylO0UeePw8JC9oQdGVVVFUUKHRNM0y7Jocw998PU8j18QPYrqLcuyHMehSY7jKIqiaZqiKIZhsOoTy7I0TWu1WpZlsS+wX9HMeJ6ncdhvaaplWfRtq9UKPY5blkWzGlrpMlhm2N2XboKgGkCwW9jSPM+ju0hRFFrJJN5vgq0zDEPTNLpGXddbrRY9mqGDNRNbtaqqhmGw/cybWAVCMzBePMQZm5ntaYslEUomX7posdE0bYHLDd3ndC2h4iQubFF2Zn7ouu55XpTqW1oG1lHRCxP5vq/rumEYcWckrVC20+HmVrfbLZfL/X6fvu10OrIsj0Yj9gVCiGma9HW73T45OWFJ/X5fluVut8velstl+mXbtmu1GltOu92uVCrsh6ZpFgoF27b5H7bbbfYF/svjQqn85nQ6Hfa60Wg0Go3xn5umyX8tSmqn06nVaqE8sJ0WItgt/M/L5TLdRaPRiOVTvN8EW0cIofuzVqvRn/T7ff5gRSHLMtuo0Krp6263W6lU+CPFm7jrxBmLku1px0tQMk3TZLtxNBqVy2W+kERHCOEPTShj4sIm2Jl5Y5qmaZr0RGCf8HuP3zmj0Sh0CYK1sm2b3hTYJQuiQ9lOiz83b7F6OfqWdhql1TD0k3K5zGo4Qg8Etm3X63XWQqkoSr1ep90vaFsm64qh63q32221Wuzn/KJoTY9hGLquLxnM0Sd++sytqmqn01lygWyxtLGWVVoKeqUJdktomXzVEX0h3m/iraPfkSSpXq/T9S5Q0UUrRWidX2gDac0trY+Za5nijC2cbUHJdByHVerQ9vU7d+7MlWe2CnZoDMNot9t86szCJtiZOUTrvZrN5nill+/77EPHcZrNZg6HWcSFVlIqioK+JgtD2U6+Pwc9ruuylhdKUZRGo8GOU+gIvXz5kr32PO/4+JhPZVFLEATjEQx/4wktdrE7dAhtVGJnb8TutxHV6/VGo0HbodiLadmYtlt4oe9Qgv221q2jXNe1bbvT6QRBEAQBH7qR28gsOTdvQckMZXLhPAsuTzMPh3hn5hMdhTdewiVJYqcDunBtXs7bXlcCZTvh/jxkXVVVVoFBBUEQ8eosiFRo5Y34nsG/pXUnkfI+nWEYfN9413VXVdNDbit7aHOsePDhMgGcYL+tdesIITTP7LT0fZ/2g2GZ8Tyv1Wrpui4+skng+z4/KcO8fZuiEB+OmTszn2h9Hu1uFfocA+Ug1VC2E+6NRqNBX52envJhfhAEtm2fnp5GWUq1Wm00GvwNnraA0KTQ04PrunwHLs/zWN9P13UNw6CtG5QkSfyNKtT/lH/Lx860kpbfkChbER2t7LFtWxywC3bLTIL9tu6to5vG3tLbcyh6Mwyj2Wzqup7wUf3NZpN1W46+8+ciPhxRdmY+0UfhdYShsBjanr6SuvacQ9lONL6Dj23btKdtrVbjOxT3+/1KpVIoFFj/zZOTE8L1HqW/lWW5VquZpsl65tIk+gnt51Uul09OTvgk0zRt265UKpVKpVarhToFd7tdWZYn/pb+vFKpsOUTQmgvY5oZtiGmadK8sazS1cmyXC6X6WvWd0+cytD1zuwzJd4tdK+ytYT6BU/bb4Ktq9VqbCfQX9GcFwqFKLmlGo1GuVzmSwLraU4zzB+aiDtWnDFxqmCxUUpmp9M5OTlhpYvM2ZGZdtlmqxiNRvQt65YoLmyCnZk3dGyELMts1/X7/UKhQDt7djodtmMFfeRhfWgNZaFQmDYyA6ZB2U6R8L+s05klye3cu/OiVRETJ5GbmESfvMX1JSxLExdLO5COJ9FfsQkxV46OrI7YSCHYLYv9dt1bR5YuCYklSdLKH2RnHo6s7kzIGFbZE3dGANYlHPRsWJSgJ4Ecx+l2u6nLNqS0vAEAwErE+YejlmWdnZ2dnZ1tbW2FBo4lE51WfGtrq1qtnp2dYaRDKrCjRqvK+B5jAACQKzHX9AAAAABsBv57CwBiE1xdOc+fP/rFLw6++93DvT396Eja2Yk7U7nmDQbq3l7cucgCbzD4r7/85dN//ud/f+fOoSzrR0dx5wgIQU0PAGyef3npdLudXu/L83P6yb95883/98c/EkLKpZK2v398cKAdHMSaxzyy2u2zJ09ufvrTuDOSYs7z590XL5xu98VwGEqq0IK9v4+wMkYIegBgQ7zBwD0/b3/11dcXF4QQuVjUDw//6euvh7/9rX925g0GnV7PPT+nqYSQk8PDQ1nGTWJjEPQshlZY0tL78vqa3MY3o1evzp48+e+m+etXr2gqjYTkYlHb30fVZizQvAUA6+UNBu2vvmLPvuVSybx/v/rOO+reXuvp01/96782dF3a2dFua3eCqyv3/JzeJD7vdgluEpBIoSC+sL2tHx3xpdRqtwkh/+Gv/5oQQpu36E+6L158+uzZp8+enT56hKrNDUNNDwCsBXv2ZbFO9Z139MNDZXeXfUcxTUKIf3Y2bSG0Iaz74gV7hsZNYn1Q0xNFqAGLVlhOLJDaRx8FV1few4eC5bCqzcL2tnbb/sWfI7BaqOkBgJVhlTTO8+c0Rjk5PKw/eDDxOt56+vTFcGi//75ggcrurvXgAX3t9no0ijp78uTsyROC9i/YlIkNWKf37oWC+BBvMND296el6kdHtPrHv7wcr9rUDw/R/XkdEPQAwLLYLYFestkzq6A1Kri6shxHLhaNe/cirgXtX7BhMxuwBIKrq5fX10qxOHMtyu6usbtLTwTaFkwje5qK7s+rhaAHABYUGoRV2N6u3b1LY52Zv2188cXL62vnhz9cYL3Szg7/lEzbv5znz9FJAlZivAHLvH9/3hLlDQaEkENZnmvV6t4eDW74yL7uOASR/YqgTw8AzGfiIKy5bgnB1ZVimurenvvjH68wY6z9C+O/FtN6+vT00aN+s5nDPiXTRmCJG7AEmo8f1x2n+5OfLF/2WPdnWq9J0LNtCajpAYBIBIOw5l0Ureap33bWWRW0fy2JtsX4w2F+gp5lGrDERq9eEUJWEnBwGrEAAA4jSURBVG2z6h/CdX+mPdvQ/XleqOkBAJEog7Dm4l9e3rGsyv7+aqt5BKvD+K+I3F7v+OOPO3/3d5nfM9FHYC1MPHRreXz3Z1qw0f05CtT0AEDYXIOw5tV4/JgQ0hIO2lohjP8CarERWAsTD91aHro/LwZBDwB8a4FBWPPyLy8/ffasdvduLFXxaP/KofU1YAlEH7q1Euj+HB2atwDybnwQln50FHEQ1rz0f/zHz7vdRHWVRfsXk6XmrQ00YAnQPdn+4Q9jbGlC9+eJUNMDkFPjg7AWGJc7F7fX+7zbNe/fT07EQ9D+lSEbbsAS8C4uyG3H8Lig+/NECHoA8mWFg7Dm1Xj8uLC9XX/vvXWvaGFo/0qjWBqwxFY4dGslMPszg+YtgFxY+SCsedEKf/P+/Wa1upk1rkqu2r+2PvggLcco3gYssXUP3VoJ1v2ZzWuVh+7PCHoAMmviIKy46rTVhw/94dA/O0t1NUnm5z9MeNCz8ikE10T68ENtf9/58MO4MxIJX6/JIsis1muieQsgazYwCGteradPv764aOh62i+gaP+KRQIbsAQ2PHRrefz/urDuz58+e5bJ/3VBTQ9A1tARUmsdhDUv47PP3PNz/+ws7oysRaj9Sy4W07ul0ocfNnU9+r/AbgCdzZIkrAFLwBsM9E8+aep6Ek69ZbDuzzTWtN9/P1EFYzEIegCyxu31CCFJuzEEV1cJfChfObfXC66v0363S5rW06c5HGeUHLT7czLr1eaFoAcAAABy4Y24MwAAAACwCQh6AAAAIBcwegsgxYIg8DyPEKKqqiRJQRBIkkQIcV2XfUfTtM1njM8Auc1exNRkortaURRFUegndCskSVJVVZzKFuL7vu/7JKaDkmQr2b2xmHYOipOSKb1HYQ43AJBOtm3LsmyapmmasiyfnJyYpnlzc9PtdmVZlmW5UqlUKpVyuVyr1Uaj0cYy1u/32appNtrtdsTUxKI7uVAo0D1Jt0KW5XK53O/3xal0CaHjJctynNuTMMvv3lhMOwfFSYmV0qMwFwQ9AKnUbrdPTk74T/irKr3UsiTbtiuVykbzd3Nzc3PT6XQEF3pxagLRiz6fZ9M0O51OlNR+vy/LMgs9R6MRnjlDltm9sRCcg+LTM8lSdxTmhT49AKlkWVar1eI/qdfrh4eHE79sGIaiKLSmHZah67rneaHmuSipQRAoisJaNyRJsm17jRlNp4V3bywE5+Bcp2fSpOsozAtBD0Aq8XdQSlVVXdenff/4+Ljdbq8/X9nXarUsy5o3lXaJsCyLhZ6GYawri2m22O6NheAcnPf0TJoUHYV5IegByIXN9KCkvXQXS00LRVE0TWs2m/Omep53584d27Y1TVNV1XGcNec0KkVRtsbEdVdbePfCCmX4KCDoAUilIAjm+v5mAg7DMASNaOLUFKnX6+12e9ouFaQahtFqtVzXdRzHsqyERIG+7493fYjxlrbw7t0wwTk47+mZQGk5CvNC0AOQSpIkjQcQgkttu92ubuSvs/k8jOdHnJoWkiTV6/VpdSETU1utFn+8FEXRdT2N94wNWGD3xkJwDs57eiZQWo7CvBD0AKRSs9kMVRU0m81GozHxy5ZlKYqygbk0Qtf6drt9fHwcMTVdaP+MaRVX46n9fp/vuRwEgeM4bL4TCJl398ZCcA7OdXomViqOwtw2P2AMAFai2+3SOXhM0yyXy2wcaafT4efpCQ0xXavRaERXSoe2NhqN6KkJx+9V+km/3y8UCnS8rjj15ubGNM1arUYPU61Wk2XZtu1YNiSZlty9cZl2DoqTEiulR2Eu+MNRgHTzPC8IgkRNakznHZ6WJXFqtrEpejEjc5YIzsEEnp45h6AHAAAAcgF9egAAACAXEPQAAABALiDoAQAAgFxA0AOQNW6v519exp2L13iDQevp07hzsSHO8+durxd3LjIiuLpye73g6irujMzBGwy8wSDuXKxSGo/CNAh6ALLm+OOP7YRFGJbjtJ8/jzsXG1L95JMOgp4V8QaD448/TlcMYTmOlZj/GFmJNB6FaRD0AMDaqaXSl+fncediQ8qlkndxEXcuAGACBD0AAKsk7ezEnQUAmAxBDwAAAOQCgh4AAADIBQQ9AAAAkAsIegAAACAXEPQAAABALiDoAYC1e/uttwghSZsycU2k7W1/OIw7FwAwAYIeAFg7tVQihOQkFFCKxRf52FKA1EHQAwAAALmAoAcAAAByAUEPAAAA5AKCHgAAAMgFBD0AAACQCwh6AAAAIBcQ9ADA2ql7e4SQTq8Xd0Y24VCWCSFuPjYWIF0Q9ADA2kk7O3FnYXOk7e24swAAk/3Fw4cP484DAKzS//39748PDpTd3bgz8preN99U33knablah+D6+oaQ//S97+Uq1Fuff/uXf3l8cJCinfm7P/zh4LvfpbWbmZG6ozDN1s3NTdx5AAAAAFg7NG8BAABALiDoAQAAgFxA0AMAAAC5gKAHAKayLEvTtFartb5VaJrmuu4KF2gYhqZpnuete0XLS2CWEmvaYU2jLG3LAlzX1TTNsqxY1o6gByA1NE3b8Bqbzaaqqv1+f32rUFVVkqQVLrDVaqmqGgTBule0vARmKbGmHdYFbP48ClnhtqQRjfXjivnejGWtAABUs9nM2IqiS2CWALINNT0AKUArhD3P0zjsSbHVatFP2MPT+Cd0CaqqKoqi67rv+3NlwLIs9VaoRYYmaZqmKIphGKHnV8dxFEWhqzYMg6/TppX80+r5Pc/TdZ2t1LIsx3GmZY9fy3gkMW1F9HO6Fl3X6fO3oij819iSQ1tH6+dpTdjEhQs2fOa2TztYS6505Vjrp2VZbC/xh4mVWM/zaN4UReFbS8XFcuJhDTWOsKLO/3Ba4RGfRxE3eWJpn+vQLBzs8vuZboumaXSnTSuolHg/CxYrPoLT9sbMgsH/nC6B/Zatke0lurSV1c/dAEBKVCoVQdJoNLq5uen3+/SFbdumadLUbrdbLpf7/T592+l0ZFmmX5vJNM1CodBoNOjbfr9fLpc7nQ77Av+60Wiwb1KyLLP1ttvt8U0wTZNfAluLLMvdbpdfKducENu2a7Ua25xarSbL8vgyJ66IEGLbNv0VzVu/3z85OZm45FD+CSEsS+12m/0q4oZPy5L4YC2/0tWixYPuw5vbI9Vut/nvVCqVcrlM9+RoNGIlRLyl4sMa2jT+Xjaz8CyzWwSlXXBoIhZRMdM02UJGo1G5XGZbLS6o4v0sWCwz7QgK9sbMgkEIYd+3bZvfXZ1Op1arhTLA8r8kBD0AqSG4WJumSS8o7PrO31BPTk5CV1jbtkPRiWDJoQtQp9MJ3Wu73W7nVig0kWXZtm12wWK3In75EwMUdrmk2u32tJvEeAAXCssEK2LXdz6V7efxJfMZK5fLfFKhUAjlSrzh07IkPljLr3S1xotHv98PFdRKpTIxYBVvqfiwCoKemYVnyVhwWmkXHJqIRVRMlmX+Le1pN235/E6YuZ+nLZaZdgRvpu+NmQUjtLvGywwfpYUWtQw0bwFkwfHxcbfb9TyPVlCT2wptmsq/phRF6XQ6ERce6mzLqr4JIXSNlmXRq954e4rruv1+n1ZQq6oasVmNLpb/RNf1afXbiqKM5zDKWmYKgoCul+HbLEIrffnyJf92sQ0nsw7Wmla6jFCWFEUZbzA6Pj4e/6F4Sxc+rHMVnrmIS7vg0KykiIY2in8rLqgz97NgLcz4EZx57osLhrgLf71ebzQa9HWj0ajX64IvzwUdmQGyQNM0eo2g0U+o+Zze/0JXSVVVF1uX7/vsgmUYhuM4bFGu6/KxFL3GsbZ53/fpFXnmkKWJN05BfkKfuK478S47L9oRYYEBVgtvOFniYC2z0mWEOq+4rjvtxhki3tKFD+tchWcu4tIusJIi6vt+EATsaPK7XVxQZ+7naYsVm7k3Fi4Y5PaCRp/faEeiiD+cCTU9AKkhSRJ/HQldSYMgcBxH1/XT01P6rM+S6Cf8N23bPj09jbjeVqvFr9eyLPbbIAjY9Ygulv9ho9HgP6EX1ig3pGq12mg0+G+6rjutW66u63xSq9VaVQ1HtVoNrdR13Sgz6yy84WSJg7XMSpfheR7rxOq6rmEYEZ/LxVsqPqz861C/4JmFR3weCYhLu8BKimiz2WSVrKEtEhdU8X4WLFZs5t5YuGBQtLLHtu0VD3JcVTsZAKxbt9uVZdk0TdM0y+XyyckJ35DPN6LLshzqTGrbNu3uU6vVxruaTmOaJl1jpVI5OTmhb/nOQLZt0y/QxdIvsGw0Go1yucyvl/UzsG27UqlUKhVZlsvlMn3N90GhS67VanRj+X6aE/NJ13JycsJ+QtclWFGtViOE0NzSn9DPC4UC653Alszvc9o7oVAosL5NJycnhOvHKtjwiNs+frCWXOma0D3DtqhWq/EdTmnJKRQKbDPnKpaCw0qXzL7AjiNbrKDwiM8jgWmlfeahEW9LdLQ7HdvP5PWeTOMFNeJ+Fi922hEUn/uCgtHtdvndNRqN6NvxvlaCvkQLw7+sA6RJEAT0IXWBee3Ybxfr3+B5Hq0VD62XLlaSpGlNMMuslz6tRtlY3/dpHf4Ka8IXyAZvmQ1f+LdLHuV50YqBZZ7FxRkWHNZpBZIRHLWFz6OZpV1g5UVUkqTxoelk1ibPLBjji51GsDeWLxiEEDqRxGrbZxH0AADAglZyb4MFrGnPr2qxyy/HcZxut7vyDUSfHgAAWIRlWWdnZ2dnZ1tbW4KpI2FV6BiCra0tWvmxqjFNK1/sMgWDZaZarZ6dna18gk3U9AAAAEAuoKYHAAAAcgFBDwAAAOQCgh4AAADIBQQ9AAAAkAv/HyKMSlV3o84MAAAAAElFTkSuQmCC",
      "text/plain": [
       "Tree('ROOT', [Tree('S', [Tree('NP', [Tree('JJ', ['Conceptually']), Tree('NN', ['cream']), Tree('NN', ['skimming'])]), Tree('VP', [Tree('VBZ', ['has']), Tree('NP', [Tree('NP', [Tree('CD', ['two']), Tree('JJ', ['basic']), Tree('NNS', ['dimensions'])]), Tree(':', ['-']), Tree('NP', [Tree('NN', ['product']), Tree('CC', ['and']), Tree('NN', ['geography'])])])]), Tree('.', ['.'])])])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnli_ex.premise_parse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The leaves of either tree are tokenized versions of them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Conceptually',\n",
       " 'cream',\n",
       " 'skimming',\n",
       " 'has',\n",
       " 'two',\n",
       " 'basic',\n",
       " 'dimensions',\n",
       " '-',\n",
       " 'product',\n",
       " 'and',\n",
       " 'geography',\n",
       " '.']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mnli_ex.premise_parse.leaves()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Annotated MultiNLI subsets\n",
    "\n",
    "MultiNLI includes additional annotations for a subset of the dev examples. The goal is to help people understand how well their models are doing on crucial NLI-related linguistic phenomena."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "matched_ann_filename = os.path.join(\n",
    "    ANNOTATIONS_HOME,\n",
    "    \"multinli_1.0_matched_annotations.txt\")\n",
    "\n",
    "mismatched_ann_filename = os.path.join(\n",
    "    ANNOTATIONS_HOME,\n",
    "    \"multinli_1.0_mismatched_annotations.txt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def view_random_example(annotations, random_state=42):\n",
    "    random.seed(random_state)\n",
    "    ann_ex = random.choice(list(annotations.items()))\n",
    "    pairid, ann_ex = ann_ex\n",
    "    ex = ann_ex['example']\n",
    "    print(\"pairID: {}\".format(pairid))\n",
    "    print(ann_ex['annotations'])\n",
    "    print(ex.premise)\n",
    "    print(ex.label)\n",
    "    print(ex.hypothesis)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "matched_ann = nli.read_annotated_subset(\n",
    "    matched_ann_filename, \n",
    "    mnli['validation_matched'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pairID: 132936n\n",
      "['#NEGATION', '#COREF']\n",
      "This one-at-a-time, uncoordinated series of regulatory requirements for the power industry is not the optimal approach for the environment, the power generation sector, or American consumers.\n",
      "entailment\n",
      "It is not the optimal approach.\n"
     ]
    }
   ],
   "source": [
    "view_random_example(matched_ann, random_state=23)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adversarial NLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adversarial NLI properties\n",
    "\n",
    "The ANLI dataset was created in response to evidence that datasets like SNLI and MultiNLI are artificially easy for modern machine learning models to solve. The team sought to tackle this weakness head-on, by designing a crowdsourcing task in which annotators were explicitly trying to confuse state-of-the-art models. In broad outline, the task worked like this:\n",
    "\n",
    "1. The crowdworker is presented with a premise (context) text and asked to construct a hypothesis sentence that entails, contradicts, or is neutral with respect to that premise. (The actual wording is more informal, along the lines of the SNLI/MultiNLI task).\n",
    "\n",
    "1. The crowdworker submits a hypothesis text.\n",
    "\n",
    "1. The premise/hypothesis pair is  fed to a trained model that makes a prediction about the correct NLI label.\n",
    "\n",
    "1. If the model's prediction is correct, then the crowdworker loops back to step 2 to try again. If the model's prediction is incorrect, then the example is validated by different crowdworkers.\n",
    "\n",
    "The dataset consists of three rounds, each involving a different model and a different set of sources for the premise texts:\n",
    "\n",
    "| Round  | Model       | Training data              | Context sources  | \n",
    "|:------:|:------------|:---------------------------|:-----------------|\n",
    "| 1 |  [BERT-large](https://www.aclweb.org/anthology/N19-1423/) | SNLI + MultiNLI | Wikipedia |\n",
    "| 2      |  [ROBERTa](https://arxiv.org/abs/1907.11692)    | SNLI + MultiNLI + [NLI-FEVER](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md) + Round 1 | Wikipedia        |\n",
    "| 3      |  [ROBERTa](https://arxiv.org/abs/1907.11692)    | SNLI + MultiNLI + [NLI-FEVER](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md) + Round 2 | Various          |\n",
    "\n",
    "Each round has train/dev/test splits. The sizes of these splits and their label distributions are calculated just below.\n",
    "\n",
    "The [project README](https://github.com/facebookresearch/anli/blob/master/README.md) seeks to establish some rules for how the rounds can be used for training and evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with Adversarial NLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Reusing dataset anli (/Users/cgpotts/.cache/huggingface/datasets/anli/plain_text/0.1.0/aabce88453b06dff21c201855ea83283bab0390bff746deadb30b65695755c0b)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1a1ba36788af4c89bad2cc6af3389eee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/9 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "anli = load_dataset(\"anli\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For ANLI, we have a lot of options. Because it is distributed in three rounds, and the rounds can be used independently or pooled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['train_r1', 'dev_r1', 'test_r1', 'train_r2', 'dev_r2', 'test_r2', 'train_r3', 'dev_r3', 'test_r3'])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anli.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is the fully pooled train setting:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "anli_pooled_reader = nli.NLIReader(\n",
    "    anli['train_r1'], anli['train_r2'], anli['train_r3'],\n",
    "    filter_unlabeled=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "neutral          68789\n",
       "entailment       52111\n",
       "contradiction    41965\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anli_pooled_labels = pd.Series([ex.label for ex in anli_pooled_reader.read()])\n",
    "\n",
    "anli_pooled_labels.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "R(1,): 16,946\n",
      "R(2,): 45,460\n",
      "R(3,): 100,459\n",
      "R(1, 2, 3): 162,865\n"
     ]
    }
   ],
   "source": [
    "for rounds in ((1,), (2,), (3,), (1,2,3)):\n",
    "    splits = [anli['train_r{}'.format(i)] for i in rounds]            \n",
    "    count = len(list(nli.NLIReader(*splits).read()))\n",
    "    print(\"R{0:}: {1:,}\".format(rounds, count))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above figures correspond to those in Table 2 of the paper."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a summary of what `NLIExample` instances offer for this corpus:\n",
    "\n",
    "* __uid__: a unique identifier; akin to `pairID` in SNLI/MultiNLI \n",
    "* __premise__: the premise; corresponds to `sentence1` in SNLI/MultiNLI\n",
    "* __hypothesis__: the hypothesis; corresponds to `sentence2` in SNLI/MultiNLI\n",
    "* __label__: the gold label; corresponds to `gold_label` in SNLI/MultiNLI\n",
    "* __reason__: a crowdworker's free-text hypothesis about why the model made an incorrect prediction for the current __context__/__hypothesis__ pair\n",
    "\n",
    "The ANLI distribution contains additional fields that are unfortunately left out of the Hugging Face distribution:\n",
    "\n",
    "* __model_label__: the label predicted by the model used in the current round\n",
    "* __emturk__: for dev (and test), this is `True` if the annotator contributed only dev (test) exmples, else `False`; in turn, it is `False` for all train examples.\n",
    "* __genre__: the source for the __context__ text\n",
    "* __tag__: information about the round and train/dev/test classification\n",
    "\n",
    "As with the other datasets, it would be a wonderful service to the field to [improve the interface](https://huggingface.co/datasets/anli)!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "anli_ex = next(iter(nli.NLIReader(anli['dev_r3']).read()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"NLIExample({'uid': '50a5d4bc-bb8b-44f9-ba07-95dfaf5536ab', 'premise': 'one of the orders issued by Ochola in April Login to license this image from 1$. In short At Kira Road police station, the photocopier business has moved behind the station, far away from the prying eyes of those passing on the road to Bukoto while at Old Kampala Police station, clients are now buying the forms across the road.', 'hypothesis': 'The decision to move the photocopier business was done for privacy reasons.', 'label': 'entailment', 'reason': 'I made an obvious inference from the text that a human would understand. This is not something that can be found by literally reading the words on the page.'})"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anli_ex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other NLI datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are a lot of NLI datasets available from the Hugging Face `datasets` library. [Here's a link to the full index](https://huggingface.co/datasets?task_ids=task_ids:natural-language-inference), and here are some highlights:\n",
    "\n",
    "* [The SemEval 2014 semantic relatedness shared task](http://alt.qcri.org/semeval2014/task1/) used an NLI dataset called [Sentences Involving Compositional Knowledge (SICK)](http://alt.qcri.org/semeval2014/task1/index.php?id=data-and-tools). HF: https://huggingface.co/datasets/sick\n",
    "\n",
    "* [XNLI](https://github.com/facebookresearch/XNLI) is a multilingual collection of test sets derived from MultiNLI. HF: https://huggingface.co/datasets/xnli\n",
    "\n",
    "* [Turkish NLI](https://github.com/boun-tabi/NLI-TR) provides human-validated machine-translated versions of SNLI and MultiNLI. These seem not to show up in the above index, but they can be accessed via:\n",
    "   ```\n",
    "   snli_tr_dataset = load_dataset('nli_tr', 'snli_tr')\n",
    "   multinli_tr_dataset = load_dataset('nli_tr', 'multinli_tr')\n",
    "   ```\n",
    "   \n",
    "And here are some tips on additional NLI datasets that aren't yet on Hugging Face:\n",
    "\n",
    "* [The FraCaS textual inference test suite](http://www-nlp.stanford.edu/~wcmac/downloads/) is a smaller, hand-built dataset that is great for evaluating a model's ability to handle complex logical patterns.\n",
    "\n",
    "* [SemEval 2013](https://www.cs.york.ac.uk/semeval-2013/) had a wide range of interesting data sets for NLI and related tasks.\n",
    "\n",
    "* [The SemEval 2014 semantic relatedness shared task](http://alt.qcri.org/semeval2014/task1/) used an NLI dataset called [Sentences Involving Compositional Knowledge (SICK)](http://alt.qcri.org/semeval2014/task1/index.php?id=data-and-tools).\n",
    "\n",
    "* [MedNLI](https://physionet.org/physiotools/mimic-code/mednli/) is specialized to the medical domain, using data derived from [MIMIC III](https://mimic.physionet.org).\n",
    "\n",
    "* [Diverse Natural Language Inference Collection (DNC)](http://decomp.io/projects/diverse-natural-language-inference/) transforms existing annotations from other tasks into NLI problems for a diverse range of reasoning challenges.\n",
    "\n",
    "* [SciTail](http://data.allenai.org/scitail/) is an NLI dataset derived from multiple-choice science exam questions and Web text.\n",
    "\n",
    "* [NLI Style FEVER](https://github.com/easonnie/combine-FEVER-NSMN/blob/master/other_resources/nli_fever.md) is a version of [the FEVER dataset](http://fever.ai) put into a standard NLI format. It was used by the Adversarial NLI team to train models for their annotation round 2.\n",
    "\n",
    "* [OCNLI: Original Chinese Natural Language Inference](https://github.com/CLUEbenchmark/OCNLI): A large, high-quality MNLI-style NLI dataset for Mandarin."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}