Homework-WS21.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Homework (Mini-project)\n",
    "\n",
    "Objective of this assignment is to implement the basic building blocks of a Deep Learning pipeline on a sample supervised-learning problem in **PyTorch**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Name: Omar Elsobky\n",
    "\n",
    "Matriculation No.: 03737994 \n",
    "\n",
    "**Important:** Do not forget to fill the places where you see `### Your code goes here ###`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "______________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task and Setup\n",
    "\n",
    "In this assignment, we want you to experience doing a mini-project in PyTorch. You are supposed to build the different parts of the pipeline as illustrated in the course notebooks (DataLoading, Model, Loss, Training, Evaluation). For this purpose, we will use a dataset from Quora containing question pairs and labels whether the pair is a duplicate or not.\n",
    "\n",
    "The data can be found in Moodle in a csv file. The data is This data is subject to Quora's [Terms of Service](https://www.quora.com/about/tos), allowing for non-commercial use. The dataset was downloaded from https://www.kaggle.com/quora/question-pairs-dataset. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "**Classifiying Duplicate Question Pairs** \n",
    "\n",
    "\n",
    "We want to build a DL Model that can predict whether two questions from a Quora dataset are duplicates or not. Note that the two questions must not be identical as you will see in the dataset, rather they semantically mean almost the same thing.\n",
    "\n",
    "To make the setup a bit simpler, we extracted and prepared a small subset of the original data consisting of 50k examples. Additionally, we removed questions that are too long or too short, we kept questions of length between 30 and 50 characters. Those 50k examples should serve as training and validation data, please consider making a reasonable split. Do not train on the validation data, just use it to evaluate your model.\n",
    "\n",
    "**Model Inputs and Label**:\n",
    "\n",
    "Input Format: 2 questions, for each question you will have an input of `BATCH_SIZE X SEQ_LEN`, where SEQ_LEN is the number of tokens in the question. Of course, if you will stack the input into batches, you will need to pad the questions (i.e. add a padding token or zeros at the end of the question to make all questions equal in length).\n",
    "\n",
    "Label Format: `BATCH_SIZE X 1`, please note that the extra dimension (`X 1`) is optional and dependent on your implementaiton, you could have a simple 1D tensor of length `BATCH_SIZE`, where each value is either 0 or 1 indicating that the two questions are either non-duplicates or not, respectively.\n",
    "\n",
    "**Hints:**\n",
    "\n",
    "1. Please read the csv file and explore the dataset a bit in order to familize yourself with the problem before working on it.\n",
    "\n",
    "2. In your custom DataLoader you have to make sure that you provide two questions for each example, this should be done in the `def __getitem__(self, idx)` method.\n",
    "\n",
    "3. Please work at the word-level, your tokens are words. You will need to preprocess the data accordingly. Feel free to write simple Python code that can do the job, but also consider using tokenizers, stemmers, and lemmatizers from known NLP libraries such as [NLTK](https://www.nltk.org/) or [SpaCy](https://spacy.io/).\n",
    "\n",
    "4. You will need to encode the words into integers to be able to pass them to the model, you will also need to keep track of the vocabulary. For this purpose, you can also write your own Python code or use an out-of-the-box module such as `torchtext.data.vocab` (see example in the data loading notebook). You can include this part in your Dataset class if you like.\n",
    "\n",
    "5. You will most probalby need to use an embedding layer as input to the model, it will then take the sequence of integers and return numeric vectors representing each word. Please consider using pre-trained embeddings, there are multiple ways how to load these into your newly-created Embedding layer, `torchtext` also provides some easy ways to do that.\n",
    "\n",
    "6. With Embeddings, you will have two options:\n",
    "    - train your own embeddings on the task, either by starting from random weights or after loading pre-trained embeddings (this will take more time and probably need Colab or GPU)\n",
    "    - or freeze the pre-trained embeddings and train the rest of the network (make sure the embedding layer is frozen, `requires_grad` is set to `False`.\n",
    "\n",
    "7. Note that you will need to encode the questions as integers based on the vocabulary you are using. This sequence of integers will be fed as input to the model (embeddings lookup, then the following layers).\n",
    "\n",
    "8. Take care that the model have two inputs (two sentences in parallel). This should be done in your implementation of the `def forward(self, question1, qustion2)` in your custom model class.\n",
    "\n",
    "9. Since you need to feed both question to your model, in the `forward` you will have to let each question go through a couple of layers to get a representation for each question. Then, you will have to combine the two representations in any way you see possible (e.g. multiply them, subtract them, concatenate them). Finally, with this final representation, you will have to let it go through a couple of layers (mostly fully-connected layers) and then predict the outcome (2 classes).\n",
    "\n",
    "10. This is generally a binary classification problem, you can use a classification loss to train your model. There are more advanced loss functions that are related to Siamese Networks (which is this architecture since it has multiple parallel inputs), feel free to use or explore them.\n",
    "\n",
    "11. A nice lecture about the topic is here: Siamese Networks and Similarity Learning Lecture, Prof. Dr. Laura Leal-Taixé, Advanced Deep Learning for Computer Vision Course:https://www.youtube.com/watch?v=6e65XfwmIWE\n",
    "\n",
    "12. Good summary and course notes of Deep Learning specialization on Coursera: https://github.com/mbadry1/DeepLearning.ai-Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing Libs below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from os.path import exists, join\n",
    "from os import mkdir\n",
    "import glob\n",
    "import nltk\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 1: Data Loading (30 Points)\n",
    "\n",
    "1. Write code to read the dataset after you download it from Moodle.\n",
    "2. Explore some examples and check if you need to do some data cleaning or remove some bad examples.\n",
    "3. Decide on what preprocessing steps you will do to the text of the questions.\n",
    "4. Build a custom PyTorch dataset where you implement the required methods `__getitem__` and `__len__`. Do not forget to integrate any preprocessing steps in the class. Make sure that you also have a function that applies the whole preprocessing to a raw example, this will be very helpful when you want to predict for test examples later.\n",
    "5. Split the data into train and validation data. Use a reasonable split ratio.\n",
    "6. Create PyTorch dataloaders for train and validation datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.\\\\data',\n",
       " '.\\\\GoogleNews-vectors-negative300.bin.gz',\n",
       " '.\\\\Homework-WS21.ipynb',\n",
       " '.\\\\mini_quora_dataset_30_50_50k.csv',\n",
       " '.\\\\models',\n",
       " '.\\\\plots',\n",
       " '.\\\\test.csv',\n",
       " '.\\\\wandb']"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### Your code goes here ####\n",
    "##checking the files in this location\n",
    "glob.glob(join(\"./\", \"*\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>is_duplicate</th>\n",
       "      <th>max_length</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>307549</td>\n",
       "      <td>603786</td>\n",
       "      <td>603787</td>\n",
       "      <td>What are some different ways to make money fast?</td>\n",
       "      <td>What are fast ways to make money?</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>221916</td>\n",
       "      <td>437426</td>\n",
       "      <td>437427</td>\n",
       "      <td>How can I continue to improve my English?</td>\n",
       "      <td>How can I understand english?</td>\n",
       "      <td>1</td>\n",
       "      <td>41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>177830</td>\n",
       "      <td>351280</td>\n",
       "      <td>351281</td>\n",
       "      <td>How do I promote my youtube videos?</td>\n",
       "      <td>What is the best way to promote YouTube videos?</td>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>128018</td>\n",
       "      <td>253605</td>\n",
       "      <td>253606</td>\n",
       "      <td>How can I organize a Quora Meetup in Pune?</td>\n",
       "      <td>Is there a Pune Quora Meetup group?</td>\n",
       "      <td>0</td>\n",
       "      <td>42</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>177761</td>\n",
       "      <td>351144</td>\n",
       "      <td>351145</td>\n",
       "      <td>What is the most badass moment of Game of Thro...</td>\n",
       "      <td>Who will die in season 5 of Game of Thrones?</td>\n",
       "      <td>0</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>95213</td>\n",
       "      <td>189005</td>\n",
       "      <td>189006</td>\n",
       "      <td>How does drop shipping work exactly?</td>\n",
       "      <td>What is drop shipping and how does it work?</td>\n",
       "      <td>1</td>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>62682</td>\n",
       "      <td>124658</td>\n",
       "      <td>124659</td>\n",
       "      <td>What are the best movies to watch in Hollywood?</td>\n",
       "      <td>Which are the best Hollywood movies of all time?</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>109939</td>\n",
       "      <td>218008</td>\n",
       "      <td>218009</td>\n",
       "      <td>Am I a sociopath, schizoid, or neither?</td>\n",
       "      <td>Am I a sociopath?</td>\n",
       "      <td>0</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>370854</td>\n",
       "      <td>725712</td>\n",
       "      <td>725713</td>\n",
       "      <td>What is your marketing strategy?</td>\n",
       "      <td>What is a market strategy?</td>\n",
       "      <td>0</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>268327</td>\n",
       "      <td>527813</td>\n",
       "      <td>527814</td>\n",
       "      <td>What will be the qualifying marks for neet 2016?</td>\n",
       "      <td>What might be the qualifing mark in neet 2016?</td>\n",
       "      <td>1</td>\n",
       "      <td>48</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id    qid1    qid2  \\\n",
       "0      307549  603786  603787   \n",
       "1      221916  437426  437427   \n",
       "2      177830  351280  351281   \n",
       "3      128018  253605  253606   \n",
       "4      177761  351144  351145   \n",
       "...       ...     ...     ...   \n",
       "49995   95213  189005  189006   \n",
       "49996   62682  124658  124659   \n",
       "49997  109939  218008  218009   \n",
       "49998  370854  725712  725713   \n",
       "49999  268327  527813  527814   \n",
       "\n",
       "                                               question1  \\\n",
       "0       What are some different ways to make money fast?   \n",
       "1              How can I continue to improve my English?   \n",
       "2                    How do I promote my youtube videos?   \n",
       "3             How can I organize a Quora Meetup in Pune?   \n",
       "4      What is the most badass moment of Game of Thro...   \n",
       "...                                                  ...   \n",
       "49995               How does drop shipping work exactly?   \n",
       "49996    What are the best movies to watch in Hollywood?   \n",
       "49997            Am I a sociopath, schizoid, or neither?   \n",
       "49998                   What is your marketing strategy?   \n",
       "49999   What will be the qualifying marks for neet 2016?   \n",
       "\n",
       "                                              question2  is_duplicate  \\\n",
       "0                     What are fast ways to make money?             1   \n",
       "1                         How can I understand english?             1   \n",
       "2       What is the best way to promote YouTube videos?             1   \n",
       "3                   Is there a Pune Quora Meetup group?             0   \n",
       "4          Who will die in season 5 of Game of Thrones?             0   \n",
       "...                                                 ...           ...   \n",
       "49995       What is drop shipping and how does it work?             1   \n",
       "49996  Which are the best Hollywood movies of all time?             1   \n",
       "49997                                 Am I a sociopath?             0   \n",
       "49998                        What is a market strategy?             0   \n",
       "49999    What might be the qualifing mark in neet 2016?             1   \n",
       "\n",
       "       max_length  \n",
       "0              48  \n",
       "1              41  \n",
       "2              47  \n",
       "3              42  \n",
       "4              50  \n",
       "...           ...  \n",
       "49995          43  \n",
       "49996          48  \n",
       "49997          39  \n",
       "49998          32  \n",
       "49999          48  \n",
       "\n",
       "[50000 rows x 7 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Reading the CSV file\n",
    "questions_df = pd.read_csv(\"mini_quora_dataset_30_50_50k.csv\", sep=\",\")\n",
    "questions_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "id                                        201871\n",
       "qid1                                      398348\n",
       "qid2                                      398349\n",
       "question1       How can I create an Android app?\n",
       "question2                                    NaN\n",
       "is_duplicate                                   0\n",
       "max_length                                    32\n",
       "Name: 12405, dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions_df.iloc[12405]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>is_duplicate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>307549</td>\n",
       "      <td>603786</td>\n",
       "      <td>603787</td>\n",
       "      <td>What are some different ways to make money fast?</td>\n",
       "      <td>What are fast ways to make money?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>221916</td>\n",
       "      <td>437426</td>\n",
       "      <td>437427</td>\n",
       "      <td>How can I continue to improve my English?</td>\n",
       "      <td>How can I understand english?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>177830</td>\n",
       "      <td>351280</td>\n",
       "      <td>351281</td>\n",
       "      <td>How do I promote my youtube videos?</td>\n",
       "      <td>What is the best way to promote YouTube videos?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>128018</td>\n",
       "      <td>253605</td>\n",
       "      <td>253606</td>\n",
       "      <td>How can I organize a Quora Meetup in Pune?</td>\n",
       "      <td>Is there a Pune Quora Meetup group?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>177761</td>\n",
       "      <td>351144</td>\n",
       "      <td>351145</td>\n",
       "      <td>What is the most badass moment of Game of Thro...</td>\n",
       "      <td>Who will die in season 5 of Game of Thrones?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>95213</td>\n",
       "      <td>189005</td>\n",
       "      <td>189006</td>\n",
       "      <td>How does drop shipping work exactly?</td>\n",
       "      <td>What is drop shipping and how does it work?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>62682</td>\n",
       "      <td>124658</td>\n",
       "      <td>124659</td>\n",
       "      <td>What are the best movies to watch in Hollywood?</td>\n",
       "      <td>Which are the best Hollywood movies of all time?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>109939</td>\n",
       "      <td>218008</td>\n",
       "      <td>218009</td>\n",
       "      <td>Am I a sociopath, schizoid, or neither?</td>\n",
       "      <td>Am I a sociopath?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>370854</td>\n",
       "      <td>725712</td>\n",
       "      <td>725713</td>\n",
       "      <td>What is your marketing strategy?</td>\n",
       "      <td>What is a market strategy?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>268327</td>\n",
       "      <td>527813</td>\n",
       "      <td>527814</td>\n",
       "      <td>What will be the qualifying marks for neet 2016?</td>\n",
       "      <td>What might be the qualifing mark in neet 2016?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id    qid1    qid2  \\\n",
       "0      307549  603786  603787   \n",
       "1      221916  437426  437427   \n",
       "2      177830  351280  351281   \n",
       "3      128018  253605  253606   \n",
       "4      177761  351144  351145   \n",
       "...       ...     ...     ...   \n",
       "49995   95213  189005  189006   \n",
       "49996   62682  124658  124659   \n",
       "49997  109939  218008  218009   \n",
       "49998  370854  725712  725713   \n",
       "49999  268327  527813  527814   \n",
       "\n",
       "                                               question1  \\\n",
       "0       What are some different ways to make money fast?   \n",
       "1              How can I continue to improve my English?   \n",
       "2                    How do I promote my youtube videos?   \n",
       "3             How can I organize a Quora Meetup in Pune?   \n",
       "4      What is the most badass moment of Game of Thro...   \n",
       "...                                                  ...   \n",
       "49995               How does drop shipping work exactly?   \n",
       "49996    What are the best movies to watch in Hollywood?   \n",
       "49997            Am I a sociopath, schizoid, or neither?   \n",
       "49998                   What is your marketing strategy?   \n",
       "49999   What will be the qualifying marks for neet 2016?   \n",
       "\n",
       "                                              question2  is_duplicate  \n",
       "0                     What are fast ways to make money?             1  \n",
       "1                         How can I understand english?             1  \n",
       "2       What is the best way to promote YouTube videos?             1  \n",
       "3                   Is there a Pune Quora Meetup group?             0  \n",
       "4          Who will die in season 5 of Game of Thrones?             0  \n",
       "...                                                 ...           ...  \n",
       "49995       What is drop shipping and how does it work?             1  \n",
       "49996  Which are the best Hollywood movies of all time?             1  \n",
       "49997                                 Am I a sociopath?             0  \n",
       "49998                        What is a market strategy?             0  \n",
       "49999    What might be the qualifing mark in neet 2016?             1  \n",
       "\n",
       "[50000 rows x 6 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Removing the max_length column, Because i think it was added to filter the data and when i checked i did not find this column in the original data \n",
    "\n",
    "questions_df = questions_df.drop(labels='max_length', axis='columns')\n",
    "questions_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1e7e5de0248>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAD1CAYAAACyaJl6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAN3ElEQVR4nO3df4jc9Z3H8efrkqaU84qxrsFL0ov09rjGwqV20YD/eBXyw/sjFiroHzWIsKUkUKF/mPafFK2gf7QFwQopLkbomUp/YOilzYXgUcpVzdoGNc15WazVbYKuF2s9hHqx7/tjPkuHzWx2s5vsJO7zAcPMvOf7nXwGok/nO98ZU1VIkha3v+r3AiRJ/WcMJEnGQJJkDCRJGANJEsZAkgQs7fcC5uryyy+vNWvW9HsZknRRee65596sqoGp84s2BmvWrGF0dLTfy5Cki0qS3/Wae5hIkmQMJEnGQJKEMZAkYQwkSRgDSRLGQJKEMZAkcRF/6exisWbHv/V7CR8Yr9z/L/1egvSB5TsDSZIxkCQZA0kSs4hBktVJnkpyNMmRJF9u868n+X2Sw+1yU9c+X00yluSlJBu75pvabCzJjq75VUmeSXIsyfeTLDvXL1SSNL3ZvDM4BXylqj4JrAe2JVnbHvt2Va1rl30A7bFbgauBTcB3kixJsgR4CNgMrAVu63qeB9pzDQJvAXeeo9cnSZqFGWNQVSeq6lft9jvAUWDlGXbZAuypqj9V1W+BMeDadhmrqper6j1gD7AlSYDPAj9o++8Gbp7rC5Iknb2z+swgyRrg08AzbbQ9yfNJRpIsb7OVwGtdu4232XTzjwF/qKpTU+aSpAUy6xgkuQT4IXBXVf0ReBj4BLAOOAF8c3LTHrvXHOa91jCcZDTJ6MTExGyXLkmawaxikORDdELwvar6EUBVvV5V71fVn4Hv0jkMBJ3/sl/dtfsq4PgZ5m8ClyZZOmV+mqraVVVDVTU0MHDa/7VNkjRHszmbKMAjwNGq+lbX/MquzT4HvNhu7wVuTfLhJFcBg8CzwCFgsJ05tIzOh8x7q6qAp4DPt/23Ak/O72VJks7GbH6O4nrgC8ALSQ632dfonA20js4hnVeALwJU1ZEkTwC/oXMm0raqeh8gyXZgP7AEGKmqI+357gb2JPkG8Gs68ZEkLZAZY1BVv6D3cf19Z9jnPuC+HvN9vfarqpf5y2EmSdIC8xvIkiRjIEnyJ6ylRcufVz+3LvafWPedgSTJGEiSjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgSWIWMUiyOslTSY4mOZLky21+WZIDSY616+VtniQPJhlL8nySa7qea2vb/liSrV3zzyR5oe3zYJKcjxcrSeptNu8MTgFfqapPAuuBbUnWAjuAg1U1CBxs9wE2A4PtMgw8DJ14ADuB64BrgZ2TAWnbDHftt2n+L02SNFszxqCqTlTVr9rtd4CjwEpgC7C7bbYbuLnd3gI8Vh1PA5cmuRLYCByoqpNV9RZwANjUHvtoVf2yqgp4rOu5JEkL4Kw+M0iyBvg08AywoqpOQCcYwBVts5XAa127jbfZmebjPeaSpAUy6xgkuQT4IXBXVf3xTJv2mNUc5r3WMJxkNMnoxMTETEuWJM3SrGKQ5EN0QvC9qvpRG7/eDvHQrt9o83Fgddfuq4DjM8xX9Zifpqp2VdVQVQ0NDAzMZumSpFmYzdlEAR4BjlbVt7oe2gtMnhG0FXiya357O6toPfB2O4y0H9iQZHn74HgDsL899k6S9e3Pur3ruSRJC2DpLLa5HvgC8EKSw232NeB+4IkkdwKvAre0x/YBNwFjwLvAHQBVdTLJvcChtt09VXWy3f4S8CjwEeCn7SJJWiAzxqCqfkHv4/oAN/bYvoBt0zzXCDDSYz4KfGqmtUiSzg+/gSxJMgaSJGMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgSWIWMUgykuSNJC92zb6e5PdJDrfLTV2PfTXJWJKXkmzsmm9qs7EkO7rmVyV5JsmxJN9PsuxcvkBJ0sxm887gUWBTj/m3q2pdu+wDSLIWuBW4uu3znSRLkiwBHgI2A2uB29q2AA+05xoE3gLunM8LkiSdvRljUFU/B07O8vm2AHuq6k9V9VtgDLi2Xcaq6uWqeg/YA2xJEuCzwA/a/ruBm8/yNUiS5mk+nxlsT/J8O4y0vM1WAq91bTPeZtPNPwb8oapOTZn3lGQ4yWiS0YmJiXksXZLUba4xeBj4BLAOOAF8s83TY9uaw7ynqtpVVUNVNTQwMHB2K5YkTWvpXHaqqtcnbyf5LvCTdnccWN216SrgeLvda/4mcGmSpe3dQff2kqQFMqd3Bkmu7Lr7OWDyTKO9wK1JPpzkKmAQeBY4BAy2M4eW0fmQeW9VFfAU8Pm2/1bgybmsSZI0dzO+M0jyOHADcHmScWAncEOSdXQO6bwCfBGgqo4keQL4DXAK2FZV77fn2Q7sB5YAI1V1pP0RdwN7knwD+DXwyDl7dZKkWZkxBlV1W4/xtP/Crqr7gPt6zPcB+3rMX6ZztpEkqU/8BrIkyRhIkoyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJAljIEnCGEiSMAaSJIyBJIlZxCDJSJI3krzYNbssyYEkx9r18jZPkgeTjCV5Psk1XftsbdsfS7K1a/6ZJC+0fR5MknP9IiVJZzabdwaPApumzHYAB6tqEDjY7gNsBgbbZRh4GDrxAHYC1wHXAjsnA9K2Ge7ab+qfJUk6z2aMQVX9HDg5ZbwF2N1u7wZu7po/Vh1PA5cmuRLYCByoqpNV9RZwANjUHvtoVf2yqgp4rOu5JEkLZK6fGayoqhMA7fqKNl8JvNa13XibnWk+3mPeU5LhJKNJRicmJua4dEnSVOf6A+Rex/trDvOeqmpXVQ1V1dDAwMAclyhJmmquMXi9HeKhXb/R5uPA6q7tVgHHZ5iv6jGXJC2gucZgLzB5RtBW4Mmu+e3trKL1wNvtMNJ+YEOS5e2D4w3A/vbYO0nWt7OIbu96LknSAlk60wZJHgduAC5PMk7nrKD7gSeS3Am8CtzSNt8H3ASMAe8CdwBU1ckk9wKH2nb3VNXkh9JfonPG0keAn7aLJGkBzRiDqrptmodu7LFtAdumeZ4RYKTHfBT41EzrkCSdP34DWZJkDCRJxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCShDGQJGEMJEkYA0kSxkCSxDxjkOSVJC8kOZxktM0uS3IgybF2vbzNk+TBJGNJnk9yTdfzbG3bH0uydX4vSZJ0ts7FO4N/rqp1VTXU7u8ADlbVIHCw3QfYDAy2yzDwMHTiAewErgOuBXZOBkSStDDOx2GiLcDudns3cHPX/LHqeBq4NMmVwEbgQFWdrKq3gAPApvOwLknSNOYbgwL+PclzSYbbbEVVnQBo11e0+Urgta59x9tsuvlpkgwnGU0yOjExMc+lS5ImLZ3n/tdX1fEkVwAHkvzXGbZNj1mdYX76sGoXsAtgaGio5zaSpLM3r3cGVXW8Xb8B/JjOMf/X2+Ef2vUbbfNxYHXX7quA42eYS5IWyJxjkOSvk/zN5G1gA/AisBeYPCNoK/Bku70XuL2dVbQeeLsdRtoPbEiyvH1wvKHNJEkLZD6HiVYAP04y+Tz/WlU/S3IIeCLJncCrwC1t+33ATcAY8C5wB0BVnUxyL3CobXdPVZ2cx7okSWdpzjGoqpeBf+ox/x/gxh7zArZN81wjwMhc1yJJmh+/gSxJMgaSJGMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgScIYSJIwBpIkjIEkCWMgSeICikGSTUleSjKWZEe/1yNJi8kFEYMkS4CHgM3AWuC2JGv7uypJWjwuiBgA1wJjVfVyVb0H7AG29HlNkrRoLO33ApqVwGtd98eB66ZulGQYGG53/zfJSwuwtsXgcuDNfi9iJnmg3ytQn/j389z6u17DCyUG6TGr0wZVu4Bd5385i0uS0aoa6vc6pF78+7kwLpTDROPA6q77q4DjfVqLJC06F0oMDgGDSa5Ksgy4Fdjb5zVJ0qJxQRwmqqpTSbYD+4ElwEhVHenzshYTD73pQubfzwWQqtMOzUuSFpkL5TCRJKmPjIEkyRhIki6QD5C1sJL8I51veK+k832O48Deqjra14VJ6hvfGSwySe6m83MfAZ6lc1pvgMf9gUBdyJLc0e81fJB5NtEik+S/gaur6v+mzJcBR6pqsD8rk84syatV9fF+r+ODysNEi8+fgb8FfjdlfmV7TOqbJM9P9xCwYiHXstgYg8XnLuBgkmP85ccBPw78PbC9b6uSOlYAG4G3pswD/OfCL2fxMAaLTFX9LMk/0PnZ8JV0/iEbBw5V1ft9XZwEPwEuqarDUx9I8h8Lv5zFw88MJEmeTSRJMgaSJIyBJAljIEnCGEiSgP8HWpvAEemSwUEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "###checking the balance of the data \n",
    "\n",
    "questions_df.is_duplicate.value_counts().plot(kind=\"bar\")\n",
    "\n",
    "## From the results the data set is quiet balanced "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>qid1</th>\n",
       "      <th>qid2</th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>307549</td>\n",
       "      <td>603786</td>\n",
       "      <td>603787</td>\n",
       "      <td>What are some different ways to make money fast?</td>\n",
       "      <td>What are fast ways to make money?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>221916</td>\n",
       "      <td>437426</td>\n",
       "      <td>437427</td>\n",
       "      <td>How can I continue to improve my English?</td>\n",
       "      <td>How can I understand english?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>177830</td>\n",
       "      <td>351280</td>\n",
       "      <td>351281</td>\n",
       "      <td>How do I promote my youtube videos?</td>\n",
       "      <td>What is the best way to promote YouTube videos?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>128018</td>\n",
       "      <td>253605</td>\n",
       "      <td>253606</td>\n",
       "      <td>How can I organize a Quora Meetup in Pune?</td>\n",
       "      <td>Is there a Pune Quora Meetup group?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>177761</td>\n",
       "      <td>351144</td>\n",
       "      <td>351145</td>\n",
       "      <td>What is the most badass moment of Game of Thro...</td>\n",
       "      <td>Who will die in season 5 of Game of Thrones?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>95213</td>\n",
       "      <td>189005</td>\n",
       "      <td>189006</td>\n",
       "      <td>How does drop shipping work exactly?</td>\n",
       "      <td>What is drop shipping and how does it work?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>62682</td>\n",
       "      <td>124658</td>\n",
       "      <td>124659</td>\n",
       "      <td>What are the best movies to watch in Hollywood?</td>\n",
       "      <td>Which are the best Hollywood movies of all time?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>109939</td>\n",
       "      <td>218008</td>\n",
       "      <td>218009</td>\n",
       "      <td>Am I a sociopath, schizoid, or neither?</td>\n",
       "      <td>Am I a sociopath?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>370854</td>\n",
       "      <td>725712</td>\n",
       "      <td>725713</td>\n",
       "      <td>What is your marketing strategy?</td>\n",
       "      <td>What is a market strategy?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>268327</td>\n",
       "      <td>527813</td>\n",
       "      <td>527814</td>\n",
       "      <td>What will be the qualifying marks for neet 2016?</td>\n",
       "      <td>What might be the qualifing mark in neet 2016?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           id    qid1    qid2  \\\n",
       "0      307549  603786  603787   \n",
       "1      221916  437426  437427   \n",
       "2      177830  351280  351281   \n",
       "3      128018  253605  253606   \n",
       "4      177761  351144  351145   \n",
       "...       ...     ...     ...   \n",
       "49995   95213  189005  189006   \n",
       "49996   62682  124658  124659   \n",
       "49997  109939  218008  218009   \n",
       "49998  370854  725712  725713   \n",
       "49999  268327  527813  527814   \n",
       "\n",
       "                                               question1  \\\n",
       "0       What are some different ways to make money fast?   \n",
       "1              How can I continue to improve my English?   \n",
       "2                    How do I promote my youtube videos?   \n",
       "3             How can I organize a Quora Meetup in Pune?   \n",
       "4      What is the most badass moment of Game of Thro...   \n",
       "...                                                  ...   \n",
       "49995               How does drop shipping work exactly?   \n",
       "49996    What are the best movies to watch in Hollywood?   \n",
       "49997            Am I a sociopath, schizoid, or neither?   \n",
       "49998                   What is your marketing strategy?   \n",
       "49999   What will be the qualifying marks for neet 2016?   \n",
       "\n",
       "                                              question2  label  \n",
       "0                     What are fast ways to make money?      1  \n",
       "1                         How can I understand english?      1  \n",
       "2       What is the best way to promote YouTube videos?      1  \n",
       "3                   Is there a Pune Quora Meetup group?      0  \n",
       "4          Who will die in season 5 of Game of Thrones?      0  \n",
       "...                                                 ...    ...  \n",
       "49995       What is drop shipping and how does it work?      1  \n",
       "49996  Which are the best Hollywood movies of all time?      1  \n",
       "49997                                 Am I a sociopath?      0  \n",
       "49998                        What is a market strategy?      0  \n",
       "49999    What might be the qualifing mark in neet 2016?      1  \n",
       "\n",
       "[50000 rows x 6 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### renaming the is_duplicate column to label\n",
    "\n",
    "questions_df.rename(columns={\"is_duplicate\": \"label\"}, inplace=True)\n",
    "questions_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim\n",
    "# ! wget -c \"https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\"\n",
    "word2vec_model = gensim.models.KeyedVectors.load_word2vec_format(\n",
    "            'GoogleNews-vectors-negative300.bin.gz', binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to C:\\Users\\Omar El-\n",
      "[nltk_data]     Sobky\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package stopwords to C:\\Users\\Omar El-\n",
      "[nltk_data]     Sobky\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "#### NOOW doing the rest if the data preprocessing in the dataLoader class!\n",
    "\n",
    "from torch.utils.data import Dataset\n",
    "import os\n",
    "import string\n",
    "import nltk\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "nltk.download('punkt')\n",
    "nltk.download('stopwords')\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "class QuestionsDataset(Dataset):\n",
    "    \"\"\" Questions Dataset.\"\"\"\n",
    "\n",
    "    #define the constructor of this dataset object\n",
    "    def __init__(self, questions_df, questions=False):\n",
    "        \"\"\" \n",
    "\n",
    "        Args: \n",
    "            questions_df(pd.DataFarme): DataFrame object contains the two questions and the label \n",
    "            questions(boolean): Boolean object to give back the questions ins text form  or not \n",
    "        \"\"\"\n",
    "        self.questions_df = questions_df\n",
    "        self.questions = questions\n",
    "        self.zero_counter = 0\n",
    "        self.word2vec_model = word2vec_model\n",
    "        self.clean_data()\n",
    "        self.prepare_data()\n",
    "        \n",
    "\n",
    "    def __len__(self):\n",
    "        return self.q1_vec_tensor.shape[0]\n",
    "\n",
    "    def __getitem__(self,idx):\n",
    "        if self.questions:\n",
    "            return self.q1_vec_tensor[idx], self.q2_vec_tensor[idx], self.label[idx], self.questions_df[\"question1\"][idx], self.questions_df[\"question2\"][idx]\n",
    "        else:\n",
    "            return self.q1_vec_tensor[idx], self.q2_vec_tensor[idx], self.label[idx]\n",
    "\n",
    "    def clean_data(self):\n",
    "        ### retriving the max length of all questions so i can get the embeddings of of all questions with the same seq size\n",
    "        # self.seq_len = self.questions_df[\"max_length\"].max()\n",
    "        # print(self.seq_len)\n",
    "        self.questions_df = self.questions_df.drop(labels=['max_length','id', 'qid1', 'qid2'], axis='columns')\n",
    "        self.questions_df.rename(columns={\"is_duplicate\": \"label\"}, inplace=True)\n",
    "        # display(self.questions_df.head())\n",
    "\n",
    "        #### Checking if there was any null values in any of the features\n",
    "        # print(self.questions_df.isna().sum())\n",
    "        # print(self.questions_df.shape)\n",
    "        # display(self.questions_df[self.questions_df.isnull().any(axis=1)])\n",
    "\n",
    "        ### After Checking, thre are two rows where question2 is NaN. We will keep only the rows with no NaN values \n",
    "        self.questions_df.dropna(inplace=True)\n",
    "        # print(self.questions_df.shape)\n",
    "        # print(self.questions_df.isna().sum())\n",
    "\n",
    "        display(self.questions_df.head())\n",
    "\n",
    "        \n",
    "    \n",
    "\n",
    "    def prepare_data(self):\n",
    "        ### Here we need to tokonize the sentnces and then get the embeddings of each work in the senctnce. \n",
    "        ### I will be creating a function called sent2vec which take the whole sentence and create a vector for the whole sentance \n",
    "        ### After checking the data a little bit I found the largest seq number is 11 so i will padd evey q to have 11 seq length\n",
    "        # self.seq_len = 0\n",
    "        # print(\"in preapare data \",self.questions_df.shape)\n",
    "        q1_vec = []\n",
    "        q2_vec = []\n",
    "        for q1, q2 in zip(self.questions_df[\"question1\"], self.questions_df[\"question2\"]):\n",
    "            q1_vec.append(self.question2vec(q1))\n",
    "            q2_vec.append(self.question2vec(q2))\n",
    "            \n",
    "\n",
    "        # print(self.zero_counter)\n",
    "            \n",
    "\n",
    "\n",
    "        # print(\"seq length: \", self.seq_len)\n",
    "\n",
    "        self.q1_vec_tensor = torch.tensor(np.array(q1_vec))\n",
    "        self.q2_vec_tensor = torch.tensor(np.array(q2_vec))\n",
    "        self.label = torch.tensor(np.array(self.questions_df[\"label\"]))\n",
    "\n",
    "\n",
    "        \n",
    "        # print(\"q1 shape:\", self.q1_vec_tensor.shape)\n",
    "        # # print(self.q1_vec_tensor)\n",
    "        # # print(\"------------------------------------\")\n",
    "        \n",
    "        # print(\"q2 shape:\", self.q2_vec_tensor.shape)\n",
    "        # print(\"labels shape:\", self.label.shape )\n",
    "        # print(\"seq length: \", self.seq_len)\n",
    "        # self.questions_df[\"q1_tokens\"] = q1_tokens\n",
    "        # self.questions_df[\"q2_tokens\"] = q2_tokens\n",
    "        # print(q1_tokens)\n",
    "\n",
    "    \n",
    "    ### this function is inspired after reading about word2vec and this artical https://hub.packtpub.com/use-tensorflow-and-nlp-to-detect-duplicate-quora-questions-tutorial/ \n",
    "    ### I will be using pretrained Word2vec model GoogleNews-vectors-negative (which kinda transferlearning for the embeddings)\n",
    "    ### I will be using question_pair2vec to retrive the embeddings of both question with the same seq size \n",
    "    def question2vec(self, q):\n",
    "        \n",
    "        M = []\n",
    "        X = []\n",
    "        # print(q)\n",
    "        words = nltk.word_tokenize(q)\n",
    "        for w in words:\n",
    "            if w not in stop_words:\n",
    "                if w.isalpha():\n",
    "                    if w in self.word2vec_model:\n",
    "                        M.append(self.word2vec_model[w])\n",
    "                        X.append(w)\n",
    "        M = np.array(M)\n",
    "        if len(M) > 0:\n",
    "            if len(M) < 11:\n",
    "                padding_size = 11 - len(M)\n",
    "                for _ in range(padding_size):\n",
    "                    M = np.append(M ,np.zeros((1,300)), axis=0)\n",
    "            # v = M.sum(axis=0)\n",
    "            # print(M)\n",
    "            # self.seq_len = len(M) if (len(M) > self.seq_len) else self.seq_len\n",
    "            # print(self.seq_len)\n",
    "            return M\n",
    "            # return v / np.sqrt((v ** 2).sum())\n",
    "        else:\n",
    "            self.zero_counter+=1\n",
    "            return np.zeros((11,300))\n",
    "\n",
    "\n",
    "\n",
    "        \n",
    "\n",
    "       \n",
    "    \n",
    "\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>12405</th>\n",
       "      <td>How can I create an Android app?</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41646</th>\n",
       "      <td>How can I develop android app?</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              question1 question2  label\n",
       "12405  How can I create an Android app?       NaN      0\n",
       "41646    How can I develop android app?       NaN      0"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What are some different ways to make money fast?</td>\n",
       "      <td>What are fast ways to make money?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How can I continue to improve my English?</td>\n",
       "      <td>How can I understand english?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>How do I promote my youtube videos?</td>\n",
       "      <td>What is the best way to promote YouTube videos?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>How can I organize a Quora Meetup in Pune?</td>\n",
       "      <td>Is there a Pune Quora Meetup group?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is the most badass moment of Game of Thro...</td>\n",
       "      <td>Who will die in season 5 of Game of Thrones?</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                           question1  \\\n",
       "0   What are some different ways to make money fast?   \n",
       "1          How can I continue to improve my English?   \n",
       "2                How do I promote my youtube videos?   \n",
       "3         How can I organize a Quora Meetup in Pune?   \n",
       "4  What is the most badass moment of Game of Thro...   \n",
       "\n",
       "                                         question2  label  \n",
       "0                What are fast ways to make money?      1  \n",
       "1                    How can I understand english?      1  \n",
       "2  What is the best way to promote YouTube videos?      1  \n",
       "3              Is there a Pune Quora Meetup group?      0  \n",
       "4     Who will die in season 5 of Game of Thrones?      0  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "in preapare data  (49998, 3)\n",
      "4\n",
      "q1 shape: torch.Size([49998, 11, 300])\n",
      "q2 shape: torch.Size([49998, 11, 300])\n",
      "labels shape: torch.Size([49998])\n"
     ]
    }
   ],
   "source": [
    "## creating instance of questions_Dataset\n",
    "questions_dataset = QuestionsDataset(pd.read_csv(\"mini_quora_dataset_30_50_50k.csv\", sep=\",\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[ 0.1602,  0.2168,  0.0549,  ...,  0.0454, -0.0811,  0.0894],\n",
       "         [-0.0742, -0.0737, -0.0194,  ...,  0.0586,  0.0481, -0.2217],\n",
       "         [ 0.1699,  0.0143, -0.0454,  ...,  0.0415, -0.1001, -0.1885],\n",
       "         ...,\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]],\n",
       "        dtype=torch.float64),\n",
       " tensor([[ 0.1602,  0.2168,  0.0549,  ...,  0.0454, -0.0811,  0.0894],\n",
       "         [-0.0742, -0.0737, -0.0194,  ...,  0.0586,  0.0481, -0.2217],\n",
       "         [ 0.1807, -0.1562,  0.2656,  ..., -0.0752,  0.0417, -0.1289],\n",
       "         ...,\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],\n",
       "         [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]],\n",
       "        dtype=torch.float64),\n",
       " tensor(0))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# for idx, (data) in enumerate(questions_dataset):\n",
    "#     print(idx)\n",
    "\n",
    "\n",
    "questions_dataset[12405]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size of the whole dataset:  49998\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([ 4084, 37399, 39989, ..., 18674, 24125, 19661])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#### Splitting the data\n",
    "\n",
    "# We have a PyTorch Subset object, what if we want to manage the indices for subsetting the original dataset?\n",
    "\n",
    "n_samples = len(questions_dataset)\n",
    "print(\"size of the whole dataset: \", n_samples)\n",
    "\n",
    "# Shuffle indices with np.random.permutation, also fix seed if you want reproducability\n",
    "\n",
    "shuffled_indices = np.random.permutation(n_samples)\n",
    "shuffled_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train samples:  39999\n",
      "val  samples:   9999\n"
     ]
    }
   ],
   "source": [
    "### ratio is 80% and 20%\n",
    "\n",
    "val_ratio = 0.2\n",
    "\n",
    "validationset_inds = shuffled_indices[:int(n_samples * val_ratio)]\n",
    "trainingset_inds = shuffled_indices[int(n_samples * val_ratio):]\n",
    "print(\"train samples: \", len(trainingset_inds))\n",
    "print(\"val  samples:  \", len(validationset_inds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(39999, 9999)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset = torch.utils.data.Subset(questions_dataset, indices=trainingset_inds)\n",
    "val_dataset = torch.utils.data.Subset(questions_dataset, indices=validationset_inds)\n",
    "len(train_dataset), len(val_dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(624, 156)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.save(join(\"./\", \"data\", \"questions\", \"train_idx.npy\"), trainingset_inds)\n",
    "np.save(join(\"./\", \"data\", \"questions\", \"val_idx.npy\"), validationset_inds)\n",
    "\n",
    "batch_size = 64\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n",
    "                                           batch_size=batch_size,\n",
    "                                           shuffle=True, sampler=None,\n",
    "                                           collate_fn=None,\n",
    "                                           drop_last=True\n",
    "                                           )\n",
    "\n",
    "val_loader = torch.utils.data.DataLoader(dataset=val_dataset,\n",
    "                                           batch_size=batch_size,\n",
    "                                           shuffle=False, sampler=None,\n",
    "                                           collate_fn=None,\n",
    "                                           drop_last=True\n",
    "                                           )\n",
    "\n",
    "# Let us check some properties of data loader object\n",
    "\n",
    "len(train_loader), len(val_loader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(39936, 39999)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# number of batches X batch_size = should give us the size of the dataset\n",
    "len(train_loader) * batch_size, len(train_dataset)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9984, 9999)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(val_loader) * batch_size, len(val_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batch_size:  64\n",
      "Sampler info:  <torch.utils.data.sampler.RandomSampler object at 0x000001E8CDF38FC8>\n",
      "Dataset is also there:\n",
      " <__main__.QuestionsDataset object at 0x000001E7E5EC5508>\n"
     ]
    }
   ],
   "source": [
    "print(\"Batch_size: \", train_loader.batch_size)\n",
    "print(\"Sampler info: \", train_loader.sampler)\n",
    "print(\"Dataset is also there:\\n\", train_loader.dataset.dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 2: Model (20 Points)\n",
    "\n",
    "1. Explore what possible models for the task could be. You do not need to come up with a very complex model, a relatively small model consisting of the following sequence would be okay: {*Embeddings for the input - LSTM or CNN to process the sequence - Linear Layers to learn features from the combined representation of the questions - Output Layer*} would also be fine, just take care of the sizing of the different layers. Please always check the sizing after each layer and make sure you understand the dimensions correclty and they map to what you have in mind.\n",
    "2. Build a model class.\n",
    "3. Test your model with one batch from your dataloader and check the input and output shapes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Your code goes here ####\n",
    "\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Dup_Qestions(nn.Module):\n",
    "    def __init__(self, input_size=300, batch_size=64):\n",
    "        super().__init__()\n",
    "        \n",
    "        self.input_size = input_size\n",
    "        self.batch_size = batch_size\n",
    "\n",
    "        # self.q1_block = nn.Sequential(\n",
    "        #                     torch.nn.Conv1d(11, 30, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     torch.nn.Conv1d(30, 20, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                      torch.nn.Conv1d(20, 10, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     torch.nn.Conv1d(10, 1, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     )\n",
    "        # self.q2_block = nn.Sequential(\n",
    "        #                     torch.nn.Conv1d(11, 30, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     torch.nn.Conv1d(30, 20, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                      torch.nn.Conv1d(20, 10, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     torch.nn.Conv1d(10, 1, 3),\n",
    "        #                     nn.ReLU(),\n",
    "        #                     )\n",
    "\n",
    "        # self.classification_block =   nn.Sequential(\n",
    "        #         nn.Linear(292,120),\n",
    "        #         nn.Tanh(), \n",
    "        #         nn.Linear(120,60),\n",
    "        #         nn.Tanh(),\n",
    "        #         nn.Linear(60,1),\n",
    "        #         nn.Sigmoid()\n",
    "        #     )\n",
    "\n",
    "        self.q1_block_lstm = nn.LSTM(input_size=300, hidden_size=300, batch_first=True)\n",
    "\n",
    "        self.q1_conv = nn.Sequential(\n",
    "            torch.nn.Conv1d(11, 1, 3),\n",
    "            nn.ReLU(),\n",
    "            )\n",
    "        self.q2_block_lstm =  nn.LSTM(input_size=300, hidden_size=300, batch_first=True)\n",
    "\n",
    "        self.q2_conv = nn.Sequential(\n",
    "            torch.nn.Conv1d(11, 1, 3),\n",
    "            nn.ReLU(),\n",
    "            )\n",
    "\n",
    "        self.classification_block_lstm =   nn.Sequential(\n",
    "                nn.Linear(input_size-2 ,120),\n",
    "                nn.Tanh(), \n",
    "                nn.Linear(120,60),\n",
    "                nn.Tanh(),\n",
    "                nn.Linear(60,1),\n",
    "                nn.Sigmoid()\n",
    "            )\n",
    "        \n",
    "\n",
    "        # self.hidden2class = nn.Linear(298, 1)\n",
    "        # self.sig = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, q1, q2):\n",
    "\n",
    "\n",
    "        print(self.batch_size)\n",
    "        q1_lstm = self.q1_block_lstm(q1)\n",
    "        q2_lstm = self.q2_block_lstm(q2)\n",
    "\n",
    "\n",
    "        q1_lstm_conv = self.q1_conv(q1_lstm[0])\n",
    "        q2_lstm_conv = self.q2_conv(q2_lstm[0])\n",
    "\n",
    "        # print(\"Lstm output size: \", q1_lstm_conv.shape)\n",
    "        \n",
    "        mult = q1_lstm_conv * q2_lstm_conv\n",
    "        # print(\"Mult size : \", mult.shape)\n",
    "        out_sig = self.classification_block_lstm(mult.view(self.batch_size,-1))\n",
    "\n",
    "        # ---------------------antother model --------------------------------\n",
    "        # q1_conv = self.q1_block(q1)\n",
    "        # q2_conv = self.q2_block(q2)\n",
    "        # # print(\"q1_conv size:\",q1_conv.shape)\n",
    "        # # print(\"q2_conv size:\",q2_conv.shape)\n",
    "\n",
    "        # # sim = F.cosine_similarity(q1_conv.view(batch_size, q1_conv.shape[2]), q2_conv.view(batch_size, q2_conv.shape[2]))\n",
    "        # # print(\"sim size:\", sim.shape)\n",
    "\n",
    "        # out_sig = self.classification_block(q1_conv.view(batch_size,-1) * q2_conv.view(batch_size,-1))\n",
    "\n",
    "        \n",
    "\n",
    "        # out = self.hidden2class(q1_conv * q2_conv)\n",
    "        # print(\"out size: \",out.shape)\n",
    "        # out_sig = self.sig(out)\n",
    "        # print(\"out_sig size: \",out_sig.shape)\n",
    "\n",
    "        return out_sig\n",
    "        # q1_conv, q2_conv, sim\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 3: Training (30 Points)\n",
    "\n",
    "1. Develop Training and Validation code.\n",
    "2. Choose a suitable loss function.\n",
    "3. Refactor the code so that it can be easily modified and adapted (use methods, classes, etc...)\n",
    "4. Make sure to save your trained model when you reach a good score on the validation dataset.\n",
    "5. Plot the training and validation losses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Your code goes here ####\n",
    "\n",
    "loss_func = torch.nn.BCELoss()\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CUDA\n",
      "q1_block_lstm.weight_ih_l0 True\n",
      "q1_block_lstm.weight_hh_l0 True\n",
      "q1_block_lstm.bias_ih_l0 True\n",
      "q1_block_lstm.bias_hh_l0 True\n",
      "q1_conv.0.weight True\n",
      "q1_conv.0.bias True\n",
      "q2_block_lstm.weight_ih_l0 True\n",
      "q2_block_lstm.weight_hh_l0 True\n",
      "q2_block_lstm.bias_ih_l0 True\n",
      "q2_block_lstm.bias_hh_l0 True\n",
      "q2_conv.0.weight True\n",
      "q2_conv.0.bias True\n",
      "classification_block_lstm.0.weight True\n",
      "classification_block_lstm.0.bias True\n",
      "classification_block_lstm.2.weight True\n",
      "classification_block_lstm.2.bias True\n",
      "classification_block_lstm.4.weight True\n",
      "classification_block_lstm.4.bias True\n"
     ]
    }
   ],
   "source": [
    "model = Dup_Qestions(batch_size=batch_size)\n",
    "model\n",
    "\n",
    "if torch.cuda.is_available():\n",
    "    model.to(\"cuda\")\n",
    "    print(\"CUDA\")\n",
    "\n",
    "for name, data in model.named_parameters():\n",
    "    print(name, data.requires_grad)\n",
    "\n",
    "# optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)\n",
    "# optimizer\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss:  0\n"
     ]
    }
   ],
   "source": [
    "total_loss = 0\n",
    "\n",
    "\n",
    "for batch in train_loader:\n",
    "    q1, q2, label = batch\n",
    "    if torch.cuda.is_available():\n",
    "        q1 = q1.to(\"cuda\")\n",
    "        q2 = q2.to(\"cuda\")\n",
    "\n",
    "    q1 = q1.float()\n",
    "    q2 = q2.float()\n",
    "    label = label.float()\n",
    "    \n",
    "    # _, _, score = model(q1, q2)\n",
    "    score = model(q1, q2)\n",
    "\n",
    "    score = score.cpu()\n",
    "\n",
    "    # print('out1', out1.dtype)\n",
    "    # print('target vector', y.dtype)\n",
    "\n",
    "    # loss_function = nn.CrossEntropyLoss()\n",
    "    # loss = loss_function(output, Variable(train_labels))\n",
    "    # criterion = nn.CosineEmbeddingLoss(margin=0, size_average=True, reduce=False)\n",
    "    # loss = criterion(out1, out2, (2 * y - 1))  # cast y to {1, -1} and float type\n",
    "    # criterion = ContrastiveLoss()\n",
    "    # loss = criterion(y, sim)\n",
    "\n",
    "    # loss = F.cross_entropy(sim, y)\n",
    "    # print(score.shape, label.shape)\n",
    "    # loss = F.binary_cross_entropy_with_logits(score, label)\n",
    "    loss = loss_func(score.view(batch_size,1), label.view(batch_size,1))\n",
    "\n",
    "\n",
    "            \n",
    "    break\n",
    "    \n",
    "print('loss: ', total_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Finishing last run (ID:1xosdijl) before initializing another..."
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<br/>Waiting for W&B process to finish, PID 15032... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "78927a92d0dd48bf9e1690a3c73885a0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value=' 0.00MB of 0.00MB uploaded (0.00MB deduped)\\r'), FloatProgress(value=1.0, max=1.0)…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>\n",
       "    table.wandb td:nth-child(1) { padding: 0 10px; text-align: right }\n",
       "    .wandb-row { display: flex; flex-direction: row; flex-wrap: wrap; width: 100% }\n",
       "    .wandb-col { display: flex; flex-direction: column; flex-basis: 100%; flex: 1; padding: 10px; }\n",
       "    </style>\n",
       "<div class=\"wandb-row\"><div class=\"wandb-col\">\n",
       "<h3>Run history:</h3><br/><table class=\"wandb\"><tr><td>Training acc</td><td>▄▄▅▂▄▂▄▅▄▂▅▄█▄▂▅▃▅▄▄▂▅▆▃▂▂▂▅▃▅▃▆▇▄▁▆</td></tr><tr><td>Training loss</td><td>▄▄▅▃▃▃▄▅▃▆▂▅█▃▃▄▃▅▃▄▆▂▆▃▂▃▅▃▄▃▃▁▆▄▂▆</td></tr><tr><td>Validation acc</td><td>▁▁▁▁█▁▁▁███▁▁█▁▁▁▁█▁██▁▁▁▁████▁█▁▁▁▁</td></tr><tr><td>Validation loss</td><td>▂▁▁▁▄▁▁▂▇▄▃▂▂▃▂▁▂▂▃▂█▃▂▁▂▂▃▄▃▃▁▅▁▁▁▁</td></tr></table><br/></div><div class=\"wandb-col\">\n",
       "<h3>Run summary:</h3><br/><table class=\"wandb\"><tr><td>Training acc</td><td>0.60938</td></tr><tr><td>Training loss</td><td>0.70123</td></tr><tr><td>Validation acc</td><td>0.48868</td></tr><tr><td>Validation loss</td><td>0.01081</td></tr></table>\n",
       "</div></div>\n",
       "Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)\n",
       "<br/>Synced <strong style=\"color:#cdcd00\">neat-fire-14</strong>: <a href=\"https://wandb.ai/osobky/dup-questions-project/runs/1xosdijl\" target=\"_blank\">https://wandb.ai/osobky/dup-questions-project/runs/1xosdijl</a><br/>\n",
       "Find logs at: <code>.\\wandb\\run-20211108_172018-1xosdijl\\logs</code><br/>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Successfully finished last run (ID:1xosdijl). Initializing new run:<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "                    Syncing run <strong><a href=\"https://wandb.ai/osobky/dup-questions-project/runs/2ur13rfn\" target=\"_blank\">dandy-puddle-15</a></strong> to <a href=\"https://wandb.ai/osobky/dup-questions-project\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://docs.wandb.com/integrations/jupyter.html\" target=\"_blank\">docs</a>).<br/>\n",
       "\n",
       "                "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import wandb\n",
    "\n",
    "wandb.init(project=\"dup-questions-project\", entity=\"osobky\")\n",
    "\n",
    "wandb.config = {'lr': 0.001, \n",
    "        'optimizer': 'Adam', \n",
    "        \"cuda\": True, \n",
    "        'epochs': 100,\n",
    "        'batch_size': batch_size,\n",
    "        'log_interval': 100,\n",
    "        'test_interval': 100,\n",
    "        'save_interval' : 1000,\n",
    "        'save_dir': './models/'\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn.functional as F\n",
    "import torch.nn as nn\n",
    "\n",
    "#This cell is adopted from https://github.com/Lapis-Hong/atec-nlp/blob/master/pytorch/train.py \n",
    "def metrics(y, y_pred):\n",
    "    # print(y, y_pred)\n",
    "    TP = ((y_pred == 1) & (y == 1)).sum().float()\n",
    "    TN = ((y_pred == 0) & (y == 0)).sum().float()\n",
    "    FN = ((y_pred == 0) & (y == 1)).sum().float()\n",
    "    FP = ((y_pred == 1) & (y == 0)).sum().float()\n",
    "    p = TP / (TP + FP).clamp(min=1e-8)\n",
    "    r = TP / (TP + FN).clamp(min=1e-8)\n",
    "    F1 = 2 * r * p / (r + p).clamp(min=1e-8)\n",
    "    acc = (TP + TN) / (TP + TN + FP + FN).clamp(min=1e-8)\n",
    "    return acc, p, r, F1\n",
    "\n",
    "\n",
    "\n",
    "def train(train_iter, dev_iter, model, args):\n",
    "    if args['cuda']:\n",
    "        model.cuda()\n",
    "    # for param in model.parameters():\n",
    "    #     print(param)\n",
    "\n",
    "    def adjust_learning_rate(optimizer, learning_rate, epoch):\n",
    "        lr = learning_rate * (0.1 ** (epoch // 10))\n",
    "        for param_group in optimizer.param_groups:\n",
    "            param_group['lr'] = lr\n",
    "        return optimizer\n",
    "\n",
    "    if args['optimizer'] == 'Adam':\n",
    "        optimizer = torch.optim.Adam(model.parameters(), lr=args['lr'], betas=[0.9, 0.999], eps=1e-8, weight_decay=0)\n",
    "    elif args['optimizer'] == 'SGD':\n",
    "        optimizer = torch.optim.SGD(model.parameters(), lr=args['lr'], momentum=0.9, weight_decay=0.01)\n",
    "    else:\n",
    "        raise Exception('For other optimizers, please add it yourself. supported ones are: SGD and Adam.')\n",
    "\n",
    "    F1_best = 0.0\n",
    "    last_improved_step = 0\n",
    "    model.train()\n",
    "    steps = 0\n",
    "    for epoch in range(1, args['epochs']+1):\n",
    "        for batch in train_iter:\n",
    "            optimizer = adjust_learning_rate(optimizer, args['lr'], epoch)\n",
    "            q1, q2, label = batch\n",
    "            if torch.cuda.is_available():\n",
    "                q1 = q1.to(\"cuda\")\n",
    "                q2 = q2.to(\"cuda\")\n",
    "\n",
    "            q1 = q1.float()\n",
    "            q2 = q2.float()\n",
    "            label = label.float()\n",
    "            \n",
    "            optimizer.zero_grad()\n",
    "            # _, _, score = model(q1, q2)\n",
    "            score = model(q1, q2)\n",
    "\n",
    "            score = score.cpu()\n",
    "\n",
    "            # print('out1', out1.dtype)\n",
    "            # print('target vector', y.dtype)\n",
    "\n",
    "            # loss_function = nn.CrossEntropyLoss()\n",
    "            # loss = loss_function(output, Variable(train_labels))\n",
    "            # criterion = nn.CosineEmbeddingLoss(margin=0, size_average=True, reduce=False)\n",
    "            # loss = criterion(out1, out2, (2 * y - 1))  # cast y to {1, -1} and float type\n",
    "            # criterion = ContrastiveLoss()\n",
    "            # loss = criterion(y, sim)\n",
    "\n",
    "            # loss = F.cross_entropy(sim, y)\n",
    "            # print(score.shape, label.shape)\n",
    "            # loss = F.binary_cross_entropy_with_logits(score, label)\n",
    "            loss = loss_func(score.view(args['batch_size'],1), label.view(args['batch_size'],1))\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "            steps += 1\n",
    "\n",
    "            if steps % args['log_interval'] == 0:\n",
    "                # _, pred = torch.max(sim.data, 1)\n",
    "                print('model sim and label tuples:')\n",
    "                for i, j in zip(score, label):\n",
    "                    print(i.item(), j.item())\n",
    "\n",
    "                pred = score.data >= 0.48\n",
    "                acc, p, r, f1 = metrics(label, torch.flatten(pred))\n",
    "                print('TRAIN[steps={}] loss={:.6f} acc={:.3f} P={:.3f} R={:.3f} F1={:.6f}'.format(\n",
    "                    steps, loss.item(), acc, p, r, f1))\n",
    "                wandb.log({\"Training loss\": loss.item()})\n",
    "                wandb.log({\"Training acc\": acc})\n",
    "\n",
    "            if steps % args['test_interval'] == 0:\n",
    "                loss, acc, p, r, f1 = eval(dev_iter, model)\n",
    "\n",
    "                if f1 > F1_best:\n",
    "                    F1_best = f1\n",
    "                    last_improved_step = steps\n",
    "                    if F1_best > 0.5:\n",
    "                        save_prefix = os.path.join(args['save_dir'], 'snapshot')\n",
    "                        save_path = '{}_steps{}.pt'.format(save_prefix, steps)\n",
    "                        torch.save(model, save_path)\n",
    "                    improved_token = '*'\n",
    "                else:\n",
    "                    improved_token = ''\n",
    "                print('DEV[steps={}] loss={:.6f} acc={:.3f} P={:.3f} R={:.3f} F1={:.6f} {}'.format(\n",
    "                    steps, loss, acc, p, r, f1, improved_token))\n",
    "                wandb.log({\"Validation loss\": loss})\n",
    "                wandb.log({\"Validation acc\": acc})\n",
    "\n",
    "            if steps % args['save_interval'] == 0:\n",
    "                if not os.path.isdir(args['save_dir']):\n",
    "                    os.makedirs(args['save_dir'])\n",
    "                save_prefix = os.path.join(args['save_dir'], 'snapshot')\n",
    "                save_path = '{}_steps{}.pt'.format(save_prefix, steps)\n",
    "                torch.save(model, save_path)\n",
    "\n",
    "            if steps - last_improved_step > 5000:  # 5000 steps\n",
    "                print(\"No improvement for a long time, early-stopping at best F1={}\".format(F1_best))\n",
    "                break\n",
    "        print(\"EPOCH: \", epoch)\n",
    "\n",
    "\n",
    "def eval(data_iter, model):\n",
    "    loss_tot, y_list, y_pred_list = 0, [], []\n",
    "    model.eval()\n",
    "    for q1, q2, y in data_iter:\n",
    "        if torch.cuda.is_available():\n",
    "                q1 = q1.to(\"cuda\")\n",
    "                q2 = q2.to(\"cuda\")\n",
    "        q1 = q1.float()\n",
    "        q2 = q2.float()\n",
    "        y = y.float()\n",
    "        score = model(q1, q2).cpu()\n",
    "        # loss = F.cross_entropy(output, y, size_average=False)\n",
    "        loss = loss_func(score.view(args['batch_size'],1), y.view(args['batch_size'],1))\n",
    "        loss_tot += loss.item()  # 0-dim scaler\n",
    "        y_pred = score.data >= 0.48\n",
    "        y_pred_list.append(y_pred)\n",
    "        y_list.append(y)\n",
    "    y_pred = torch.cat(y_pred_list, 0)\n",
    "    y = torch.cat(y_list, 0)\n",
    "    # print(y, torch.flatten(y_pred))\n",
    "    acc, p, r, f1 = metrics(y, torch.flatten(y_pred))\n",
    "    size = len(data_iter.dataset)\n",
    "    loss_avg = loss_tot / float(size)\n",
    "    model.train()\n",
    "    return loss_avg, acc, p, r, f1\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model sim and label tuples:\n",
      "0.4299338459968567 1.0\n",
      "0.5457780361175537 1.0\n",
      "0.39723512530326843 0.0\n",
      "0.45691755414009094 0.0\n",
      "0.45537635684013367 1.0\n",
      "0.5891271233558655 1.0\n",
      "0.3987533748149872 1.0\n",
      "0.5676745772361755 1.0\n",
      "0.3626561462879181 0.0\n",
      "0.37897562980651855 1.0\n",
      "0.4063228964805603 0.0\n",
      "0.5292271375656128 0.0\n",
      "0.3911306858062744 1.0\n",
      "0.6227343678474426 1.0\n",
      "0.3672198951244354 0.0\n",
      "0.48999348282814026 0.0\n",
      "0.3940660357475281 1.0\n",
      "0.3703342080116272 0.0\n",
      "0.37639734148979187 0.0\n",
      "0.5727044939994812 1.0\n",
      "0.5457738041877747 0.0\n",
      "0.49027734994888306 1.0\n",
      "0.4702213704586029 0.0\n",
      "0.39535605907440186 0.0\n",
      "0.56715989112854 1.0\n",
      "0.4842568337917328 1.0\n",
      "0.49808287620544434 1.0\n",
      "0.32570090889930725 0.0\n",
      "0.41008004546165466 1.0\n",
      "0.4297690987586975 0.0\n",
      "0.5320093631744385 0.0\n",
      "0.369868665933609 0.0\n",
      "0.5104302763938904 1.0\n",
      "0.42727282643318176 1.0\n",
      "0.3876344859600067 0.0\n",
      "0.4758448600769043 0.0\n",
      "0.5172259211540222 0.0\n",
      "0.41260743141174316 0.0\n",
      "0.5131067037582397 0.0\n",
      "0.5156345367431641 0.0\n",
      "0.5711231827735901 0.0\n",
      "0.5275759100914001 1.0\n",
      "0.5402612090110779 0.0\n",
      "0.6180413365364075 1.0\n",
      "0.3851938247680664 0.0\n",
      "0.4510944187641144 0.0\n",
      "0.4875487685203552 0.0\n",
      "0.5734238624572754 1.0\n",
      "0.45289137959480286 0.0\n",
      "0.5112383365631104 0.0\n",
      "0.4981752634048462 0.0\n",
      "0.577683687210083 0.0\n",
      "0.587846040725708 1.0\n",
      "0.36682769656181335 0.0\n",
      "0.4592238962650299 1.0\n",
      "0.4578479528427124 1.0\n",
      "0.41551122069358826 1.0\n",
      "0.6393276453018188 1.0\n",
      "0.5561888217926025 1.0\n",
      "0.5167391896247864 0.0\n",
      "0.41940006613731384 1.0\n",
      "0.5810560584068298 0.0\n",
      "0.5253333449363708 1.0\n",
      "0.48207443952560425 1.0\n",
      "TRAIN[steps=100] loss=0.661209 acc=0.578 P=0.545 R=0.600 F1=0.571429\n",
      "DEV[steps=100] loss=0.010336 acc=0.613 P=0.603 R=0.612 F1=0.607422 *\n",
      "model sim and label tuples:\n",
      "0.35936060547828674 0.0\n",
      "0.5757537484169006 1.0\n",
      "0.2870752215385437 1.0\n",
      "0.4108469784259796 0.0\n",
      "0.4558050036430359 0.0\n",
      "0.44997483491897583 1.0\n",
      "0.28106755018234253 0.0\n",
      "0.5655273795127869 1.0\n",
      "0.4904221296310425 0.0\n",
      "0.2912171483039856 0.0\n",
      "0.4638574719429016 1.0\n",
      "0.5252302289009094 1.0\n",
      "0.38847997784614563 0.0\n",
      "0.3057791590690613 0.0\n",
      "0.5817369818687439 0.0\n",
      "0.36965513229370117 0.0\n",
      "0.3502585291862488 1.0\n",
      "0.40326303243637085 0.0\n",
      "0.4011520743370056 0.0\n",
      "0.408992737531662 0.0\n",
      "0.45077115297317505 0.0\n",
      "0.4870573878288269 1.0\n",
      "0.47922512888908386 0.0\n",
      "0.37924739718437195 1.0\n",
      "0.42487961053848267 0.0\n",
      "0.4623377323150635 0.0\n",
      "0.3297704756259918 0.0\n",
      "0.30300551652908325 0.0\n",
      "0.20419248938560486 0.0\n",
      "0.5364816188812256 1.0\n",
      "0.488302081823349 1.0\n",
      "0.47429800033569336 0.0\n",
      "0.38205787539482117 0.0\n",
      "0.5501166582107544 1.0\n",
      "0.4576161801815033 1.0\n",
      "0.39510875940322876 0.0\n",
      "0.40021049976348877 1.0\n",
      "0.2315400242805481 0.0\n",
      "0.3021984398365021 0.0\n",
      "0.4127005636692047 0.0\n",
      "0.44407758116722107 1.0\n",
      "0.3609590530395508 0.0\n",
      "0.3255777955055237 0.0\n",
      "0.49313101172447205 1.0\n",
      "0.4428941309452057 0.0\n",
      "0.33726125955581665 0.0\n",
      "0.3574744760990143 0.0\n",
      "0.2298380732536316 0.0\n",
      "0.31314557790756226 0.0\n",
      "0.4011785387992859 0.0\n",
      "0.2743651568889618 0.0\n",
      "0.40293699502944946 1.0\n",
      "0.48232290148735046 1.0\n",
      "0.5026041865348816 1.0\n",
      "0.44115692377090454 0.0\n",
      "0.3722105622291565 0.0\n",
      "0.3714308440685272 0.0\n",
      "0.554526150226593 1.0\n",
      "0.353758305311203 0.0\n",
      "0.5166456699371338 0.0\n",
      "0.10012704879045486 0.0\n",
      "0.34461525082588196 1.0\n",
      "0.20620842278003693 0.0\n",
      "0.48930832743644714 1.0\n",
      "TRAIN[steps=200] loss=0.575060 acc=0.797 P=0.800 R=0.545 F1=0.648649\n",
      "DEV[steps=200] loss=0.010345 acc=0.611 P=0.708 R=0.347 F1=0.466043 \n",
      "model sim and label tuples:\n",
      "0.7130900621414185 0.0\n",
      "0.40274113416671753 1.0\n",
      "0.4023307263851166 1.0\n",
      "0.5061100125312805 0.0\n",
      "0.43127918243408203 0.0\n",
      "0.4390014410018921 0.0\n",
      "0.5445087552070618 0.0\n",
      "0.46075788140296936 1.0\n",
      "0.21847878396511078 0.0\n",
      "0.4376390874385834 0.0\n",
      "0.39646291732788086 0.0\n",
      "0.6823384761810303 1.0\n",
      "0.6982545852661133 1.0\n",
      "0.07983672618865967 0.0\n",
      "0.6971989274024963 1.0\n",
      "0.6295241117477417 0.0\n",
      "0.43727192282676697 1.0\n",
      "0.29707226157188416 0.0\n",
      "0.22144800424575806 1.0\n",
      "0.6720237731933594 1.0\n",
      "0.4108169674873352 0.0\n",
      "0.7634243369102478 1.0\n",
      "0.29695603251457214 0.0\n",
      "0.19521267712116241 1.0\n",
      "0.548567533493042 0.0\n",
      "0.14208748936653137 1.0\n",
      "0.6145640015602112 1.0\n",
      "0.3024027347564697 0.0\n",
      "0.772175133228302 1.0\n",
      "0.2490818351507187 1.0\n",
      "0.4719815254211426 0.0\n",
      "0.6267508864402771 0.0\n",
      "0.47712188959121704 1.0\n",
      "0.4451477825641632 1.0\n",
      "0.26211878657341003 0.0\n",
      "0.20976121723651886 0.0\n",
      "0.43634387850761414 0.0\n",
      "0.5497531294822693 0.0\n",
      "0.6311284303665161 0.0\n",
      "0.39776191115379333 0.0\n",
      "0.583296537399292 1.0\n",
      "0.49871334433555603 0.0\n",
      "0.7345319986343384 1.0\n",
      "0.659895658493042 0.0\n",
      "0.6610457301139832 1.0\n",
      "0.7357480525970459 1.0\n",
      "0.21190297603607178 0.0\n",
      "0.3989398777484894 0.0\n",
      "0.22305504977703094 1.0\n",
      "0.44070541858673096 1.0\n",
      "0.662605881690979 1.0\n",
      "0.38563305139541626 1.0\n",
      "0.05591903626918793 0.0\n",
      "0.7485567927360535 1.0\n",
      "0.7783803939819336 1.0\n",
      "0.08449622243642807 1.0\n",
      "0.715196967124939 1.0\n",
      "0.8433127403259277 0.0\n",
      "0.3456965386867523 0.0\n",
      "0.7830453515052795 1.0\n",
      "0.28468576073646545 0.0\n",
      "0.6211090683937073 0.0\n",
      "0.6567996144294739 1.0\n",
      "0.5889347195625305 0.0\n",
      "TRAIN[steps=300] loss=0.687394 acc=0.578 P=0.567 R=0.548 F1=0.557377\n",
      "DEV[steps=300] loss=0.010026 acc=0.629 P=0.614 R=0.653 F1=0.632572 *\n",
      "model sim and label tuples:\n",
      "0.14355604350566864 1.0\n",
      "0.33386602997779846 0.0\n",
      "0.3841593861579895 1.0\n",
      "0.847629725933075 1.0\n",
      "0.7333893179893494 1.0\n",
      "0.4513079524040222 1.0\n",
      "0.2691759467124939 1.0\n",
      "0.660029947757721 1.0\n",
      "0.5964841246604919 1.0\n",
      "0.57135009765625 0.0\n",
      "0.1798933893442154 1.0\n",
      "0.6125960350036621 0.0\n",
      "0.4725111126899719 0.0\n",
      "0.421174556016922 0.0\n",
      "0.24855226278305054 0.0\n",
      "0.5600417256355286 0.0\n",
      "0.5715194344520569 0.0\n",
      "0.2663302421569824 0.0\n",
      "0.7245171666145325 1.0\n",
      "0.42763203382492065 0.0\n",
      "0.2996503412723541 1.0\n",
      "0.4633086025714874 1.0\n",
      "0.3138878643512726 0.0\n",
      "0.23564331233501434 1.0\n",
      "0.2713313400745392 1.0\n",
      "0.6060458421707153 1.0\n",
      "0.24275435507297516 0.0\n",
      "0.19695822894573212 1.0\n",
      "0.2663414478302002 1.0\n",
      "0.48223620653152466 1.0\n",
      "0.34598255157470703 0.0\n",
      "0.26792532205581665 0.0\n",
      "0.156426340341568 1.0\n",
      "0.3180236518383026 0.0\n",
      "0.39542385935783386 0.0\n",
      "0.500775158405304 0.0\n",
      "0.2734144926071167 0.0\n",
      "0.2115795612335205 0.0\n",
      "0.6498031616210938 1.0\n",
      "0.5454103350639343 1.0\n",
      "0.6643480658531189 1.0\n",
      "0.5154097676277161 0.0\n",
      "0.7433211803436279 1.0\n",
      "0.5713045597076416 0.0\n",
      "0.2772296667098999 0.0\n",
      "0.4636312425136566 0.0\n",
      "0.2838880121707916 0.0\n",
      "0.7965824007987976 1.0\n",
      "0.2625425457954407 0.0\n",
      "0.34802624583244324 0.0\n",
      "0.41850098967552185 0.0\n",
      "0.6111477613449097 0.0\n",
      "0.3024488687515259 1.0\n",
      "0.48077598214149475 0.0\n",
      "0.2618667781352997 1.0\n",
      "0.483446329832077 1.0\n",
      "0.3533141613006592 1.0\n",
      "0.8484939932823181 1.0\n",
      "0.6041650176048279 1.0\n",
      "0.11011282354593277 0.0\n",
      "0.1873747706413269 1.0\n",
      "0.6557275652885437 1.0\n",
      "0.8430510759353638 0.0\n",
      "0.8128423690795898 1.0\n",
      "TRAIN[steps=400] loss=0.717011 acc=0.594 P=0.630 R=0.515 F1=0.566667\n",
      "DEV[steps=400] loss=0.010054 acc=0.640 P=0.701 R=0.460 F1=0.555597 \n",
      "model sim and label tuples:\n",
      "0.3119870126247406 1.0\n",
      "0.31366902589797974 1.0\n",
      "0.16755205392837524 0.0\n",
      "0.38631245493888855 1.0\n",
      "0.4391418993473053 1.0\n",
      "0.42128899693489075 0.0\n",
      "0.3709948658943176 1.0\n",
      "0.6217420697212219 1.0\n",
      "0.3163297474384308 1.0\n",
      "0.404364675283432 0.0\n",
      "0.5401723980903625 1.0\n",
      "0.28719085454940796 0.0\n",
      "0.5166126489639282 1.0\n",
      "0.40096741914749146 1.0\n",
      "0.36210328340530396 0.0\n",
      "0.39492323994636536 1.0\n",
      "0.4928734302520752 1.0\n",
      "0.32034003734588623 0.0\n",
      "0.3401883840560913 0.0\n",
      "0.5565025210380554 1.0\n",
      "0.356469064950943 0.0\n",
      "0.42403456568717957 0.0\n",
      "0.3624823987483978 0.0\n",
      "0.3611462414264679 0.0\n",
      "0.39596566557884216 1.0\n",
      "0.44013792276382446 0.0\n",
      "0.37522247433662415 1.0\n",
      "0.6384780406951904 1.0\n",
      "0.38929152488708496 0.0\n",
      "0.7733505964279175 1.0\n",
      "0.5914579033851624 0.0\n",
      "0.33939680457115173 0.0\n",
      "0.30787932872772217 0.0\n",
      "0.32047829031944275 0.0\n",
      "0.5780266523361206 1.0\n",
      "0.5359905958175659 1.0\n",
      "0.3880750834941864 0.0\n",
      "0.3964354693889618 1.0\n",
      "0.5697461366653442 1.0\n",
      "0.47839802503585815 0.0\n",
      "0.3359140455722809 1.0\n",
      "0.3119567334651947 0.0\n",
      "0.6841000318527222 1.0\n",
      "0.30435702204704285 0.0\n",
      "0.6276829838752747 1.0\n",
      "0.3193448483943939 0.0\n",
      "0.678861141204834 1.0\n",
      "0.3071804940700531 0.0\n",
      "0.5538231730461121 0.0\n",
      "0.6976661086082458 1.0\n",
      "0.4404923617839813 1.0\n",
      "0.45559078454971313 1.0\n",
      "0.3303133249282837 0.0\n",
      "0.370818555355072 1.0\n",
      "0.2509666085243225 0.0\n",
      "0.3689589500427246 1.0\n",
      "0.6127781271934509 1.0\n",
      "0.34820106625556946 0.0\n",
      "0.5132655501365662 1.0\n",
      "0.47244635224342346 1.0\n",
      "0.35937193036079407 1.0\n",
      "0.2250537872314453 0.0\n",
      "0.5597392916679382 1.0\n",
      "0.3341583013534546 0.0\n",
      "TRAIN[steps=500] loss=0.613649 acc=0.688 P=0.895 R=0.486 F1=0.629630\n",
      "DEV[steps=500] loss=0.009945 acc=0.642 P=0.700 R=0.468 F1=0.560973 \n",
      "model sim and label tuples:\n",
      "0.5388504266738892 1.0\n",
      "0.7118787169456482 0.0\n",
      "0.44655361771583557 0.0\n",
      "0.28515055775642395 0.0\n",
      "0.8027316331863403 1.0\n",
      "0.3079185485839844 1.0\n",
      "0.16148115694522858 0.0\n",
      "0.3111608028411865 1.0\n",
      "0.8451018929481506 1.0\n",
      "0.650377631187439 1.0\n",
      "0.35757574439048767 0.0\n",
      "0.6180520057678223 0.0\n",
      "0.250962495803833 0.0\n",
      "0.730598509311676 1.0\n",
      "0.6497126817703247 1.0\n",
      "0.3603019118309021 0.0\n",
      "0.6022894382476807 1.0\n",
      "0.6053348183631897 1.0\n",
      "0.3639337122440338 1.0\n",
      "0.27931421995162964 0.0\n",
      "0.29417064785957336 0.0\n",
      "0.1784132421016693 0.0\n",
      "0.7335173487663269 1.0\n",
      "0.35228294134140015 0.0\n",
      "0.4387977421283722 1.0\n",
      "0.504153311252594 1.0\n",
      "0.4400399327278137 1.0\n",
      "0.22658048570156097 0.0\n",
      "0.3006507158279419 1.0\n",
      "0.31441742181777954 1.0\n",
      "0.3054249584674835 0.0\n",
      "0.6342729330062866 1.0\n",
      "0.18318988382816315 1.0\n",
      "0.31091395020484924 0.0\n",
      "0.5274627804756165 1.0\n",
      "0.7130962014198303 1.0\n",
      "0.32605159282684326 0.0\n",
      "0.1624211072921753 0.0\n",
      "0.6102172136306763 1.0\n",
      "0.3667369484901428 0.0\n",
      "0.27134719491004944 0.0\n",
      "0.27149903774261475 1.0\n",
      "0.22036553919315338 0.0\n",
      "0.23947261273860931 0.0\n",
      "0.29674994945526123 1.0\n",
      "0.48678532242774963 1.0\n",
      "0.602583110332489 1.0\n",
      "0.23963786661624908 1.0\n",
      "0.6699908971786499 0.0\n",
      "0.1625511199235916 0.0\n",
      "0.20816946029663086 0.0\n",
      "0.25655898451805115 0.0\n",
      "0.5548287630081177 1.0\n",
      "0.30035486817359924 1.0\n",
      "0.23379360139369965 0.0\n",
      "0.40142789483070374 0.0\n",
      "0.6954903602600098 1.0\n",
      "0.4633680582046509 1.0\n",
      "0.2575201988220215 0.0\n",
      "0.3072078824043274 1.0\n",
      "0.6072455644607544 1.0\n",
      "0.5381693840026855 0.0\n",
      "0.22424748539924622 0.0\n",
      "0.4118282198905945 0.0\n",
      "TRAIN[steps=600] loss=0.594589 acc=0.719 P=0.826 R=0.576 F1=0.678571\n",
      "DEV[steps=600] loss=0.009852 acc=0.648 P=0.698 R=0.494 F1=0.578594 \n",
      "EPOCH:  1\n",
      "model sim and label tuples:\n",
      "0.6654760241508484 1.0\n",
      "0.42183247208595276 1.0\n",
      "0.665288507938385 1.0\n",
      "0.1529427170753479 0.0\n",
      "0.885556161403656 1.0\n",
      "0.48608094453811646 1.0\n",
      "0.484089732170105 0.0\n",
      "0.3813648521900177 1.0\n",
      "0.2793537676334381 1.0\n",
      "0.7911695837974548 0.0\n",
      "0.47527340054512024 0.0\n",
      "0.525658905506134 0.0\n",
      "0.47451624274253845 0.0\n",
      "0.40538936853408813 0.0\n",
      "0.40701115131378174 1.0\n",
      "0.5888782143592834 0.0\n",
      "0.6211819648742676 1.0\n",
      "0.6841747164726257 1.0\n",
      "0.6540190577507019 1.0\n",
      "0.5327994227409363 0.0\n",
      "0.8528022170066833 0.0\n",
      "0.4660179316997528 0.0\n",
      "0.44127416610717773 1.0\n",
      "0.6816681027412415 1.0\n",
      "0.5632511973381042 0.0\n",
      "0.2488495260477066 0.0\n",
      "0.48660969734191895 0.0\n",
      "0.8802086710929871 1.0\n",
      "0.7101516723632812 1.0\n",
      "0.49379539489746094 0.0\n",
      "0.7437518239021301 1.0\n",
      "0.10380716621875763 0.0\n",
      "0.42312300205230713 0.0\n",
      "0.7979143261909485 1.0\n",
      "0.5757881999015808 0.0\n",
      "0.5020763874053955 0.0\n",
      "0.6168872714042664 0.0\n",
      "0.5809458494186401 1.0\n",
      "0.1524682343006134 0.0\n",
      "0.573835015296936 1.0\n",
      "0.7464334964752197 1.0\n",
      "0.28961828351020813 0.0\n",
      "0.522466242313385 0.0\n",
      "0.7237609624862671 1.0\n",
      "0.7136753797531128 1.0\n",
      "0.6359386444091797 0.0\n",
      "0.5384411215782166 1.0\n",
      "0.5184130072593689 0.0\n",
      "0.6172285079956055 0.0\n",
      "0.3671984076499939 0.0\n",
      "0.41014841198921204 0.0\n",
      "0.5087029933929443 1.0\n",
      "0.3018840253353119 0.0\n",
      "0.4729398488998413 0.0\n",
      "0.5719654560089111 1.0\n",
      "0.5081271529197693 0.0\n",
      "0.5077934861183167 1.0\n",
      "0.14993758499622345 0.0\n",
      "0.32980406284332275 1.0\n",
      "0.6827635765075684 1.0\n",
      "0.12116867303848267 0.0\n",
      "0.6380971074104309 0.0\n",
      "0.24630701541900635 0.0\n",
      "0.4499891698360443 0.0\n",
      "TRAIN[steps=700] loss=0.606480 acc=0.625 P=0.550 R=0.786 F1=0.647059\n",
      "DEV[steps=700] loss=0.009732 acc=0.638 P=0.621 R=0.667 F1=0.643012 *\n",
      "model sim and label tuples:\n",
      "0.6217465400695801 1.0\n",
      "0.3556358218193054 1.0\n",
      "0.45254355669021606 0.0\n",
      "0.44091707468032837 1.0\n",
      "0.5400665998458862 1.0\n",
      "0.5970298051834106 1.0\n",
      "0.5235180258750916 0.0\n",
      "0.3459238111972809 0.0\n",
      "0.5840662121772766 1.0\n",
      "0.5006011128425598 1.0\n",
      "0.5733954310417175 0.0\n",
      "0.5897682905197144 1.0\n",
      "0.6676503419876099 1.0\n",
      "0.31973084807395935 0.0\n",
      "0.5074310898780823 0.0\n",
      "0.5886006951332092 1.0\n",
      "0.9043396711349487 1.0\n",
      "0.7404595017433167 1.0\n",
      "0.7076523303985596 0.0\n",
      "0.5870345234870911 1.0\n",
      "0.43746429681777954 0.0\n",
      "0.4893234968185425 0.0\n",
      "0.8508712649345398 1.0\n",
      "0.5517850518226624 0.0\n",
      "0.7553993463516235 0.0\n",
      "0.4864014685153961 0.0\n",
      "0.524042010307312 0.0\n",
      "0.6523528695106506 1.0\n",
      "0.5808907151222229 1.0\n",
      "0.4659385085105896 1.0\n",
      "0.5911213755607605 1.0\n",
      "0.5345243811607361 0.0\n",
      "0.49186971783638 0.0\n",
      "0.5156354308128357 1.0\n",
      "0.6282055974006653 1.0\n",
      "0.7801156044006348 1.0\n",
      "0.3957602083683014 0.0\n",
      "0.7279475331306458 0.0\n",
      "0.5340908169746399 0.0\n",
      "0.905497133731842 0.0\n",
      "0.3805311322212219 0.0\n",
      "0.8988974690437317 1.0\n",
      "0.5721905827522278 0.0\n",
      "0.7732910513877869 1.0\n",
      "0.5244733095169067 0.0\n",
      "0.1917046755552292 0.0\n",
      "0.44035691022872925 0.0\n",
      "0.4681142270565033 0.0\n",
      "0.6473076343536377 1.0\n",
      "0.41349533200263977 1.0\n",
      "0.22127315402030945 0.0\n",
      "0.08291066437959671 0.0\n",
      "0.4140121340751648 1.0\n",
      "0.47520923614501953 0.0\n",
      "0.929506778717041 1.0\n",
      "0.5931748151779175 1.0\n",
      "0.4499618411064148 0.0\n",
      "0.5012519955635071 1.0\n",
      "0.2830958366394043 0.0\n",
      "0.4208748936653137 0.0\n",
      "0.14097842574119568 0.0\n",
      "0.8380603790283203 0.0\n",
      "0.553968608379364 0.0\n",
      "0.5612033009529114 1.0\n",
      "TRAIN[steps=800] loss=0.628494 acc=0.641 P=0.581 R=0.833 F1=0.684931\n",
      "DEV[steps=800] loss=0.009694 acc=0.640 P=0.622 R=0.672 F1=0.646108 *\n",
      "model sim and label tuples:\n",
      "0.22304357588291168 0.0\n",
      "0.32884129881858826 1.0\n",
      "0.48169684410095215 1.0\n",
      "0.5355391502380371 0.0\n",
      "0.21801544725894928 0.0\n",
      "0.44025591015815735 0.0\n",
      "0.5582215189933777 1.0\n",
      "0.21495001018047333 0.0\n",
      "0.44652941823005676 0.0\n",
      "0.3483208417892456 1.0\n",
      "0.28819915652275085 0.0\n",
      "0.8398443460464478 1.0\n",
      "0.4102283716201782 1.0\n",
      "0.4565145969390869 0.0\n",
      "0.560500979423523 0.0\n",
      "0.3744041919708252 0.0\n",
      "0.4429028332233429 0.0\n",
      "0.45252686738967896 0.0\n",
      "0.635545551776886 0.0\n",
      "0.6283716559410095 0.0\n",
      "0.82613605260849 1.0\n",
      "0.34116119146347046 0.0\n",
      "0.34382250905036926 1.0\n",
      "0.8198050260543823 1.0\n",
      "0.3559601902961731 0.0\n",
      "0.45552828907966614 0.0\n",
      "0.9627885222434998 1.0\n",
      "0.6346245408058167 1.0\n",
      "0.6877084970474243 1.0\n",
      "0.6254255771636963 1.0\n",
      "0.5744317770004272 0.0\n",
      "0.5714399218559265 1.0\n",
      "0.46702200174331665 1.0\n",
      "0.6143741607666016 0.0\n",
      "0.39812374114990234 1.0\n",
      "0.39522913098335266 1.0\n",
      "0.4512057602405548 0.0\n",
      "0.34601062536239624 0.0\n",
      "0.41335025429725647 0.0\n",
      "0.6390509009361267 1.0\n",
      "0.4844719171524048 0.0\n",
      "0.44061192870140076 0.0\n",
      "0.3480921685695648 0.0\n",
      "0.494985431432724 0.0\n",
      "0.4594719111919403 0.0\n",
      "0.5936971306800842 1.0\n",
      "0.41367119550704956 0.0\n",
      "0.6172046065330505 0.0\n",
      "0.4016684591770172 1.0\n",
      "0.5357714891433716 1.0\n",
      "0.41331034898757935 1.0\n",
      "0.3487682640552521 1.0\n",
      "0.5149181485176086 1.0\n",
      "0.5992169976234436 1.0\n",
      "0.7016212344169617 1.0\n",
      "0.43753188848495483 1.0\n",
      "0.4645884037017822 1.0\n",
      "0.576206624507904 0.0\n",
      "0.7088249921798706 1.0\n",
      "0.7441920638084412 0.0\n",
      "0.597887396812439 1.0\n",
      "0.5242665410041809 0.0\n",
      "0.39240962266921997 0.0\n",
      "0.738924503326416 1.0\n",
      "TRAIN[steps=900] loss=0.626568 acc=0.625 P=0.613 R=0.613 F1=0.612903\n",
      "DEV[steps=900] loss=0.009622 acc=0.649 P=0.641 R=0.643 F1=0.641564 \n",
      "model sim and label tuples:\n",
      "0.0989033579826355 1.0\n",
      "0.3319830000400543 0.0\n",
      "0.28808802366256714 0.0\n",
      "0.25330325961112976 1.0\n",
      "0.5965145230293274 1.0\n",
      "0.26707297563552856 1.0\n",
      "0.4726014733314514 1.0\n",
      "0.2487141191959381 0.0\n",
      "0.23649531602859497 0.0\n",
      "0.30136075615882874 0.0\n",
      "0.25051403045654297 0.0\n",
      "0.6844757795333862 1.0\n",
      "0.35532915592193604 0.0\n",
      "0.28700438141822815 0.0\n",
      "0.4594701826572418 0.0\n",
      "0.0495106503367424 0.0\n",
      "0.49375680088996887 1.0\n",
      "0.5060308575630188 1.0\n",
      "0.2287103533744812 0.0\n",
      "0.7337000370025635 1.0\n",
      "0.2843715250492096 0.0\n",
      "0.24968306720256805 0.0\n",
      "0.39553844928741455 0.0\n",
      "0.6936082243919373 1.0\n",
      "0.5910910964012146 0.0\n",
      "0.39881429076194763 1.0\n",
      "0.5985221862792969 0.0\n",
      "0.5196059346199036 0.0\n",
      "0.5902945399284363 1.0\n",
      "0.21949677169322968 1.0\n",
      "0.34862807393074036 1.0\n",
      "0.40588364005088806 1.0\n",
      "0.8447727560997009 1.0\n",
      "0.3917049467563629 0.0\n",
      "0.7182294726371765 0.0\n",
      "0.3842647969722748 0.0\n",
      "0.33433806896209717 0.0\n",
      "0.1209576427936554 0.0\n",
      "0.40224549174308777 1.0\n",
      "0.383247971534729 0.0\n",
      "0.12327177822589874 0.0\n",
      "0.18797920644283295 0.0\n",
      "0.4498174488544464 0.0\n",
      "0.7130298018455505 0.0\n",
      "0.7237363457679749 0.0\n",
      "0.6195001602172852 1.0\n",
      "0.08565476536750793 0.0\n",
      "0.25146472454071045 1.0\n",
      "0.5678732395172119 1.0\n",
      "0.3897635340690613 0.0\n",
      "0.4867117702960968 1.0\n",
      "0.7128832936286926 1.0\n",
      "0.6664408445358276 1.0\n",
      "0.38417181372642517 0.0\n",
      "0.6000894904136658 1.0\n",
      "0.8806450963020325 1.0\n",
      "0.2537696957588196 0.0\n",
      "0.3972761631011963 1.0\n",
      "0.2816093862056732 0.0\n",
      "0.37143242359161377 1.0\n",
      "0.7201296091079712 1.0\n",
      "0.3505401015281677 0.0\n",
      "0.3457123041152954 0.0\n",
      "0.3479148745536804 1.0\n",
      "TRAIN[steps=1000] loss=0.611546 acc=0.703 P=0.727 R=0.552 F1=0.627451\n",
      "DEV[steps=1000] loss=0.009827 acc=0.645 P=0.661 R=0.562 F1=0.607337 \n",
      "model sim and label tuples:\n",
      "0.5333285331726074 0.0\n",
      "0.6676004528999329 1.0\n",
      "0.40717247128486633 0.0\n",
      "0.4865727126598358 0.0\n",
      "0.278018981218338 0.0\n",
      "0.2769971489906311 0.0\n",
      "0.4600864052772522 1.0\n",
      "0.749059796333313 1.0\n",
      "0.37254035472869873 0.0\n",
      "0.2539505958557129 0.0\n",
      "0.6935639977455139 1.0\n",
      "0.24260510504245758 0.0\n",
      "0.775598406791687 1.0\n",
      "0.33242157101631165 1.0\n",
      "0.31523457169532776 0.0\n",
      "0.11846522986888885 0.0\n",
      "0.4964558184146881 0.0\n",
      "0.6346455812454224 0.0\n",
      "0.47254088521003723 1.0\n",
      "0.5698052048683167 0.0\n",
      "0.43891534209251404 0.0\n",
      "0.5298779606819153 0.0\n",
      "0.5904258489608765 1.0\n",
      "0.46742114424705505 0.0\n",
      "0.7375924587249756 1.0\n",
      "0.47894757986068726 0.0\n",
      "0.4857780635356903 0.0\n",
      "0.24134089052677155 0.0\n",
      "0.4679070711135864 0.0\n",
      "0.4935908317565918 1.0\n",
      "0.3038933575153351 0.0\n",
      "0.3698045015335083 1.0\n",
      "0.4365396201610565 0.0\n",
      "0.3307131826877594 0.0\n",
      "0.18137584626674652 1.0\n",
      "0.22251176834106445 0.0\n",
      "0.3769820034503937 0.0\n",
      "0.26732149720191956 1.0\n",
      "0.43513041734695435 0.0\n",
      "0.33562731742858887 0.0\n",
      "0.5306671261787415 0.0\n",
      "0.3441002368927002 1.0\n",
      "0.32136139273643494 1.0\n",
      "0.2867037355899811 0.0\n",
      "0.6433684229850769 0.0\n",
      "0.6023153066635132 1.0\n",
      "0.18338878452777863 0.0\n",
      "0.6145934462547302 1.0\n",
      "0.331987589597702 0.0\n",
      "0.27043190598487854 0.0\n",
      "0.3507239818572998 0.0\n",
      "0.5677132606506348 1.0\n",
      "0.34575024247169495 0.0\n",
      "0.37243661284446716 0.0\n",
      "0.35925164818763733 1.0\n",
      "0.3798859417438507 0.0\n",
      "0.3385259807109833 0.0\n",
      "0.4158136546611786 1.0\n",
      "0.6130567193031311 1.0\n",
      "0.4845098555088043 0.0\n",
      "0.22294265031814575 0.0\n",
      "0.10119643807411194 1.0\n",
      "0.18592558801174164 0.0\n",
      "0.4783024489879608 1.0\n",
      "TRAIN[steps=1100] loss=0.608894 acc=0.656 P=0.524 R=0.478 F1=0.500000\n",
      "DEV[steps=1100] loss=0.009533 acc=0.658 P=0.673 R=0.584 F1=0.625356 \n",
      "model sim and label tuples:\n",
      "0.3349003195762634 0.0\n",
      "0.39254000782966614 0.0\n",
      "0.14201053977012634 0.0\n",
      "0.5538721680641174 1.0\n",
      "0.4719102084636688 0.0\n",
      "0.5067474246025085 0.0\n",
      "0.43648844957351685 0.0\n",
      "0.8112072348594666 1.0\n",
      "0.6672067046165466 1.0\n",
      "0.15989390015602112 0.0\n",
      "0.5991352796554565 0.0\n",
      "0.6323468089103699 0.0\n",
      "0.38447850942611694 1.0\n",
      "0.8254849910736084 1.0\n",
      "0.2610756456851959 1.0\n",
      "0.3310385048389435 0.0\n",
      "0.3867138922214508 0.0\n",
      "0.1757931113243103 1.0\n",
      "0.8526065349578857 1.0\n",
      "0.7740971446037292 1.0\n",
      "0.2424197643995285 0.0\n",
      "0.2740083336830139 1.0\n",
      "0.1203283742070198 0.0\n",
      "0.6256505846977234 1.0\n",
      "0.5221593379974365 1.0\n",
      "0.5169771313667297 0.0\n",
      "0.47906294465065 0.0\n",
      "0.4207788407802582 0.0\n",
      "0.6195387244224548 1.0\n",
      "0.5740026831626892 0.0\n",
      "0.4526572525501251 1.0\n",
      "0.3613491356372833 1.0\n",
      "0.7364500761032104 0.0\n",
      "0.39165863394737244 0.0\n",
      "0.396345317363739 0.0\n",
      "0.3027573227882385 1.0\n",
      "0.8859667778015137 1.0\n",
      "0.471586138010025 1.0\n",
      "0.47508037090301514 0.0\n",
      "0.28433775901794434 1.0\n",
      "0.1338837891817093 0.0\n",
      "0.8023334741592407 1.0\n",
      "0.47625938057899475 0.0\n",
      "0.27767452597618103 0.0\n",
      "0.5396023988723755 0.0\n",
      "0.6690847873687744 1.0\n",
      "0.5482490658760071 0.0\n",
      "0.9219533205032349 1.0\n",
      "0.5107547044754028 1.0\n",
      "0.21670196950435638 0.0\n",
      "0.8464155793190002 1.0\n",
      "0.29223206639289856 0.0\n",
      "0.7369111180305481 1.0\n",
      "0.055082302540540695 0.0\n",
      "0.1779305636882782 0.0\n",
      "0.3045303523540497 0.0\n",
      "0.7165729403495789 1.0\n",
      "0.34656885266304016 0.0\n",
      "0.448693186044693 0.0\n",
      "0.2633775472640991 1.0\n",
      "0.5098050236701965 1.0\n",
      "0.4057333171367645 0.0\n",
      "0.7710071802139282 1.0\n",
      "0.33285167813301086 0.0\n",
      "TRAIN[steps=1200] loss=0.566975 acc=0.719 P=0.704 R=0.655 F1=0.678571\n",
      "DEV[steps=1200] loss=0.009513 acc=0.655 P=0.650 R=0.638 F1=0.643914 \n",
      "EPOCH:  2\n",
      "model sim and label tuples:\n",
      "0.341838538646698 0.0\n",
      "0.8772749304771423 1.0\n",
      "0.7175218462944031 1.0\n",
      "0.3061065375804901 0.0\n",
      "0.20346741378307343 1.0\n",
      "0.16894827783107758 0.0\n",
      "0.49040332436561584 1.0\n",
      "0.5291293263435364 0.0\n",
      "0.28646934032440186 1.0\n",
      "0.17462441325187683 1.0\n",
      "0.14071746170520782 0.0\n",
      "0.6246485710144043 1.0\n",
      "0.22545836865901947 1.0\n",
      "0.9227052927017212 1.0\n",
      "0.2710794508457184 1.0\n",
      "0.8747493028640747 1.0\n",
      "0.1356218308210373 0.0\n",
      "0.42490336298942566 0.0\n",
      "0.44468656182289124 0.0\n",
      "0.5450395345687866 0.0\n",
      "0.5056968927383423 0.0\n",
      "0.25860685110092163 1.0\n",
      "0.40008240938186646 1.0\n",
      "0.5061065554618835 1.0\n",
      "0.26834750175476074 0.0\n",
      "0.8027684092521667 1.0\n",
      "0.2623388469219208 0.0\n",
      "0.1868690699338913 0.0\n",
      "0.23567485809326172 0.0\n",
      "0.10681379586458206 0.0\n",
      "0.3961295187473297 1.0\n",
      "0.6549758911132812 1.0\n",
      "0.3534993529319763 0.0\n",
      "0.5198150277137756 0.0\n",
      "0.6912885308265686 1.0\n",
      "0.7021970748901367 1.0\n",
      "0.4323824942111969 0.0\n",
      "0.5986025929450989 1.0\n",
      "0.400941401720047 1.0\n",
      "0.2627323269844055 1.0\n",
      "0.5959634184837341 0.0\n",
      "0.56092369556427 1.0\n",
      "0.2947580814361572 1.0\n",
      "0.23018494248390198 0.0\n",
      "0.5633791089057922 1.0\n",
      "0.14695210754871368 0.0\n",
      "0.4129422605037689 0.0\n",
      "0.41395053267478943 0.0\n",
      "0.6589314937591553 0.0\n",
      "0.8809655904769897 1.0\n",
      "0.4043000340461731 1.0\n",
      "0.41076523065567017 1.0\n",
      "0.2806635797023773 0.0\n",
      "0.13718697428703308 0.0\n",
      "0.586358368396759 1.0\n",
      "0.49493640661239624 0.0\n",
      "0.3592511713504791 1.0\n",
      "0.48575735092163086 0.0\n",
      "0.6278293132781982 1.0\n",
      "0.43932685256004333 1.0\n",
      "0.6341432332992554 0.0\n",
      "0.4933086037635803 0.0\n",
      "0.20491237938404083 0.0\n",
      "0.6039389371871948 1.0\n",
      "TRAIN[steps=1300] loss=0.620260 acc=0.609 P=0.643 R=0.545 F1=0.590164\n",
      "DEV[steps=1300] loss=0.009692 acc=0.658 P=0.709 R=0.509 F1=0.592124 \n",
      "model sim and label tuples:\n",
      "0.1949903666973114 0.0\n",
      "0.7221857309341431 1.0\n",
      "0.7043811678886414 1.0\n",
      "0.6245443224906921 1.0\n",
      "0.7339668273925781 1.0\n",
      "0.36930572986602783 1.0\n",
      "0.9673125743865967 1.0\n",
      "0.371330201625824 1.0\n",
      "0.5127958655357361 1.0\n",
      "0.612386167049408 1.0\n",
      "0.4188562035560608 0.0\n",
      "0.34178832173347473 0.0\n",
      "0.8529766201972961 0.0\n",
      "0.23533102869987488 0.0\n",
      "0.4191375970840454 1.0\n",
      "0.6262270212173462 1.0\n",
      "0.8486329913139343 1.0\n",
      "0.47256505489349365 1.0\n",
      "0.7191987633705139 1.0\n",
      "0.28229081630706787 0.0\n",
      "0.23202520608901978 0.0\n",
      "0.4063393771648407 0.0\n",
      "0.05731215700507164 0.0\n",
      "0.6601420044898987 1.0\n",
      "0.2908678352832794 0.0\n",
      "0.6322384476661682 1.0\n",
      "0.3416002094745636 1.0\n",
      "0.32810238003730774 0.0\n",
      "0.5407861471176147 0.0\n",
      "0.40060698986053467 0.0\n",
      "0.3533839285373688 1.0\n",
      "0.40387994050979614 0.0\n",
      "0.45113247632980347 0.0\n",
      "0.2853659689426422 0.0\n",
      "0.38379621505737305 0.0\n",
      "0.5320598483085632 0.0\n",
      "0.45362943410873413 1.0\n",
      "0.3472607433795929 0.0\n",
      "0.27598658204078674 1.0\n",
      "0.6033883094787598 1.0\n",
      "0.4019329845905304 0.0\n",
      "0.5256376266479492 0.0\n",
      "0.4098225235939026 1.0\n",
      "0.9388939142227173 1.0\n",
      "0.5236527919769287 0.0\n",
      "0.9127826690673828 1.0\n",
      "0.11063329130411148 0.0\n",
      "0.10822397470474243 1.0\n",
      "0.5062956213951111 1.0\n",
      "0.3989820182323456 1.0\n",
      "0.7414970397949219 1.0\n",
      "0.5772894620895386 1.0\n",
      "0.45703038573265076 1.0\n",
      "0.29131123423576355 0.0\n",
      "0.7715000510215759 1.0\n",
      "0.6363847255706787 1.0\n",
      "0.5738959312438965 1.0\n",
      "0.5558608770370483 0.0\n",
      "0.33834412693977356 0.0\n",
      "0.39550626277923584 1.0\n",
      "0.3601488471031189 1.0\n",
      "0.7501831650733948 1.0\n",
      "0.9130489230155945 1.0\n",
      "0.28928765654563904 0.0\n",
      "TRAIN[steps=1400] loss=0.571947 acc=0.688 P=0.793 R=0.622 F1=0.696970\n",
      "DEV[steps=1400] loss=0.009548 acc=0.664 P=0.657 R=0.652 F1=0.654519 *\n",
      "model sim and label tuples:\n",
      "0.9174800515174866 1.0\n",
      "0.06100032106041908 0.0\n",
      "0.37580791115760803 0.0\n",
      "0.07901307940483093 0.0\n",
      "0.3245680034160614 1.0\n",
      "0.1560487151145935 0.0\n",
      "0.5685933232307434 1.0\n",
      "0.2550622224807739 0.0\n",
      "0.7269935011863708 1.0\n",
      "0.8145554661750793 1.0\n",
      "0.06263642758131027 0.0\n",
      "0.38412654399871826 1.0\n",
      "0.9308931827545166 1.0\n",
      "0.426790326833725 0.0\n",
      "0.18855412304401398 0.0\n",
      "0.3075585961341858 0.0\n",
      "0.25885987281799316 1.0\n",
      "0.49439457058906555 0.0\n",
      "0.4556626081466675 1.0\n",
      "0.4363085925579071 0.0\n",
      "0.5146766901016235 1.0\n",
      "0.34397947788238525 0.0\n",
      "0.5922043919563293 0.0\n",
      "0.34903743863105774 0.0\n",
      "0.5472181439399719 0.0\n",
      "0.37222209572792053 0.0\n",
      "0.36141660809516907 0.0\n",
      "0.18317796289920807 1.0\n",
      "0.493592768907547 0.0\n",
      "0.463951975107193 1.0\n",
      "0.030710067600011826 0.0\n",
      "0.4452084004878998 0.0\n",
      "0.8349630832672119 1.0\n",
      "0.32291701436042786 1.0\n",
      "0.15931858122348785 0.0\n",
      "0.41809308528900146 1.0\n",
      "0.27793896198272705 1.0\n",
      "0.6962810754776001 1.0\n",
      "0.849595844745636 1.0\n",
      "0.5070685744285583 0.0\n",
      "0.7109113931655884 1.0\n",
      "0.3407967984676361 0.0\n",
      "0.5742783546447754 1.0\n",
      "0.38408148288726807 0.0\n",
      "0.29836684465408325 0.0\n",
      "0.6983674168586731 1.0\n",
      "0.5522323250770569 1.0\n",
      "0.16510984301567078 1.0\n",
      "0.34838566184043884 0.0\n",
      "0.6792315244674683 1.0\n",
      "0.689346194267273 1.0\n",
      "0.40657395124435425 0.0\n",
      "0.2690759301185608 1.0\n",
      "0.4225669503211975 0.0\n",
      "0.30639711022377014 1.0\n",
      "0.46301186084747314 0.0\n",
      "0.3346433639526367 0.0\n",
      "0.5406262874603271 1.0\n",
      "0.4796004891395569 0.0\n",
      "0.690112829208374 1.0\n",
      "0.05790756270289421 0.0\n",
      "0.7923046350479126 0.0\n",
      "0.22844238579273224 0.0\n",
      "0.3523305356502533 1.0\n",
      "TRAIN[steps=1500] loss=0.578349 acc=0.703 P=0.739 R=0.567 F1=0.641509\n",
      "DEV[steps=1500] loss=0.009359 acc=0.669 P=0.663 R=0.658 F1=0.660566 *\n",
      "model sim and label tuples:\n",
      "0.13076910376548767 0.0\n",
      "0.721261203289032 1.0\n",
      "0.3072468936443329 0.0\n",
      "0.5813305377960205 0.0\n",
      "0.1364954710006714 1.0\n",
      "0.873275637626648 1.0\n",
      "0.622290849685669 0.0\n",
      "0.4133390784263611 0.0\n",
      "0.3500247597694397 1.0\n",
      "0.19805854558944702 0.0\n",
      "0.9361804127693176 1.0\n",
      "0.43283048272132874 0.0\n",
      "0.42643728852272034 1.0\n",
      "0.3769434988498688 0.0\n",
      "0.34534481167793274 0.0\n",
      "0.08521868288516998 1.0\n",
      "0.880513608455658 1.0\n",
      "0.4133312702178955 0.0\n",
      "0.23641693592071533 0.0\n",
      "0.6556813716888428 0.0\n",
      "0.37520599365234375 1.0\n",
      "0.1747349202632904 1.0\n",
      "0.4952560365200043 0.0\n",
      "0.43203723430633545 0.0\n",
      "0.4457477927207947 1.0\n",
      "0.541744589805603 1.0\n",
      "0.3211318254470825 0.0\n",
      "0.7674093842506409 1.0\n",
      "0.4872153699398041 1.0\n",
      "0.6633942127227783 1.0\n",
      "0.39373427629470825 0.0\n",
      "0.27654948830604553 0.0\n",
      "0.556533694267273 0.0\n",
      "0.858079731464386 1.0\n",
      "0.6874264478683472 0.0\n",
      "0.14072152972221375 0.0\n",
      "0.1808410882949829 0.0\n",
      "0.892423152923584 1.0\n",
      "0.43809595704078674 1.0\n",
      "0.15711218118667603 0.0\n",
      "0.811206579208374 1.0\n",
      "0.6116213202476501 1.0\n",
      "0.051021188497543335 0.0\n",
      "0.5487284064292908 1.0\n",
      "0.6795516610145569 1.0\n",
      "0.11816858500242233 0.0\n",
      "0.28232109546661377 0.0\n",
      "0.585344135761261 0.0\n",
      "0.16874825954437256 0.0\n",
      "0.8831871747970581 0.0\n",
      "0.46494486927986145 0.0\n",
      "0.641328752040863 1.0\n",
      "0.3979839086532593 1.0\n",
      "0.28231489658355713 1.0\n",
      "0.881718099117279 1.0\n",
      "0.7520413398742676 0.0\n",
      "0.5594140291213989 1.0\n",
      "0.17565271258354187 1.0\n",
      "0.5892446637153625 0.0\n",
      "0.6293485164642334 1.0\n",
      "0.5494952201843262 0.0\n",
      "0.4263382852077484 1.0\n",
      "0.5917755365371704 1.0\n",
      "0.7845954895019531 1.0\n",
      "TRAIN[steps=1600] loss=0.645576 acc=0.641 P=0.645 R=0.625 F1=0.634921\n",
      "DEV[steps=1600] loss=0.009346 acc=0.671 P=0.642 R=0.738 F1=0.686761 *\n",
      "model sim and label tuples:\n",
      "0.613664984703064 0.0\n",
      "0.5655555129051208 1.0\n",
      "0.5874322056770325 1.0\n",
      "0.8663005828857422 1.0\n",
      "0.5327150821685791 0.0\n",
      "0.20220063626766205 0.0\n",
      "0.43305832147598267 0.0\n",
      "0.3471243977546692 0.0\n",
      "0.3118513226509094 1.0\n",
      "0.41332337260246277 0.0\n",
      "0.36555469036102295 1.0\n",
      "0.6492969989776611 1.0\n",
      "0.3274375796318054 0.0\n",
      "0.43026062846183777 0.0\n",
      "0.18265333771705627 0.0\n",
      "0.46347683668136597 1.0\n",
      "0.4112370014190674 0.0\n",
      "0.540111780166626 1.0\n",
      "0.5881684422492981 1.0\n",
      "0.17169153690338135 0.0\n",
      "0.8204537034034729 1.0\n",
      "0.27918121218681335 0.0\n",
      "0.20750625431537628 0.0\n",
      "0.5953902006149292 1.0\n",
      "0.12948620319366455 0.0\n",
      "0.6814648509025574 1.0\n",
      "0.3536823093891144 0.0\n",
      "0.3247908055782318 1.0\n",
      "0.0733296200633049 0.0\n",
      "0.5126980543136597 1.0\n",
      "0.6265298128128052 0.0\n",
      "0.5340181589126587 0.0\n",
      "0.5309424996376038 0.0\n",
      "0.15366065502166748 1.0\n",
      "0.03272158280014992 0.0\n",
      "0.5917511582374573 1.0\n",
      "0.43752148747444153 1.0\n",
      "0.18406303226947784 0.0\n",
      "0.6480882167816162 1.0\n",
      "0.3468950688838959 0.0\n",
      "0.16589504480361938 0.0\n",
      "0.3609754741191864 1.0\n",
      "0.9673787951469421 1.0\n",
      "0.11403306573629379 0.0\n",
      "0.9333161115646362 1.0\n",
      "0.7560462951660156 1.0\n",
      "0.5729102492332458 0.0\n",
      "0.5347967147827148 0.0\n",
      "0.8072944283485413 0.0\n",
      "0.7319211959838867 1.0\n",
      "0.20214992761611938 1.0\n",
      "0.36086341738700867 0.0\n",
      "0.2459459900856018 0.0\n",
      "0.3394201397895813 0.0\n",
      "0.5778046250343323 1.0\n",
      "0.14163550734519958 1.0\n",
      "0.9146831035614014 1.0\n",
      "0.46698060631752014 1.0\n",
      "0.12989769876003265 0.0\n",
      "0.2437896877527237 0.0\n",
      "0.6305649280548096 1.0\n",
      "0.4764650762081146 1.0\n",
      "0.030271008610725403 0.0\n",
      "0.7594486474990845 0.0\n",
      "TRAIN[steps=1700] loss=0.570160 acc=0.688 P=0.679 R=0.633 F1=0.655172\n",
      "DEV[steps=1700] loss=0.009393 acc=0.674 P=0.669 R=0.657 F1=0.662942 \n",
      "model sim and label tuples:\n",
      "0.5289382338523865 1.0\n",
      "0.441207617521286 1.0\n",
      "0.3956961929798126 1.0\n",
      "0.20690591633319855 0.0\n",
      "0.7204899787902832 1.0\n",
      "0.19612258672714233 0.0\n",
      "0.8835565447807312 1.0\n",
      "0.555519163608551 1.0\n",
      "0.5979307293891907 1.0\n",
      "0.5968686938285828 0.0\n",
      "0.30827459692955017 0.0\n",
      "0.2923520505428314 1.0\n",
      "0.29289892315864563 0.0\n",
      "0.18678918480873108 0.0\n",
      "0.46891096234321594 0.0\n",
      "0.852863609790802 1.0\n",
      "0.9356401562690735 0.0\n",
      "0.20142962038516998 1.0\n",
      "0.8298591375350952 1.0\n",
      "0.9166785478591919 1.0\n",
      "0.7762757539749146 0.0\n",
      "0.4392572343349457 0.0\n",
      "0.474142462015152 0.0\n",
      "0.6705729365348816 0.0\n",
      "0.31472155451774597 1.0\n",
      "0.24749694764614105 0.0\n",
      "0.635708749294281 1.0\n",
      "0.08508389443159103 0.0\n",
      "0.5949311256408691 1.0\n",
      "0.3068055510520935 0.0\n",
      "0.7849454283714294 1.0\n",
      "0.4162862002849579 0.0\n",
      "0.5369894504547119 0.0\n",
      "0.46194973587989807 0.0\n",
      "0.9485611319541931 1.0\n",
      "0.23883920907974243 0.0\n",
      "0.6212677359580994 1.0\n",
      "0.7339353561401367 1.0\n",
      "0.34236884117126465 1.0\n",
      "0.1835450679063797 0.0\n",
      "0.5620846748352051 0.0\n",
      "0.44337573647499084 1.0\n",
      "0.34129559993743896 0.0\n",
      "0.239814892411232 0.0\n",
      "0.4580753743648529 0.0\n",
      "0.44423583149909973 1.0\n",
      "0.6613731980323792 1.0\n",
      "0.025371622294187546 0.0\n",
      "0.116365447640419 0.0\n",
      "0.6263734698295593 1.0\n",
      "0.07659032195806503 0.0\n",
      "0.857977569103241 1.0\n",
      "0.2575034201145172 0.0\n",
      "0.3055375814437866 1.0\n",
      "0.5021007657051086 1.0\n",
      "0.2751874625682831 0.0\n",
      "0.1907622218132019 0.0\n",
      "0.6593618392944336 1.0\n",
      "0.5515153408050537 1.0\n",
      "0.5261512398719788 1.0\n",
      "0.2458028644323349 0.0\n",
      "0.5717734098434448 0.0\n",
      "0.37376415729522705 0.0\n",
      "0.3697814643383026 0.0\n",
      "TRAIN[steps=1800] loss=0.555441 acc=0.750 P=0.750 R=0.700 F1=0.724138\n",
      "DEV[steps=1800] loss=0.009169 acc=0.680 P=0.659 R=0.716 F1=0.686253 \n",
      "EPOCH:  3\n",
      "model sim and label tuples:\n",
      "0.3617098033428192 1.0\n",
      "0.9090789556503296 1.0\n",
      "0.19976364076137543 0.0\n",
      "0.943530797958374 1.0\n",
      "0.639276385307312 1.0\n",
      "0.724074125289917 1.0\n",
      "0.3391110301017761 1.0\n",
      "0.28046560287475586 0.0\n",
      "0.621484100818634 0.0\n",
      "0.6586583852767944 0.0\n",
      "0.8565906882286072 1.0\n",
      "0.7725489735603333 1.0\n",
      "0.9253998398780823 1.0\n",
      "0.7755296230316162 0.0\n",
      "0.9020524621009827 1.0\n",
      "0.43126168847084045 0.0\n",
      "0.6986948251724243 1.0\n",
      "0.49127495288848877 0.0\n",
      "0.517347514629364 0.0\n",
      "0.7527798414230347 0.0\n",
      "0.48153048753738403 0.0\n",
      "0.8085993528366089 1.0\n",
      "0.6364626288414001 1.0\n",
      "0.3300822377204895 0.0\n",
      "0.8205336928367615 1.0\n",
      "0.9277570843696594 1.0\n",
      "0.035314761102199554 0.0\n",
      "0.44821983575820923 0.0\n",
      "0.5482515692710876 0.0\n",
      "0.9295564889907837 1.0\n",
      "0.3051404356956482 0.0\n",
      "0.6166536808013916 1.0\n",
      "0.4645625650882721 0.0\n",
      "0.1353987157344818 0.0\n",
      "0.547558069229126 0.0\n",
      "0.5239085555076599 0.0\n",
      "0.688534677028656 0.0\n",
      "0.7294729948043823 1.0\n",
      "0.9943066239356995 1.0\n",
      "0.4019288122653961 0.0\n",
      "0.528567373752594 1.0\n",
      "0.638266921043396 0.0\n",
      "0.21603068709373474 1.0\n",
      "0.571073591709137 0.0\n",
      "0.8243018388748169 1.0\n",
      "0.8374330401420593 1.0\n",
      "0.5620757937431335 0.0\n",
      "0.8202798962593079 1.0\n",
      "0.8859052062034607 1.0\n",
      "0.6755667328834534 0.0\n",
      "0.5469441413879395 1.0\n",
      "0.11110324412584305 1.0\n",
      "0.48649072647094727 0.0\n",
      "0.648932933807373 1.0\n",
      "0.3666020929813385 0.0\n",
      "0.5718353390693665 1.0\n",
      "0.740778923034668 1.0\n",
      "0.8576704859733582 1.0\n",
      "0.4208342730998993 0.0\n",
      "0.2931578755378723 0.0\n",
      "0.08449672162532806 0.0\n",
      "0.6680346131324768 1.0\n",
      "0.5085363388061523 0.0\n",
      "0.13492965698242188 0.0\n",
      "TRAIN[steps=1900] loss=0.536927 acc=0.672 P=0.622 R=0.875 F1=0.727273\n",
      "DEV[steps=1900] loss=0.009471 acc=0.665 P=0.630 R=0.763 F1=0.690224 *\n",
      "model sim and label tuples:\n",
      "0.3595275282859802 0.0\n",
      "0.571894109249115 1.0\n",
      "0.9898440837860107 1.0\n",
      "0.44712990522384644 1.0\n",
      "0.9282082915306091 1.0\n",
      "0.6593866944313049 1.0\n",
      "0.8174421787261963 1.0\n",
      "0.48715776205062866 1.0\n",
      "0.09543361514806747 0.0\n",
      "0.492562472820282 1.0\n",
      "0.30661311745643616 1.0\n",
      "0.3008882403373718 0.0\n",
      "0.07813224196434021 0.0\n",
      "0.25956863164901733 1.0\n",
      "0.5770331621170044 0.0\n",
      "0.7383443713188171 1.0\n",
      "0.7820718288421631 1.0\n",
      "0.9388331770896912 1.0\n",
      "0.2895323634147644 0.0\n",
      "0.39706695079803467 1.0\n",
      "0.6301908493041992 1.0\n",
      "0.571155846118927 1.0\n",
      "0.7479898929595947 1.0\n",
      "0.5388445258140564 0.0\n",
      "0.7169732451438904 1.0\n",
      "0.007142450660467148 0.0\n",
      "0.5602313876152039 1.0\n",
      "0.06922552734613419 0.0\n",
      "0.45115602016448975 0.0\n",
      "0.6658897995948792 1.0\n",
      "0.01416452694684267 0.0\n",
      "0.6391701102256775 0.0\n",
      "0.6236693859100342 1.0\n",
      "0.04070281982421875 0.0\n",
      "0.0290219746530056 0.0\n",
      "0.913533091545105 1.0\n",
      "0.964419960975647 1.0\n",
      "0.9849079847335815 1.0\n",
      "0.1934664398431778 0.0\n",
      "0.7290130257606506 0.0\n",
      "0.07786116749048233 0.0\n",
      "0.8481251001358032 0.0\n",
      "0.5379975438117981 0.0\n",
      "0.6468613743782043 0.0\n",
      "0.4092702269554138 0.0\n",
      "0.332731157541275 1.0\n",
      "0.06850431859493256 1.0\n",
      "0.8469975590705872 1.0\n",
      "0.49023228883743286 1.0\n",
      "0.6273085474967957 0.0\n",
      "0.4254220426082611 1.0\n",
      "0.38558200001716614 0.0\n",
      "0.7182368040084839 1.0\n",
      "0.15353253483772278 0.0\n",
      "0.17000563442707062 0.0\n",
      "0.08826480805873871 0.0\n",
      "0.7527632117271423 1.0\n",
      "0.20534665882587433 0.0\n",
      "0.9277786612510681 1.0\n",
      "0.6006616353988647 0.0\n",
      "0.5676599144935608 1.0\n",
      "0.49207815527915955 0.0\n",
      "0.8806887865066528 1.0\n",
      "0.4880043864250183 1.0\n",
      "TRAIN[steps=2000] loss=0.511510 acc=0.734 P=0.737 R=0.800 F1=0.767123\n",
      "DEV[steps=2000] loss=0.009716 acc=0.675 P=0.651 R=0.719 F1=0.683492 \n",
      "model sim and label tuples:\n",
      "0.15689197182655334 0.0\n",
      "0.6314151883125305 0.0\n",
      "0.712567150592804 1.0\n",
      "0.5112640857696533 0.0\n",
      "0.7068170309066772 0.0\n",
      "0.46684059500694275 0.0\n",
      "0.7356150150299072 1.0\n",
      "0.49184930324554443 1.0\n",
      "0.6641307473182678 1.0\n",
      "0.7454906105995178 1.0\n",
      "0.9762488007545471 1.0\n",
      "0.3686106503009796 1.0\n",
      "0.15422075986862183 0.0\n",
      "0.6907601952552795 1.0\n",
      "0.18641680479049683 0.0\n",
      "0.9939900636672974 1.0\n",
      "0.5315161347389221 1.0\n",
      "0.2031884640455246 1.0\n",
      "0.22581396996974945 0.0\n",
      "0.41465821862220764 1.0\n",
      "0.738503098487854 1.0\n",
      "0.7344189286231995 1.0\n",
      "0.5822137594223022 0.0\n",
      "0.8779320120811462 0.0\n",
      "0.04201905429363251 0.0\n",
      "0.6411005258560181 1.0\n",
      "0.5900972485542297 0.0\n",
      "0.5248780250549316 0.0\n",
      "0.6920680999755859 1.0\n",
      "0.6211490035057068 1.0\n",
      "0.6115871667861938 1.0\n",
      "0.045186206698417664 0.0\n",
      "0.9490976333618164 1.0\n",
      "0.3939902186393738 0.0\n",
      "0.400639146566391 0.0\n",
      "0.7516310811042786 0.0\n",
      "0.9751302599906921 1.0\n",
      "0.6088067293167114 0.0\n",
      "0.7767524719238281 1.0\n",
      "0.43896499276161194 1.0\n",
      "0.013906779699027538 0.0\n",
      "0.20702888071537018 0.0\n",
      "0.1463077962398529 0.0\n",
      "0.1758037805557251 0.0\n",
      "0.6318053603172302 1.0\n",
      "0.27937033772468567 0.0\n",
      "0.4371778070926666 1.0\n",
      "0.22086964547634125 0.0\n",
      "0.8324389457702637 1.0\n",
      "0.15828083455562592 0.0\n",
      "0.6273894906044006 0.0\n",
      "0.6011605858802795 0.0\n",
      "0.3851095139980316 0.0\n",
      "0.223639577627182 1.0\n",
      "0.49034932255744934 0.0\n",
      "0.9456015229225159 1.0\n",
      "0.530379056930542 1.0\n",
      "0.9209911227226257 1.0\n",
      "0.6277825236320496 0.0\n",
      "0.5113897919654846 0.0\n",
      "0.35072025656700134 1.0\n",
      "0.4862622916698456 1.0\n",
      "0.6898075938224792 1.0\n",
      "0.05071059986948967 0.0\n",
      "TRAIN[steps=2100] loss=0.540224 acc=0.672 P=0.641 R=0.781 F1=0.704225\n",
      "DEV[steps=2100] loss=0.009374 acc=0.670 P=0.635 R=0.763 F1=0.693167 *\n",
      "model sim and label tuples:\n",
      "0.4213791489601135 0.0\n",
      "0.9920397996902466 1.0\n",
      "0.980525553226471 1.0\n",
      "0.9758038520812988 1.0\n",
      "0.5464863181114197 0.0\n",
      "0.27983394265174866 1.0\n",
      "0.7675986886024475 1.0\n",
      "0.21106857061386108 0.0\n",
      "0.3929566740989685 1.0\n",
      "0.6904587745666504 1.0\n",
      "0.10715939104557037 0.0\n",
      "0.08433191478252411 1.0\n",
      "0.6472144722938538 1.0\n",
      "0.9762208461761475 1.0\n",
      "0.4424266815185547 1.0\n",
      "0.13690881431102753 0.0\n",
      "0.13213178515434265 0.0\n",
      "0.09769310057163239 0.0\n",
      "0.357355535030365 0.0\n",
      "0.24724571406841278 0.0\n",
      "0.1309371292591095 0.0\n",
      "0.022612590342760086 0.0\n",
      "0.5664798617362976 1.0\n",
      "0.4992797076702118 0.0\n",
      "0.25591349601745605 0.0\n",
      "0.13537470996379852 0.0\n",
      "0.6023405194282532 0.0\n",
      "0.17390277981758118 0.0\n",
      "0.9825897216796875 1.0\n",
      "0.1685544103384018 0.0\n",
      "0.8352357149124146 1.0\n",
      "0.7874703407287598 1.0\n",
      "0.3775424063205719 1.0\n",
      "0.7691888213157654 1.0\n",
      "0.22660355269908905 0.0\n",
      "0.021881241351366043 0.0\n",
      "0.8591984510421753 0.0\n",
      "0.7217569351196289 1.0\n",
      "0.9755503535270691 1.0\n",
      "0.4850356876850128 0.0\n",
      "0.3627350926399231 1.0\n",
      "0.20100194215774536 0.0\n",
      "0.363485723733902 0.0\n",
      "0.9855949878692627 1.0\n",
      "0.8553540110588074 1.0\n",
      "0.684883177280426 1.0\n",
      "0.4733726382255554 1.0\n",
      "0.43156394362449646 1.0\n",
      "0.01336831133812666 0.0\n",
      "0.13478484749794006 0.0\n",
      "0.4658040404319763 1.0\n",
      "0.34332743287086487 0.0\n",
      "0.06910192966461182 0.0\n",
      "0.7267735600471497 1.0\n",
      "0.9852774739265442 1.0\n",
      "0.38223323225975037 0.0\n",
      "0.20591042935848236 0.0\n",
      "0.7778868675231934 1.0\n",
      "0.6272128224372864 1.0\n",
      "0.7438238263130188 1.0\n",
      "0.04191414639353752 0.0\n",
      "0.349740594625473 1.0\n",
      "0.10783860087394714 0.0\n",
      "0.18034633994102478 1.0\n",
      "TRAIN[steps=2200] loss=0.433747 acc=0.750 P=0.815 R=0.667 F1=0.733333\n",
      "DEV[steps=2200] loss=0.009483 acc=0.681 P=0.678 R=0.660 F1=0.669090 \n",
      "model sim and label tuples:\n",
      "0.333473265171051 1.0\n",
      "0.054096393287181854 0.0\n",
      "0.9118872880935669 1.0\n",
      "0.1767583042383194 0.0\n",
      "0.2895699739456177 0.0\n",
      "0.620425283908844 1.0\n",
      "0.9204683303833008 1.0\n",
      "0.771400511264801 1.0\n",
      "0.049551822245121 0.0\n",
      "0.3200116753578186 0.0\n",
      "0.5547395348548889 0.0\n",
      "0.7429402470588684 1.0\n",
      "0.7759946584701538 1.0\n",
      "0.7066884636878967 1.0\n",
      "0.08990111202001572 0.0\n",
      "0.40358904004096985 1.0\n",
      "0.43011805415153503 1.0\n",
      "0.9466522932052612 1.0\n",
      "0.9953750967979431 1.0\n",
      "0.6775681376457214 1.0\n",
      "0.32007133960723877 0.0\n",
      "0.32679134607315063 1.0\n",
      "0.2660031020641327 0.0\n",
      "0.6393631100654602 1.0\n",
      "0.6488648653030396 1.0\n",
      "0.7269538044929504 1.0\n",
      "0.7892829179763794 1.0\n",
      "0.9821445941925049 1.0\n",
      "0.48679855465888977 0.0\n",
      "0.6949096918106079 0.0\n",
      "0.5795376300811768 1.0\n",
      "0.23731271922588348 1.0\n",
      "0.00671772425994277 0.0\n",
      "0.6233938932418823 0.0\n",
      "0.8621178865432739 1.0\n",
      "0.5894109606742859 1.0\n",
      "0.7192589640617371 1.0\n",
      "0.9668009281158447 1.0\n",
      "0.9952595829963684 1.0\n",
      "0.0765920877456665 1.0\n",
      "0.3949287533760071 1.0\n",
      "0.4832494854927063 0.0\n",
      "0.6270951628684998 0.0\n",
      "0.5832377076148987 0.0\n",
      "0.5042992830276489 1.0\n",
      "0.006958626676350832 0.0\n",
      "0.1985071450471878 0.0\n",
      "0.5443214774131775 1.0\n",
      "0.7386990189552307 1.0\n",
      "0.0801347941160202 0.0\n",
      "0.3767554461956024 0.0\n",
      "0.18386168777942657 0.0\n",
      "0.8089990019798279 1.0\n",
      "0.6262282729148865 0.0\n",
      "0.6256991028785706 1.0\n",
      "0.5669829249382019 1.0\n",
      "0.46771058440208435 1.0\n",
      "0.8518816232681274 0.0\n",
      "0.08106078207492828 0.0\n",
      "0.7431542277336121 1.0\n",
      "0.6275660991668701 0.0\n",
      "0.524468183517456 1.0\n",
      "0.8601037263870239 1.0\n",
      "0.5876426100730896 1.0\n",
      "TRAIN[steps=2300] loss=0.505866 acc=0.719 P=0.756 R=0.795 F1=0.775000\n",
      "DEV[steps=2300] loss=0.009257 acc=0.680 P=0.650 R=0.747 F1=0.695329 *\n",
      "model sim and label tuples:\n",
      "0.48357251286506653 0.0\n",
      "0.5304719805717468 1.0\n",
      "0.3043794333934784 0.0\n",
      "0.32584452629089355 1.0\n",
      "0.3743535280227661 0.0\n",
      "0.44333595037460327 0.0\n",
      "0.2502444088459015 0.0\n",
      "0.7384892106056213 0.0\n",
      "0.6350517272949219 1.0\n",
      "0.5056374073028564 0.0\n",
      "0.4469447135925293 0.0\n",
      "0.787929356098175 1.0\n",
      "0.3628649115562439 0.0\n",
      "0.008352851495146751 0.0\n",
      "0.031524352729320526 0.0\n",
      "0.6836973428726196 1.0\n",
      "0.09930823743343353 1.0\n",
      "0.5635355114936829 1.0\n",
      "0.3200189769268036 1.0\n",
      "0.2527036666870117 1.0\n",
      "0.4707004725933075 0.0\n",
      "0.6822516918182373 0.0\n",
      "0.6387483477592468 0.0\n",
      "0.9913572072982788 1.0\n",
      "0.9426482319831848 1.0\n",
      "0.5444867014884949 0.0\n",
      "0.8843731880187988 1.0\n",
      "0.5311070084571838 0.0\n",
      "0.48809194564819336 1.0\n",
      "0.4196856617927551 1.0\n",
      "0.991297721862793 1.0\n",
      "0.673094630241394 1.0\n",
      "0.7185860276222229 1.0\n",
      "0.9161918759346008 1.0\n",
      "0.4967030882835388 1.0\n",
      "0.5813949108123779 1.0\n",
      "0.20074725151062012 0.0\n",
      "0.5174018144607544 0.0\n",
      "0.632706344127655 1.0\n",
      "0.5039133429527283 1.0\n",
      "0.006074340082705021 0.0\n",
      "0.336895227432251 0.0\n",
      "0.48120445013046265 0.0\n",
      "0.34518611431121826 1.0\n",
      "0.26596856117248535 0.0\n",
      "0.8555475473403931 1.0\n",
      "0.6342174410820007 0.0\n",
      "0.6539834141731262 0.0\n",
      "0.3782862424850464 0.0\n",
      "0.8274573087692261 1.0\n",
      "0.34920430183410645 0.0\n",
      "0.1395411193370819 1.0\n",
      "0.5375353693962097 0.0\n",
      "0.19831378757953644 0.0\n",
      "0.21989169716835022 1.0\n",
      "0.5292056798934937 1.0\n",
      "0.028023462742567062 0.0\n",
      "0.2171529084444046 0.0\n",
      "0.05382155627012253 0.0\n",
      "0.05160113424062729 0.0\n",
      "0.9287819266319275 1.0\n",
      "0.6308333873748779 1.0\n",
      "0.5648472905158997 1.0\n",
      "0.8431414365768433 1.0\n",
      "TRAIN[steps=2400] loss=0.570886 acc=0.688 P=0.667 R=0.750 F1=0.705882\n",
      "DEV[steps=2400] loss=0.009231 acc=0.686 P=0.677 R=0.687 F1=0.681587 \n",
      "EPOCH:  4\n",
      "model sim and label tuples:\n",
      "0.03496580943465233 1.0\n",
      "0.012851182371377945 0.0\n",
      "0.6079250574111938 1.0\n",
      "0.011327063664793968 0.0\n",
      "0.2124844193458557 0.0\n",
      "0.015519428066909313 0.0\n",
      "0.7590817809104919 1.0\n",
      "0.4957422912120819 0.0\n",
      "0.12512314319610596 0.0\n",
      "0.051756300032138824 0.0\n",
      "0.3466734290122986 0.0\n",
      "0.4236099123954773 1.0\n",
      "0.34792041778564453 0.0\n",
      "0.44340935349464417 0.0\n",
      "0.2952200770378113 1.0\n",
      "0.1780773103237152 0.0\n",
      "0.43828508257865906 0.0\n",
      "0.9556969404220581 1.0\n",
      "0.31588444113731384 0.0\n",
      "0.2696992754936218 1.0\n",
      "0.6942213773727417 0.0\n",
      "0.08592046052217484 0.0\n",
      "0.9290058612823486 1.0\n",
      "0.06656897068023682 0.0\n",
      "0.6036979556083679 1.0\n",
      "0.7217069864273071 0.0\n",
      "0.5261493921279907 1.0\n",
      "0.8000696897506714 1.0\n",
      "0.27200445532798767 0.0\n",
      "0.3519396483898163 0.0\n",
      "0.052816394716501236 0.0\n",
      "0.048099469393491745 0.0\n",
      "0.2546011209487915 0.0\n",
      "0.6617752909660339 1.0\n",
      "0.15298014879226685 0.0\n",
      "0.9845060706138611 1.0\n",
      "0.8619062900543213 1.0\n",
      "0.2955956757068634 0.0\n",
      "0.23359927535057068 0.0\n",
      "0.7618197798728943 0.0\n",
      "0.0523202158510685 0.0\n",
      "0.6038020253181458 1.0\n",
      "0.00925888866186142 0.0\n",
      "0.635086178779602 1.0\n",
      "0.7916663885116577 1.0\n",
      "0.5017259120941162 1.0\n",
      "0.06881530582904816 0.0\n",
      "0.626226544380188 0.0\n",
      "0.7650353312492371 1.0\n",
      "0.3161744773387909 0.0\n",
      "0.3907855153083801 0.0\n",
      "0.6153029799461365 1.0\n",
      "0.26535966992378235 1.0\n",
      "0.4276183545589447 0.0\n",
      "0.5875064730644226 0.0\n",
      "0.4968351125717163 1.0\n",
      "0.5693985819816589 1.0\n",
      "0.449674129486084 0.0\n",
      "0.5384870171546936 0.0\n",
      "0.13304348289966583 0.0\n",
      "0.10631561279296875 0.0\n",
      "0.009227309376001358 0.0\n",
      "0.970084547996521 1.0\n",
      "0.2032647728919983 0.0\n",
      "TRAIN[steps=2500] loss=0.466677 acc=0.812 P=0.731 R=0.792 F1=0.760000\n",
      "DEV[steps=2500] loss=0.009488 acc=0.684 P=0.676 R=0.677 F1=0.676772 \n",
      "model sim and label tuples:\n",
      "0.06386857479810715 0.0\n",
      "0.9137282967567444 1.0\n",
      "0.028563430532813072 0.0\n",
      "0.9275694489479065 1.0\n",
      "0.34962114691734314 0.0\n",
      "0.8383843302726746 0.0\n",
      "0.8441254496574402 1.0\n",
      "0.6692842245101929 0.0\n",
      "0.735687792301178 1.0\n",
      "0.3123181462287903 0.0\n",
      "0.11797773838043213 0.0\n",
      "0.7276432514190674 1.0\n",
      "0.9419758915901184 0.0\n",
      "0.9685262441635132 1.0\n",
      "0.63417649269104 1.0\n",
      "0.03187919780611992 0.0\n",
      "0.42965030670166016 0.0\n",
      "0.9976135492324829 1.0\n",
      "0.7451432347297668 0.0\n",
      "0.5724747180938721 0.0\n",
      "0.007205406669527292 0.0\n",
      "0.9549434185028076 1.0\n",
      "0.15375512838363647 0.0\n",
      "0.2954562306404114 1.0\n",
      "0.10393659770488739 0.0\n",
      "0.28008610010147095 0.0\n",
      "0.9968900084495544 1.0\n",
      "0.03710934892296791 0.0\n",
      "0.8230762481689453 1.0\n",
      "0.10043806582689285 0.0\n",
      "0.00903069507330656 0.0\n",
      "0.05119447037577629 0.0\n",
      "0.0909072756767273 0.0\n",
      "0.5920435190200806 0.0\n",
      "0.18316085636615753 0.0\n",
      "0.2879243791103363 0.0\n",
      "0.06764541566371918 0.0\n",
      "0.9540023803710938 1.0\n",
      "0.6062273383140564 0.0\n",
      "0.885210394859314 1.0\n",
      "0.3732399642467499 0.0\n",
      "0.4448563754558563 0.0\n",
      "0.9027794599533081 1.0\n",
      "0.9927606582641602 1.0\n",
      "0.18000838160514832 0.0\n",
      "0.5015212297439575 1.0\n",
      "0.09104476124048233 0.0\n",
      "0.9548175930976868 1.0\n",
      "0.925940215587616 0.0\n",
      "0.5656864643096924 0.0\n",
      "0.5769610404968262 1.0\n",
      "0.925504207611084 0.0\n",
      "0.13580909371376038 0.0\n",
      "0.4543662965297699 1.0\n",
      "0.6962376832962036 1.0\n",
      "0.1959903985261917 0.0\n",
      "0.07725321501493454 0.0\n",
      "0.010599503293633461 0.0\n",
      "0.7900453805923462 1.0\n",
      "0.1822046935558319 1.0\n",
      "0.574077844619751 1.0\n",
      "0.9185067415237427 1.0\n",
      "0.41780897974967957 1.0\n",
      "0.988878071308136 1.0\n",
      "TRAIN[steps=2600] loss=0.467107 acc=0.781 P=0.697 R=0.852 F1=0.766667\n",
      "DEV[steps=2600] loss=0.010408 acc=0.676 P=0.688 R=0.617 F1=0.650519 \n",
      "model sim and label tuples:\n",
      "0.8489976525306702 1.0\n",
      "0.4581282138824463 0.0\n",
      "0.16465148329734802 0.0\n",
      "0.5987123847007751 0.0\n",
      "0.007449842989444733 0.0\n",
      "0.5092974901199341 1.0\n",
      "0.9002061486244202 1.0\n",
      "0.5335724353790283 0.0\n",
      "0.6727959513664246 1.0\n",
      "0.8957977294921875 1.0\n",
      "0.5748913288116455 1.0\n",
      "0.06197499483823776 0.0\n",
      "0.24395467340946198 0.0\n",
      "0.6953310966491699 0.0\n",
      "0.75172358751297 1.0\n",
      "0.08000651001930237 0.0\n",
      "0.9335098266601562 1.0\n",
      "0.6997785568237305 1.0\n",
      "0.6072460412979126 0.0\n",
      "0.15914519131183624 0.0\n",
      "0.9089276790618896 1.0\n",
      "0.6288253664970398 0.0\n",
      "0.19081613421440125 0.0\n",
      "0.8243474960327148 1.0\n",
      "0.9397457242012024 1.0\n",
      "0.04673971235752106 0.0\n",
      "0.6105670928955078 1.0\n",
      "0.0679251030087471 0.0\n",
      "0.990293025970459 1.0\n",
      "0.8093950748443604 1.0\n",
      "0.9604241847991943 1.0\n",
      "0.8557987809181213 1.0\n",
      "0.9277184009552002 1.0\n",
      "0.97648686170578 1.0\n",
      "0.996000349521637 1.0\n",
      "0.9923809170722961 1.0\n",
      "0.429534912109375 0.0\n",
      "0.7503530383110046 0.0\n",
      "0.20459644496440887 0.0\n",
      "0.32227763533592224 1.0\n",
      "0.9452675580978394 1.0\n",
      "0.39872878789901733 0.0\n",
      "0.5164234638214111 1.0\n",
      "0.9311431646347046 1.0\n",
      "0.8200693130493164 1.0\n",
      "0.9924958944320679 1.0\n",
      "0.023494822904467583 0.0\n",
      "0.5035924315452576 1.0\n",
      "0.809313952922821 0.0\n",
      "0.8999583125114441 1.0\n",
      "0.3081722557544708 1.0\n",
      "0.6616562008857727 0.0\n",
      "0.5293708443641663 1.0\n",
      "0.34394049644470215 1.0\n",
      "0.01711597479879856 0.0\n",
      "0.7978307604789734 1.0\n",
      "0.06829465925693512 0.0\n",
      "0.04619092494249344 0.0\n",
      "0.9116236567497253 1.0\n",
      "0.566257655620575 1.0\n",
      "0.22122657299041748 0.0\n",
      "0.9015579223632812 0.0\n",
      "0.5831269025802612 1.0\n",
      "0.8416793346405029 1.0\n",
      "TRAIN[steps=2700] loss=0.408736 acc=0.812 P=0.791 R=0.919 F1=0.850000\n",
      "DEV[steps=2700] loss=0.010331 acc=0.677 P=0.655 R=0.718 F1=0.684792 \n",
      "model sim and label tuples:\n",
      "0.42486491799354553 0.0\n",
      "0.37479260563850403 1.0\n",
      "0.7707141041755676 1.0\n",
      "0.7257025241851807 1.0\n",
      "0.5081754326820374 1.0\n",
      "0.3896474242210388 1.0\n",
      "0.9968585968017578 1.0\n",
      "0.6154391765594482 1.0\n",
      "0.8989235162734985 1.0\n",
      "0.6965308785438538 1.0\n",
      "0.8456166982650757 1.0\n",
      "0.8751386404037476 1.0\n",
      "0.49655163288116455 1.0\n",
      "0.7086915373802185 1.0\n",
      "0.27645108103752136 0.0\n",
      "0.24752593040466309 1.0\n",
      "0.023178547620773315 0.0\n",
      "0.6507528424263 0.0\n",
      "0.01147066242992878 0.0\n",
      "0.14467254281044006 0.0\n",
      "0.024396734312176704 0.0\n",
      "0.06370958685874939 0.0\n",
      "0.998314619064331 1.0\n",
      "0.011402793228626251 0.0\n",
      "0.3110732436180115 1.0\n",
      "0.9983844757080078 1.0\n",
      "0.998548686504364 1.0\n",
      "0.010964270681142807 0.0\n",
      "0.27597466111183167 0.0\n",
      "0.7688454985618591 1.0\n",
      "0.5817506909370422 0.0\n",
      "0.646604597568512 0.0\n",
      "0.44219914078712463 1.0\n",
      "0.0464613139629364 1.0\n",
      "0.6013251543045044 1.0\n",
      "0.1730075180530548 1.0\n",
      "0.2404792755842209 0.0\n",
      "0.8984953761100769 1.0\n",
      "0.6796112656593323 1.0\n",
      "0.7602127194404602 0.0\n",
      "0.2809571921825409 0.0\n",
      "0.2359851598739624 0.0\n",
      "0.45310139656066895 1.0\n",
      "0.03163263946771622 0.0\n",
      "0.4688454866409302 0.0\n",
      "0.02452758140861988 0.0\n",
      "0.11590901762247086 1.0\n",
      "0.8480769395828247 1.0\n",
      "0.8238585591316223 1.0\n",
      "0.6407256722450256 1.0\n",
      "0.8675007224082947 1.0\n",
      "0.15254950523376465 0.0\n",
      "0.7636393308639526 1.0\n",
      "0.8731021881103516 1.0\n",
      "0.9825881719589233 1.0\n",
      "0.4694480299949646 0.0\n",
      "0.5170031189918518 0.0\n",
      "0.41086316108703613 0.0\n",
      "0.5418941974639893 1.0\n",
      "0.975883960723877 1.0\n",
      "0.6155715584754944 0.0\n",
      "0.3367946445941925 0.0\n",
      "0.9974555373191833 1.0\n",
      "0.17850086092948914 0.0\n",
      "TRAIN[steps=2800] loss=0.484237 acc=0.766 P=0.824 R=0.757 F1=0.788732\n",
      "DEV[steps=2800] loss=0.009844 acc=0.687 P=0.665 R=0.725 F1=0.693922 \n",
      "model sim and label tuples:\n",
      "0.9822731614112854 1.0\n",
      "0.9848610162734985 1.0\n",
      "0.40209972858428955 0.0\n",
      "0.31629693508148193 0.0\n",
      "0.6177691221237183 0.0\n",
      "0.748186469078064 1.0\n",
      "0.4215206503868103 1.0\n",
      "0.99891197681427 1.0\n",
      "0.06605816632509232 1.0\n",
      "0.39136338233947754 0.0\n",
      "0.9190611839294434 1.0\n",
      "0.6449854373931885 1.0\n",
      "0.323824942111969 0.0\n",
      "0.09109830856323242 0.0\n",
      "0.7910813689231873 1.0\n",
      "0.6033952236175537 0.0\n",
      "0.8375951051712036 1.0\n",
      "0.2586936950683594 1.0\n",
      "0.11304675042629242 0.0\n",
      "0.021076403558254242 0.0\n",
      "0.6813596487045288 1.0\n",
      "0.12570582330226898 0.0\n",
      "0.015638040378689766 0.0\n",
      "0.2740944027900696 0.0\n",
      "0.033709410578012466 0.0\n",
      "0.2193434089422226 0.0\n",
      "0.03375517949461937 0.0\n",
      "0.8295297622680664 1.0\n",
      "0.9319681525230408 1.0\n",
      "0.0990963727235794 0.0\n",
      "0.008561973460018635 0.0\n",
      "0.8036670684814453 1.0\n",
      "0.14438089728355408 0.0\n",
      "0.7666065692901611 1.0\n",
      "0.5776367783546448 1.0\n",
      "0.9186059832572937 1.0\n",
      "0.48330211639404297 1.0\n",
      "0.3404211103916168 1.0\n",
      "0.03914041817188263 0.0\n",
      "0.0016348042991012335 0.0\n",
      "0.01992729678750038 0.0\n",
      "0.32464951276779175 0.0\n",
      "0.2952827513217926 1.0\n",
      "0.5963590145111084 0.0\n",
      "0.8644946217536926 1.0\n",
      "0.010477674193680286 0.0\n",
      "0.8890455365180969 1.0\n",
      "0.3297814428806305 0.0\n",
      "0.44663456082344055 1.0\n",
      "0.3114403188228607 1.0\n",
      "0.988520622253418 1.0\n",
      "0.618558406829834 1.0\n",
      "0.0023861024528741837 0.0\n",
      "0.0026421439833939075 0.0\n",
      "0.8718608021736145 1.0\n",
      "0.019876370206475258 0.0\n",
      "0.06143612042069435 0.0\n",
      "0.33594730496406555 1.0\n",
      "0.04719250649213791 0.0\n",
      "0.41763922572135925 0.0\n",
      "0.07924815267324448 0.0\n",
      "0.24060828983783722 0.0\n",
      "0.8345150351524353 1.0\n",
      "0.10474226623773575 0.0\n",
      "TRAIN[steps=2900] loss=0.359243 acc=0.828 P=0.880 R=0.733 F1=0.800000\n",
      "DEV[steps=2900] loss=0.010432 acc=0.686 P=0.680 R=0.674 F1=0.677031 \n",
      "model sim and label tuples:\n",
      "0.050446800887584686 0.0\n",
      "0.9691809415817261 1.0\n",
      "0.14168588817119598 0.0\n",
      "0.9977582693099976 1.0\n",
      "0.14297887682914734 0.0\n",
      "0.9962456822395325 1.0\n",
      "0.3262996971607208 0.0\n",
      "0.908467173576355 1.0\n",
      "0.664686381816864 1.0\n",
      "0.8987084031105042 1.0\n",
      "0.2734575867652893 1.0\n",
      "0.06816039234399796 0.0\n",
      "0.9017283916473389 1.0\n",
      "0.04225626215338707 0.0\n",
      "0.21494871377944946 0.0\n",
      "0.4665033519268036 1.0\n",
      "0.40281084179878235 1.0\n",
      "0.9655479192733765 1.0\n",
      "0.8753060102462769 1.0\n",
      "0.578619122505188 1.0\n",
      "0.01682453602552414 0.0\n",
      "0.30728134512901306 0.0\n",
      "0.6113759279251099 1.0\n",
      "0.9900596141815186 1.0\n",
      "0.046804822981357574 0.0\n",
      "0.8217283487319946 1.0\n",
      "0.05522758886218071 0.0\n",
      "0.5586287975311279 1.0\n",
      "0.7683829665184021 0.0\n",
      "0.3961888551712036 1.0\n",
      "0.6033220291137695 0.0\n",
      "0.6162741780281067 0.0\n",
      "0.5424321293830872 1.0\n",
      "0.18136481940746307 1.0\n",
      "0.04499025642871857 0.0\n",
      "0.5361849665641785 1.0\n",
      "0.5147594809532166 0.0\n",
      "0.42021840810775757 1.0\n",
      "0.2826996147632599 0.0\n",
      "0.872113823890686 1.0\n",
      "0.9888719916343689 1.0\n",
      "0.7034286260604858 1.0\n",
      "0.572182834148407 1.0\n",
      "0.7599905133247375 0.0\n",
      "0.7218238711357117 0.0\n",
      "0.8908263444900513 1.0\n",
      "0.5417108535766602 1.0\n",
      "0.9352097511291504 1.0\n",
      "0.5841811895370483 0.0\n",
      "0.7400544285774231 0.0\n",
      "0.24872586131095886 0.0\n",
      "0.21442605555057526 0.0\n",
      "0.09359879046678543 0.0\n",
      "0.8673598766326904 1.0\n",
      "0.0857699066400528 0.0\n",
      "0.4614564776420593 1.0\n",
      "0.06669041514396667 0.0\n",
      "0.013954673893749714 0.0\n",
      "0.0012736861826851964 0.0\n",
      "0.9876642823219299 1.0\n",
      "0.17004039883613586 0.0\n",
      "0.29142314195632935 0.0\n",
      "0.7102806568145752 1.0\n",
      "0.8584163784980774 0.0\n",
      "TRAIN[steps=3000] loss=0.434953 acc=0.750 P=0.743 R=0.788 F1=0.764706\n",
      "DEV[steps=3000] loss=0.010389 acc=0.688 P=0.691 R=0.653 F1=0.671305 \n",
      "model sim and label tuples:\n",
      "0.5164081454277039 1.0\n",
      "0.2939453125 1.0\n",
      "0.798599362373352 1.0\n",
      "0.3050383925437927 0.0\n",
      "0.6250813603401184 1.0\n",
      "0.17727363109588623 0.0\n",
      "0.719643771648407 1.0\n",
      "0.15172001719474792 0.0\n",
      "0.43687117099761963 1.0\n",
      "0.6165502071380615 1.0\n",
      "0.7214281558990479 0.0\n",
      "0.37585264444351196 0.0\n",
      "0.30989009141921997 0.0\n",
      "0.4119871258735657 1.0\n",
      "0.5042173862457275 0.0\n",
      "0.8899418711662292 1.0\n",
      "0.31842318177223206 0.0\n",
      "0.9453112483024597 1.0\n",
      "0.8220394253730774 1.0\n",
      "0.9487613439559937 1.0\n",
      "0.6603067517280579 0.0\n",
      "0.032561540603637695 0.0\n",
      "0.00390922324731946 0.0\n",
      "0.9567655920982361 1.0\n",
      "0.027169782668352127 0.0\n",
      "0.033665239810943604 0.0\n",
      "0.9230070114135742 1.0\n",
      "0.6105231046676636 1.0\n",
      "0.5703874826431274 1.0\n",
      "0.262349396944046 0.0\n",
      "0.8321224451065063 1.0\n",
      "0.9988009929656982 1.0\n",
      "0.8823658227920532 0.0\n",
      "0.3811494708061218 0.0\n",
      "0.9917984008789062 1.0\n",
      "0.9665106534957886 1.0\n",
      "0.18144382536411285 1.0\n",
      "0.06882541626691818 0.0\n",
      "0.289394736289978 0.0\n",
      "0.9953435063362122 1.0\n",
      "0.9984606504440308 1.0\n",
      "0.9986962676048279 1.0\n",
      "0.0038321614265441895 0.0\n",
      "0.38369855284690857 0.0\n",
      "0.7346683740615845 1.0\n",
      "0.3768576979637146 0.0\n",
      "0.001122550806030631 0.0\n",
      "0.5726715922355652 1.0\n",
      "0.9710839986801147 1.0\n",
      "0.6280317902565002 0.0\n",
      "0.9155896902084351 1.0\n",
      "0.734620988368988 1.0\n",
      "0.012339632958173752 0.0\n",
      "0.5970767140388489 1.0\n",
      "0.019986441358923912 0.0\n",
      "0.19496741890907288 0.0\n",
      "0.14707468450069427 0.0\n",
      "0.471575528383255 1.0\n",
      "0.512789785861969 1.0\n",
      "0.7773731350898743 0.0\n",
      "0.9753268957138062 1.0\n",
      "0.5232874155044556 1.0\n",
      "0.789286196231842 1.0\n",
      "0.5453521013259888 0.0\n",
      "TRAIN[steps=3100] loss=0.404384 acc=0.812 P=0.811 R=0.857 F1=0.833333\n",
      "DEV[steps=3100] loss=0.009960 acc=0.700 P=0.690 R=0.700 F1=0.694900 \n",
      "EPOCH:  5\n",
      "model sim and label tuples:\n",
      "0.05778219923377037 0.0\n",
      "0.0021331333555281162 0.0\n",
      "0.9991568326950073 1.0\n",
      "0.6884901523590088 0.0\n",
      "0.04273035004734993 0.0\n",
      "0.8170495629310608 1.0\n",
      "0.028474433347582817 0.0\n",
      "0.6795991063117981 1.0\n",
      "0.021441232413053513 0.0\n",
      "0.036257367581129074 0.0\n",
      "0.0015623910585418344 0.0\n",
      "0.9970159530639648 1.0\n",
      "0.7340110540390015 1.0\n",
      "0.5804163217544556 1.0\n",
      "0.012842281721532345 0.0\n",
      "0.8120517134666443 1.0\n",
      "0.0003996841551270336 0.0\n",
      "0.6812240481376648 0.0\n",
      "0.007927940227091312 0.0\n",
      "0.20001205801963806 1.0\n",
      "0.012236814014613628 0.0\n",
      "0.46695008873939514 0.0\n",
      "0.0005004233680665493 0.0\n",
      "0.0003520440368447453 0.0\n",
      "0.3538835942745209 1.0\n",
      "0.2216476947069168 0.0\n",
      "0.007619064301252365 0.0\n",
      "0.019609419628977776 0.0\n",
      "0.006985271815210581 0.0\n",
      "0.1426287293434143 0.0\n",
      "0.05729156360030174 0.0\n",
      "0.026386775076389313 0.0\n",
      "0.9699787497520447 1.0\n",
      "0.0014117325190454721 0.0\n",
      "0.1447761356830597 0.0\n",
      "0.015240460634231567 0.0\n",
      "0.9986093044281006 1.0\n",
      "0.0875491350889206 0.0\n",
      "0.0031388166826218367 0.0\n",
      "0.9964929223060608 1.0\n",
      "0.762901782989502 0.0\n",
      "0.37394681572914124 0.0\n",
      "0.016651613637804985 0.0\n",
      "0.5877931714057922 0.0\n",
      "0.002072244184091687 0.0\n",
      "0.0444757305085659 0.0\n",
      "0.9155070781707764 1.0\n",
      "0.9575151205062866 1.0\n",
      "0.6242406368255615 1.0\n",
      "0.016824236139655113 0.0\n",
      "0.014519310556352139 0.0\n",
      "0.9925888776779175 1.0\n",
      "0.9761744737625122 1.0\n",
      "0.0002995597606059164 0.0\n",
      "0.8767769932746887 1.0\n",
      "0.17604924738407135 0.0\n",
      "0.9996919631958008 1.0\n",
      "0.000765583012253046 0.0\n",
      "0.9996819496154785 1.0\n",
      "0.026446430012583733 0.0\n",
      "0.08441755920648575 0.0\n",
      "0.22635842859745026 0.0\n",
      "0.015579566359519958 0.0\n",
      "0.004727527499198914 0.0\n",
      "TRAIN[steps=3200] loss=0.195944 acc=0.906 P=0.818 R=0.900 F1=0.857143\n",
      "DEV[steps=3200] loss=0.012535 acc=0.686 P=0.703 R=0.620 F1=0.658970 \n",
      "model sim and label tuples:\n",
      "0.8425770998001099 0.0\n",
      "0.36063480377197266 0.0\n",
      "0.8440581560134888 1.0\n",
      "0.01850326731801033 0.0\n",
      "0.23999229073524475 0.0\n",
      "0.7315775752067566 1.0\n",
      "0.9997270703315735 1.0\n",
      "0.5393035411834717 0.0\n",
      "0.06795774400234222 0.0\n",
      "0.1468716412782669 0.0\n",
      "0.1566147804260254 1.0\n",
      "0.8610236048698425 0.0\n",
      "0.0744306668639183 0.0\n",
      "0.9738469123840332 1.0\n",
      "0.04988042265176773 0.0\n",
      "0.9618735313415527 1.0\n",
      "0.9997268319129944 1.0\n",
      "0.9845527410507202 0.0\n",
      "0.0002735296729952097 0.0\n",
      "0.3831312358379364 0.0\n",
      "0.9906933903694153 1.0\n",
      "0.22197017073631287 0.0\n",
      "0.9993422627449036 1.0\n",
      "0.5110083818435669 1.0\n",
      "0.008790726773440838 0.0\n",
      "0.27699580788612366 0.0\n",
      "0.9381895661354065 1.0\n",
      "0.009976360946893692 0.0\n",
      "0.9867621064186096 1.0\n",
      "0.02784419059753418 0.0\n",
      "0.744368851184845 1.0\n",
      "0.012788629159331322 0.0\n",
      "0.9980637431144714 1.0\n",
      "0.07543621212244034 0.0\n",
      "0.93356853723526 1.0\n",
      "0.5841653943061829 0.0\n",
      "0.13474170863628387 0.0\n",
      "0.22031767666339874 0.0\n",
      "0.8063778877258301 1.0\n",
      "0.962821900844574 1.0\n",
      "0.18679480254650116 0.0\n",
      "0.08451222628355026 0.0\n",
      "0.40082302689552307 1.0\n",
      "0.964743971824646 1.0\n",
      "0.690217912197113 1.0\n",
      "0.7792842984199524 0.0\n",
      "0.005141164641827345 0.0\n",
      "0.049834106117486954 0.0\n",
      "0.9400199055671692 1.0\n",
      "0.09360646456480026 0.0\n",
      "0.026813063770532608 0.0\n",
      "0.0021439173724502325 0.0\n",
      "0.9993189573287964 1.0\n",
      "0.11823936551809311 0.0\n",
      "0.9268589019775391 1.0\n",
      "0.9517604112625122 1.0\n",
      "0.8697536587715149 1.0\n",
      "0.9186014533042908 1.0\n",
      "0.9912973642349243 1.0\n",
      "0.9982287287712097 1.0\n",
      "0.006627251394093037 0.0\n",
      "0.0011545781744644046 0.0\n",
      "0.9988813996315002 1.0\n",
      "0.09110122919082642 0.0\n",
      "TRAIN[steps=3300] loss=0.313707 acc=0.875 P=0.818 R=0.931 F1=0.870968\n",
      "DEV[steps=3300] loss=0.011764 acc=0.680 P=0.653 R=0.736 F1=0.692167 \n",
      "model sim and label tuples:\n",
      "0.9297102093696594 1.0\n",
      "0.9995037317276001 1.0\n",
      "0.0006094902637414634 0.0\n",
      "0.7127729058265686 1.0\n",
      "0.9917873740196228 1.0\n",
      "0.8872624635696411 1.0\n",
      "0.22567012906074524 0.0\n",
      "0.48832443356513977 1.0\n",
      "0.6017223596572876 0.0\n",
      "0.5507845878601074 0.0\n",
      "0.03510167449712753 0.0\n",
      "0.0021073261741548777 0.0\n",
      "0.03240148723125458 0.0\n",
      "0.9250920414924622 1.0\n",
      "0.5576476454734802 1.0\n",
      "0.006352210883051157 0.0\n",
      "0.3114078938961029 1.0\n",
      "0.0446985587477684 0.0\n",
      "0.09906671196222305 0.0\n",
      "0.7278554439544678 0.0\n",
      "0.2948470413684845 1.0\n",
      "0.07675797492265701 0.0\n",
      "0.0066235545091331005 0.0\n",
      "0.785354733467102 1.0\n",
      "0.8267714381217957 1.0\n",
      "0.0007093894528225064 0.0\n",
      "0.3539183437824249 0.0\n",
      "0.2287469208240509 0.0\n",
      "0.9019562602043152 1.0\n",
      "0.3428956866264343 1.0\n",
      "0.5679811835289001 0.0\n",
      "0.035497087985277176 0.0\n",
      "0.7939674258232117 1.0\n",
      "0.9757117033004761 1.0\n",
      "0.714785635471344 1.0\n",
      "0.43255704641342163 1.0\n",
      "0.6644653677940369 1.0\n",
      "0.006197634618729353 0.0\n",
      "0.0012924099573865533 0.0\n",
      "0.03307003155350685 0.0\n",
      "0.9892991781234741 1.0\n",
      "0.7421350479125977 1.0\n",
      "0.93959641456604 1.0\n",
      "0.039400070905685425 0.0\n",
      "0.0003800207341555506 0.0\n",
      "0.33829420804977417 0.0\n",
      "0.8972022533416748 1.0\n",
      "0.532543957233429 1.0\n",
      "0.03296748176217079 0.0\n",
      "0.13781128823757172 0.0\n",
      "0.3961182236671448 0.0\n",
      "0.7603625059127808 1.0\n",
      "0.09587881714105606 0.0\n",
      "0.9831005334854126 1.0\n",
      "0.021063249558210373 0.0\n",
      "0.10840531438589096 1.0\n",
      "0.0008467173902317882 0.0\n",
      "0.9470335245132446 1.0\n",
      "0.07490658015012741 0.0\n",
      "0.15823963284492493 1.0\n",
      "0.010464838705956936 0.0\n",
      "0.988991379737854 1.0\n",
      "0.9975666999816895 1.0\n",
      "0.9631809592247009 1.0\n",
      "TRAIN[steps=3400] loss=0.310605 acc=0.844 P=0.867 R=0.812 F1=0.838710\n",
      "DEV[steps=3400] loss=0.011805 acc=0.679 P=0.669 R=0.679 F1=0.674123 \n",
      "model sim and label tuples:\n",
      "0.005667069461196661 0.0\n",
      "0.12648577988147736 0.0\n",
      "0.4850142300128937 0.0\n",
      "0.960849940776825 1.0\n",
      "0.36518576741218567 0.0\n",
      "0.9968593120574951 1.0\n",
      "0.3479584753513336 1.0\n",
      "0.8375146389007568 1.0\n",
      "0.9457350969314575 1.0\n",
      "0.6911571621894836 1.0\n",
      "0.353180468082428 0.0\n",
      "0.7576450109481812 0.0\n",
      "0.10721635818481445 0.0\n",
      "0.9639129638671875 0.0\n",
      "0.9881272912025452 1.0\n",
      "0.06013910099864006 0.0\n",
      "0.9296985864639282 0.0\n",
      "0.014119108207523823 0.0\n",
      "0.06601294875144958 0.0\n",
      "0.559467613697052 0.0\n",
      "0.8669745922088623 1.0\n",
      "0.9625005125999451 1.0\n",
      "0.9470601677894592 1.0\n",
      "0.9970998764038086 1.0\n",
      "0.9903234243392944 1.0\n",
      "0.03019885905086994 0.0\n",
      "0.2041999250650406 0.0\n",
      "0.8446296453475952 1.0\n",
      "0.45968717336654663 0.0\n",
      "0.9902907013893127 1.0\n",
      "0.9995406866073608 1.0\n",
      "0.049795836210250854 0.0\n",
      "0.32268571853637695 1.0\n",
      "0.22115333378314972 0.0\n",
      "0.9894848465919495 1.0\n",
      "0.44451603293418884 1.0\n",
      "0.5604715347290039 1.0\n",
      "0.6807434558868408 1.0\n",
      "0.9253270626068115 1.0\n",
      "0.09006768465042114 1.0\n",
      "0.22137923538684845 0.0\n",
      "0.6140393614768982 1.0\n",
      "0.0004426318046171218 0.0\n",
      "0.2046239972114563 0.0\n",
      "0.13291792571544647 0.0\n",
      "0.3157287836074829 0.0\n",
      "0.9856224656105042 1.0\n",
      "0.8952381610870361 1.0\n",
      "0.5042675137519836 0.0\n",
      "0.008207838051021099 0.0\n",
      "0.9937143921852112 1.0\n",
      "0.9872629046440125 1.0\n",
      "0.4095984101295471 0.0\n",
      "0.6766199469566345 0.0\n",
      "0.32084527611732483 0.0\n",
      "0.9748272895812988 1.0\n",
      "0.8861153721809387 0.0\n",
      "0.0027463380247354507 0.0\n",
      "0.001451976946555078 0.0\n",
      "0.11369450390338898 1.0\n",
      "0.8864190578460693 1.0\n",
      "0.046699561178684235 0.0\n",
      "0.9440841674804688 1.0\n",
      "0.9391803741455078 1.0\n",
      "TRAIN[steps=3500] loss=0.436480 acc=0.797 P=0.771 R=0.844 F1=0.805970\n",
      "DEV[steps=3500] loss=0.011801 acc=0.684 P=0.653 R=0.755 F1=0.700200 *\n",
      "model sim and label tuples:\n",
      "0.0638965591788292 0.0\n",
      "0.001811252092011273 0.0\n",
      "0.9900839328765869 1.0\n",
      "0.8886044025421143 1.0\n",
      "0.5335913896560669 1.0\n",
      "0.003348149126395583 0.0\n",
      "0.21807052195072174 0.0\n",
      "0.2775399088859558 1.0\n",
      "0.6439247131347656 1.0\n",
      "0.652887761592865 1.0\n",
      "0.18510450422763824 1.0\n",
      "0.2779960036277771 0.0\n",
      "0.32722383737564087 1.0\n",
      "0.3226126432418823 1.0\n",
      "0.1551750898361206 0.0\n",
      "0.9655165076255798 1.0\n",
      "0.0018211484421044588 0.0\n",
      "0.9934316277503967 1.0\n",
      "0.037752240896224976 0.0\n",
      "0.9826593399047852 1.0\n",
      "0.11202804744243622 0.0\n",
      "0.9962393045425415 1.0\n",
      "0.0029504643753170967 0.0\n",
      "0.015071623027324677 0.0\n",
      "0.9820021390914917 1.0\n",
      "0.587634265422821 0.0\n",
      "0.33018264174461365 0.0\n",
      "0.15188980102539062 0.0\n",
      "0.00027338627842254937 0.0\n",
      "0.12677718698978424 0.0\n",
      "0.8539736866950989 1.0\n",
      "0.5697323679924011 1.0\n",
      "0.9492907524108887 1.0\n",
      "0.5831741690635681 0.0\n",
      "0.05903133004903793 0.0\n",
      "0.8151167631149292 1.0\n",
      "0.8919098377227783 1.0\n",
      "0.2319350689649582 0.0\n",
      "0.8540024161338806 1.0\n",
      "0.12570416927337646 0.0\n",
      "0.5107619762420654 0.0\n",
      "0.694513738155365 0.0\n",
      "0.2471945434808731 0.0\n",
      "0.00029693273245356977 0.0\n",
      "0.8447957634925842 1.0\n",
      "0.6200915575027466 0.0\n",
      "0.9990252256393433 1.0\n",
      "0.726144552230835 1.0\n",
      "0.7970594763755798 1.0\n",
      "0.537740170955658 0.0\n",
      "0.6930726766586304 1.0\n",
      "0.6339931488037109 1.0\n",
      "0.3934728503227234 1.0\n",
      "0.9869018793106079 1.0\n",
      "0.6382389068603516 0.0\n",
      "0.03681914135813713 0.0\n",
      "0.15434126555919647 0.0\n",
      "0.23454990983009338 1.0\n",
      "0.0004943656967952847 0.0\n",
      "0.404691219329834 1.0\n",
      "0.8679602146148682 1.0\n",
      "0.04747433215379715 1.0\n",
      "0.050524868071079254 0.0\n",
      "0.9915429353713989 1.0\n",
      "TRAIN[steps=3600] loss=0.395562 acc=0.766 P=0.781 R=0.758 F1=0.769231\n",
      "DEV[steps=3600] loss=0.011613 acc=0.694 P=0.676 R=0.717 F1=0.695825 \n",
      "model sim and label tuples:\n",
      "0.9292302131652832 1.0\n",
      "0.000628731562756002 0.0\n",
      "0.9969558715820312 1.0\n",
      "0.9436833262443542 1.0\n",
      "0.23185621201992035 0.0\n",
      "0.012906030751764774 0.0\n",
      "0.9974168539047241 1.0\n",
      "0.29670917987823486 0.0\n",
      "0.9089881181716919 1.0\n",
      "0.21344849467277527 0.0\n",
      "0.0033350063022226095 0.0\n",
      "0.024098195135593414 0.0\n",
      "0.05824211612343788 0.0\n",
      "0.015015354380011559 0.0\n",
      "0.131972998380661 1.0\n",
      "0.9101943969726562 1.0\n",
      "0.807631254196167 1.0\n",
      "0.08769101649522781 0.0\n",
      "0.9987995624542236 1.0\n",
      "0.009765788912773132 0.0\n",
      "0.1372210681438446 1.0\n",
      "0.8952177166938782 1.0\n",
      "0.5682587027549744 0.0\n",
      "0.42469561100006104 1.0\n",
      "0.5058917999267578 1.0\n",
      "0.5277969837188721 0.0\n",
      "0.14855553209781647 0.0\n",
      "0.0993415042757988 0.0\n",
      "0.9863457083702087 1.0\n",
      "0.03452712297439575 0.0\n",
      "0.9470064640045166 1.0\n",
      "0.10910593718290329 0.0\n",
      "0.9587571024894714 1.0\n",
      "0.02905447408556938 0.0\n",
      "0.17215169966220856 0.0\n",
      "0.08987432718276978 0.0\n",
      "0.513695478439331 1.0\n",
      "0.010292298160493374 0.0\n",
      "0.96329265832901 1.0\n",
      "0.5742701292037964 0.0\n",
      "0.21568050980567932 0.0\n",
      "0.6116538643836975 1.0\n",
      "0.2993517518043518 1.0\n",
      "0.841197669506073 1.0\n",
      "0.9995204210281372 1.0\n",
      "0.3046049475669861 0.0\n",
      "0.03990255296230316 0.0\n",
      "0.3455257713794708 0.0\n",
      "0.6551125645637512 1.0\n",
      "0.9826265573501587 1.0\n",
      "0.010843480937182903 0.0\n",
      "0.007737987209111452 0.0\n",
      "0.0027753727044910192 0.0\n",
      "0.024667389690876007 0.0\n",
      "0.502133846282959 1.0\n",
      "0.3284963369369507 1.0\n",
      "0.9730080366134644 1.0\n",
      "0.744458794593811 1.0\n",
      "0.853474497795105 1.0\n",
      "0.5621242523193359 1.0\n",
      "0.9604337811470032 1.0\n",
      "0.9558059573173523 1.0\n",
      "0.9826599359512329 1.0\n",
      "0.9133086204528809 1.0\n",
      "TRAIN[steps=3700] loss=0.277405 acc=0.875 P=0.906 R=0.853 F1=0.878788\n",
      "DEV[steps=3700] loss=0.011440 acc=0.694 P=0.680 R=0.705 F1=0.692230 \n",
      "EPOCH:  6\n",
      "model sim and label tuples:\n",
      "0.9513437747955322 1.0\n",
      "0.33580079674720764 1.0\n",
      "0.00027613816200755537 0.0\n",
      "0.0003086568904109299 0.0\n",
      "0.6696447134017944 0.0\n",
      "0.0006313225603662431 0.0\n",
      "9.032361413119361e-05 0.0\n",
      "0.9997705817222595 1.0\n",
      "0.9569295644760132 1.0\n",
      "0.00022872160479892045 0.0\n",
      "0.0029643308371305466 0.0\n",
      "0.9985350370407104 1.0\n",
      "0.9742487668991089 1.0\n",
      "0.001100740977562964 0.0\n",
      "0.9843682646751404 1.0\n",
      "0.8842187523841858 1.0\n",
      "0.000544593611266464 0.0\n",
      "0.4171289801597595 0.0\n",
      "0.026512155309319496 0.0\n",
      "0.0047355531714856625 0.0\n",
      "0.9159299731254578 1.0\n",
      "0.0016078515909612179 0.0\n",
      "0.0006574941799044609 0.0\n",
      "0.8314152956008911 0.0\n",
      "0.9041285514831543 1.0\n",
      "0.34495025873184204 1.0\n",
      "0.0208896417170763 0.0\n",
      "0.00014386410475708544 0.0\n",
      "0.8799404501914978 1.0\n",
      "0.36330604553222656 1.0\n",
      "0.00022737699327990413 0.0\n",
      "0.942274808883667 1.0\n",
      "0.004735265392810106 0.0\n",
      "0.0006324978312477469 0.0\n",
      "0.0001512214948888868 0.0\n",
      "0.0063844299875199795 0.0\n",
      "0.9220759272575378 1.0\n",
      "0.002836988540366292 0.0\n",
      "0.0018368118908256292 0.0\n",
      "0.0005673322011716664 0.0\n",
      "0.0026590556371957064 0.0\n",
      "0.0975627452135086 0.0\n",
      "0.00043936914880760014 0.0\n",
      "0.5114729404449463 1.0\n",
      "0.9952337145805359 1.0\n",
      "0.9944382905960083 1.0\n",
      "0.0008203001925721765 0.0\n",
      "0.4577023684978485 1.0\n",
      "0.08882777392864227 0.0\n",
      "0.9998655319213867 1.0\n",
      "0.9999014139175415 1.0\n",
      "0.7468359470367432 1.0\n",
      "0.029235975816845894 0.0\n",
      "0.003568282350897789 0.0\n",
      "0.9980054497718811 1.0\n",
      "0.146703839302063 0.0\n",
      "0.00011046866711694747 0.0\n",
      "0.003838361008092761 0.0\n",
      "0.9992018342018127 1.0\n",
      "0.9940710663795471 1.0\n",
      "0.984941303730011 1.0\n",
      "0.9922789335250854 1.0\n",
      "0.9985882639884949 1.0\n",
      "0.0007238441030494869 0.0\n",
      "TRAIN[steps=3800] loss=0.149622 acc=0.906 P=0.923 R=0.857 F1=0.888889\n",
      "DEV[steps=3800] loss=0.015303 acc=0.693 P=0.706 R=0.636 F1=0.669040 \n",
      "model sim and label tuples:\n",
      "0.05414803326129913 0.0\n",
      "0.9998973608016968 1.0\n",
      "0.9997695088386536 1.0\n",
      "0.2683559060096741 1.0\n",
      "0.9430155754089355 1.0\n",
      "0.9228358864784241 1.0\n",
      "0.08673810958862305 0.0\n",
      "0.6628963947296143 1.0\n",
      "0.006643400061875582 0.0\n",
      "0.9323672652244568 1.0\n",
      "0.045785874128341675 0.0\n",
      "0.9633709192276001 1.0\n",
      "0.9704269170761108 1.0\n",
      "0.07451832294464111 0.0\n",
      "0.9927364587783813 1.0\n",
      "0.02150857262313366 0.0\n",
      "0.0011280348990112543 0.0\n",
      "0.988054096698761 1.0\n",
      "0.06837941706180573 0.0\n",
      "0.9997114539146423 1.0\n",
      "0.9897848963737488 1.0\n",
      "0.002759478287771344 0.0\n",
      "9.424649033462629e-05 0.0\n",
      "0.3503914475440979 0.0\n",
      "0.9956075549125671 1.0\n",
      "0.10966796427965164 0.0\n",
      "0.9672834277153015 1.0\n",
      "0.03578101098537445 0.0\n",
      "0.7558484077453613 1.0\n",
      "0.9757331013679504 1.0\n",
      "0.0003122521156910807 0.0\n",
      "0.9891210198402405 1.0\n",
      "0.3643403649330139 0.0\n",
      "0.3278277814388275 0.0\n",
      "0.1663319170475006 0.0\n",
      "0.9843881130218506 1.0\n",
      "0.35430657863616943 1.0\n",
      "0.6221896409988403 1.0\n",
      "0.9997032284736633 1.0\n",
      "0.0728803426027298 1.0\n",
      "0.06560070067644119 0.0\n",
      "0.6245551109313965 1.0\n",
      "0.6914700865745544 1.0\n",
      "0.7556939721107483 1.0\n",
      "0.992297351360321 1.0\n",
      "0.11459598690271378 0.0\n",
      "0.00016729248454794288 0.0\n",
      "0.37408626079559326 0.0\n",
      "0.10008466243743896 0.0\n",
      "0.5919731259346008 0.0\n",
      "0.959797739982605 1.0\n",
      "0.28777191042900085 1.0\n",
      "0.005194591823965311 0.0\n",
      "0.01286701112985611 0.0\n",
      "0.07019207626581192 0.0\n",
      "0.9643888473510742 1.0\n",
      "0.1914263814687729 1.0\n",
      "0.9925876259803772 1.0\n",
      "0.005080839619040489 0.0\n",
      "0.9997883439064026 1.0\n",
      "0.9971895813941956 1.0\n",
      "0.2867187261581421 0.0\n",
      "0.004821871407330036 0.0\n",
      "0.9986321330070496 1.0\n",
      "TRAIN[steps=3900] loss=0.230283 acc=0.906 P=0.968 R=0.857 F1=0.909091\n",
      "DEV[steps=3900] loss=0.014885 acc=0.680 P=0.654 R=0.730 F1=0.690176 \n",
      "model sim and label tuples:\n",
      "0.0003423025773372501 0.0\n",
      "0.9923673272132874 1.0\n",
      "0.003289916319772601 0.0\n",
      "0.0014759316109120846 0.0\n",
      "0.9930775165557861 1.0\n",
      "0.984022855758667 1.0\n",
      "0.9776197671890259 1.0\n",
      "0.008144962601363659 0.0\n",
      "0.4532698094844818 1.0\n",
      "0.01450923178344965 0.0\n",
      "0.0018821079283952713 0.0\n",
      "0.0076607041992247105 0.0\n",
      "0.6814111471176147 1.0\n",
      "0.06793418526649475 0.0\n",
      "0.00023984642757568508 0.0\n",
      "0.9933953881263733 1.0\n",
      "0.04626883566379547 0.0\n",
      "0.01408472005277872 0.0\n",
      "0.018775656819343567 0.0\n",
      "0.009884563274681568 0.0\n",
      "0.0001703788439044729 0.0\n",
      "0.012675430625677109 0.0\n",
      "0.9872676730155945 1.0\n",
      "0.00012927445641253144 0.0\n",
      "0.09670429676771164 0.0\n",
      "0.0004919223138131201 0.0\n",
      "0.27088460326194763 0.0\n",
      "0.10954981297254562 0.0\n",
      "0.9590681195259094 1.0\n",
      "0.8712947368621826 1.0\n",
      "0.9762755632400513 1.0\n",
      "0.07976293563842773 0.0\n",
      "0.0007157506188377738 0.0\n",
      "0.0016617925139144063 0.0\n",
      "0.4950489401817322 0.0\n",
      "0.041851986199617386 0.0\n",
      "0.47143474221229553 1.0\n",
      "0.0006240850780159235 0.0\n",
      "0.8053666353225708 0.0\n",
      "0.9946080446243286 1.0\n",
      "0.999346911907196 1.0\n",
      "0.8736372590065002 1.0\n",
      "0.9737076163291931 1.0\n",
      "0.895226240158081 1.0\n",
      "0.9937229752540588 1.0\n",
      "0.9973624348640442 1.0\n",
      "0.8697815537452698 1.0\n",
      "0.9647904634475708 1.0\n",
      "0.9693911671638489 1.0\n",
      "0.837180495262146 1.0\n",
      "0.393633633852005 0.0\n",
      "0.0770600363612175 0.0\n",
      "0.002060397993773222 0.0\n",
      "0.013413315638899803 0.0\n",
      "0.000681919336784631 0.0\n",
      "0.3328568637371063 0.0\n",
      "0.002891696058213711 0.0\n",
      "0.046539731323719025 0.0\n",
      "0.0050114900805056095 0.0\n",
      "0.00392310693860054 0.0\n",
      "0.7784488201141357 1.0\n",
      "0.9886465668678284 1.0\n",
      "0.9997269511222839 1.0\n",
      "0.862882673740387 1.0\n",
      "TRAIN[steps=4000] loss=0.117811 acc=0.938 P=0.926 R=0.926 F1=0.925926\n",
      "DEV[steps=4000] loss=0.014759 acc=0.697 P=0.695 R=0.675 F1=0.684906 \n",
      "model sim and label tuples:\n",
      "0.8777094483375549 1.0\n",
      "0.0008313863654620945 0.0\n",
      "0.8152618408203125 0.0\n",
      "0.9921521544456482 1.0\n",
      "0.9951962828636169 1.0\n",
      "0.9965969920158386 1.0\n",
      "0.9770395755767822 1.0\n",
      "0.7893020510673523 1.0\n",
      "0.4810293912887573 0.0\n",
      "0.767828643321991 0.0\n",
      "0.9996182918548584 1.0\n",
      "0.38937196135520935 0.0\n",
      "0.9853274822235107 1.0\n",
      "0.8863062858581543 1.0\n",
      "0.9992414712905884 1.0\n",
      "0.42289018630981445 1.0\n",
      "0.9734470844268799 1.0\n",
      "0.7405865788459778 1.0\n",
      "0.9924853444099426 1.0\n",
      "0.0003014130052179098 0.0\n",
      "0.2547670304775238 0.0\n",
      "0.9962698221206665 1.0\n",
      "0.9847175478935242 1.0\n",
      "0.00014028001169208437 0.0\n",
      "0.9934417009353638 1.0\n",
      "0.8430877327919006 1.0\n",
      "0.042459938675165176 0.0\n",
      "0.9965818524360657 1.0\n",
      "0.0011010996531695127 0.0\n",
      "0.9992228746414185 1.0\n",
      "8.843252726364881e-05 0.0\n",
      "0.9540793299674988 1.0\n",
      "0.025873880833387375 0.0\n",
      "0.0036877370439469814 0.0\n",
      "0.9829727411270142 1.0\n",
      "0.9904955625534058 1.0\n",
      "0.9954877495765686 1.0\n",
      "0.8955226540565491 1.0\n",
      "0.3302850127220154 1.0\n",
      "0.008597089909017086 0.0\n",
      "0.9990731477737427 1.0\n",
      "0.9975628852844238 1.0\n",
      "0.9536662697792053 1.0\n",
      "0.9335967898368835 1.0\n",
      "0.2900524139404297 0.0\n",
      "0.999920129776001 1.0\n",
      "0.8472411036491394 0.0\n",
      "0.7733612060546875 0.0\n",
      "9.759407112142071e-05 0.0\n",
      "0.007210948504507542 0.0\n",
      "0.9439913034439087 1.0\n",
      "0.09986626356840134 0.0\n",
      "0.0008505871519446373 0.0\n",
      "0.9998292922973633 1.0\n",
      "0.00010105994442710653 0.0\n",
      "0.22801175713539124 0.0\n",
      "0.004717572126537561 0.0\n",
      "0.0719800516963005 0.0\n",
      "0.0006828935584053397 0.0\n",
      "0.8630013465881348 1.0\n",
      "0.00130078149959445 0.0\n",
      "0.9576539397239685 1.0\n",
      "0.9107066988945007 0.0\n",
      "0.9971476197242737 1.0\n",
      "TRAIN[steps=4100] loss=0.232162 acc=0.875 P=0.850 R=0.944 F1=0.894737\n",
      "DEV[steps=4100] loss=0.015228 acc=0.688 P=0.663 R=0.738 F1=0.698342 \n",
      "model sim and label tuples:\n",
      "0.9721779823303223 1.0\n",
      "0.9998843669891357 1.0\n",
      "0.0002742302604019642 0.0\n",
      "0.07926338165998459 0.0\n",
      "0.001047503319568932 0.0\n",
      "0.006992892827838659 0.0\n",
      "0.00020383646187838167 0.0\n",
      "0.057288773357868195 0.0\n",
      "0.03230365738272667 0.0\n",
      "0.9959055185317993 1.0\n",
      "0.0005673438427038491 0.0\n",
      "0.0031987077090889215 0.0\n",
      "0.6182368397712708 1.0\n",
      "0.004170963075011969 0.0\n",
      "0.6798887848854065 1.0\n",
      "0.000234318504226394 0.0\n",
      "0.00047839959734119475 0.0\n",
      "0.02986532263457775 0.0\n",
      "0.003726136637851596 0.0\n",
      "0.008902248926460743 0.0\n",
      "0.8195338845252991 1.0\n",
      "0.9996507167816162 1.0\n",
      "0.2576386332511902 0.0\n",
      "0.892715573310852 1.0\n",
      "0.002207051729783416 0.0\n",
      "0.0013733597006648779 0.0\n",
      "0.002697838470339775 0.0\n",
      "0.020428182557225227 0.0\n",
      "0.8444982767105103 1.0\n",
      "0.4558189809322357 0.0\n",
      "0.00016315387620124966 0.0\n",
      "0.40287846326828003 0.0\n",
      "0.9230347275733948 1.0\n",
      "0.9982507824897766 1.0\n",
      "0.0009106042562052608 0.0\n",
      "0.11455310136079788 0.0\n",
      "0.005766330752521753 0.0\n",
      "0.06987302750349045 0.0\n",
      "0.00845325831323862 0.0\n",
      "0.01982300728559494 0.0\n",
      "0.0005727758398279548 0.0\n",
      "0.4319598376750946 1.0\n",
      "0.8991613388061523 1.0\n",
      "0.0193334873765707 0.0\n",
      "0.9718884229660034 1.0\n",
      "0.03662189468741417 0.0\n",
      "0.9710578918457031 1.0\n",
      "0.025256287306547165 0.0\n",
      "0.9799447059631348 1.0\n",
      "0.0010061935754492879 0.0\n",
      "0.9886582493782043 1.0\n",
      "0.9876378774642944 1.0\n",
      "0.6675586700439453 1.0\n",
      "0.01940428465604782 0.0\n",
      "0.00403058435767889 0.0\n",
      "0.016426414251327515 0.0\n",
      "0.9580182433128357 1.0\n",
      "0.8443676233291626 1.0\n",
      "0.010815223678946495 0.0\n",
      "0.0006719002849422395 0.0\n",
      "0.002731999149546027 0.0\n",
      "0.12632998824119568 0.0\n",
      "0.9992891550064087 1.0\n",
      "0.7653840780258179 1.0\n",
      "TRAIN[steps=4200] loss=0.087209 acc=0.984 P=1.000 R=0.957 F1=0.977778\n",
      "DEV[steps=4200] loss=0.014330 acc=0.692 P=0.692 R=0.667 F1=0.679056 \n",
      "model sim and label tuples:\n",
      "0.9879732131958008 1.0\n",
      "0.00015668357082176954 0.0\n",
      "0.10197274386882782 0.0\n",
      "0.9914628863334656 1.0\n",
      "0.7059701085090637 0.0\n",
      "0.9948083758354187 1.0\n",
      "0.9995918869972229 1.0\n",
      "0.9884247779846191 1.0\n",
      "0.047108542174100876 0.0\n",
      "0.9997898936271667 1.0\n",
      "0.6735144853591919 0.0\n",
      "0.9027849435806274 1.0\n",
      "0.8981008529663086 1.0\n",
      "0.08845943212509155 0.0\n",
      "0.00015582940250169486 0.0\n",
      "0.14061622321605682 0.0\n",
      "0.022427868098020554 0.0\n",
      "0.9462403655052185 1.0\n",
      "0.5691367387771606 1.0\n",
      "0.004329176619648933 0.0\n",
      "0.9981239438056946 1.0\n",
      "0.00026735506253317 0.0\n",
      "0.00025073959841392934 0.0\n",
      "0.9423238039016724 1.0\n",
      "0.39706116914749146 0.0\n",
      "0.0008486562874168158 0.0\n",
      "0.4734206795692444 0.0\n",
      "0.01241188496351242 0.0\n",
      "0.9880605340003967 1.0\n",
      "0.10796715319156647 0.0\n",
      "0.014479363337159157 0.0\n",
      "0.7971479892730713 0.0\n",
      "0.9994620680809021 1.0\n",
      "0.9998811483383179 1.0\n",
      "0.013881247490644455 0.0\n",
      "0.9989087581634521 1.0\n",
      "0.0077951340936124325 0.0\n",
      "0.00017084868159145117 0.0\n",
      "0.8664342761039734 1.0\n",
      "0.43806618452072144 0.0\n",
      "0.9995712637901306 1.0\n",
      "0.8865284323692322 1.0\n",
      "0.00918373093008995 0.0\n",
      "0.9962078332901001 1.0\n",
      "0.019438039511442184 0.0\n",
      "0.9691651463508606 1.0\n",
      "0.8981781005859375 1.0\n",
      "0.2965937852859497 1.0\n",
      "0.9962217807769775 1.0\n",
      "0.009798225946724415 0.0\n",
      "0.9576256275177002 1.0\n",
      "0.00045405206037685275 0.0\n",
      "0.9823679327964783 1.0\n",
      "0.0012113364646211267 0.0\n",
      "0.03850522264838219 0.0\n",
      "0.0006858412525616586 0.0\n",
      "0.9427254796028137 1.0\n",
      "0.38313207030296326 0.0\n",
      "0.9861310720443726 1.0\n",
      "0.9934930205345154 1.0\n",
      "0.9895070195198059 1.0\n",
      "0.6237188577651978 1.0\n",
      "0.01013885997235775 0.0\n",
      "0.9991493225097656 1.0\n",
      "TRAIN[steps=4300] loss=0.156558 acc=0.938 P=0.912 R=0.969 F1=0.939394\n",
      "DEV[steps=4300] loss=0.014320 acc=0.699 P=0.697 R=0.677 F1=0.687051 \n",
      "EPOCH:  7\n",
      "model sim and label tuples:\n",
      "0.9973574280738831 1.0\n",
      "0.29583847522735596 0.0\n",
      "0.003214891767129302 0.0\n",
      "0.9986963868141174 1.0\n",
      "0.9998201727867126 1.0\n",
      "0.9997124075889587 1.0\n",
      "0.0019084084779024124 0.0\n",
      "0.40877223014831543 0.0\n",
      "0.9992671608924866 1.0\n",
      "0.9998027682304382 1.0\n",
      "0.00010097209451487288 0.0\n",
      "0.0002238340675830841 0.0\n",
      "0.999903678894043 1.0\n",
      "0.9999204874038696 1.0\n",
      "0.9923253059387207 1.0\n",
      "0.9816562533378601 1.0\n",
      "0.00125088170170784 0.0\n",
      "0.9999268054962158 1.0\n",
      "0.7315059900283813 1.0\n",
      "0.9945532083511353 1.0\n",
      "0.27353715896606445 1.0\n",
      "0.03268979862332344 0.0\n",
      "0.9918374419212341 1.0\n",
      "0.9987297654151917 1.0\n",
      "0.4569045603275299 1.0\n",
      "0.6573407649993896 1.0\n",
      "0.6657060384750366 1.0\n",
      "0.0015931110829114914 0.0\n",
      "0.00024550920352339745 0.0\n",
      "0.0002716548042371869 0.0\n",
      "0.035374827682971954 0.0\n",
      "0.9930036664009094 1.0\n",
      "0.0003331250336486846 0.0\n",
      "0.9999291896820068 1.0\n",
      "0.0008725067600607872 0.0\n",
      "0.0003267499268986285 0.0\n",
      "0.9952636957168579 1.0\n",
      "0.001866100705228746 0.0\n",
      "0.673348069190979 1.0\n",
      "0.3347680866718292 1.0\n",
      "0.00028490854310803115 0.0\n",
      "0.044704750180244446 0.0\n",
      "6.609057891182601e-05 0.0\n",
      "0.9999139308929443 1.0\n",
      "0.002317140344530344 0.0\n",
      "0.9962158799171448 1.0\n",
      "6.5319036366418e-05 0.0\n",
      "0.001960308291018009 0.0\n",
      "0.003200266510248184 0.0\n",
      "9.89072141237557e-05 0.0\n",
      "0.999525785446167 1.0\n",
      "0.12929211556911469 0.0\n",
      "0.9732521176338196 1.0\n",
      "0.9997163414955139 1.0\n",
      "0.5607477426528931 0.0\n",
      "0.9927504062652588 1.0\n",
      "0.9999319314956665 1.0\n",
      "0.0042257183231413364 0.0\n",
      "0.0013815159909427166 0.0\n",
      "0.9996953010559082 1.0\n",
      "0.017109805718064308 0.0\n",
      "0.8493000268936157 1.0\n",
      "0.03794962540268898 0.0\n",
      "0.12347396463155746 0.0\n",
      "TRAIN[steps=4400] loss=0.111497 acc=0.938 P=0.968 R=0.909 F1=0.937500\n",
      "DEV[steps=4400] loss=0.017020 acc=0.692 P=0.691 R=0.669 F1=0.679933 \n",
      "model sim and label tuples:\n",
      "6.097884033806622e-05 0.0\n",
      "0.36193862557411194 0.0\n",
      "0.01743546687066555 0.0\n",
      "0.9999487400054932 1.0\n",
      "0.023676659911870956 0.0\n",
      "0.7186628580093384 1.0\n",
      "0.9998224377632141 1.0\n",
      "0.03749551996588707 0.0\n",
      "0.9938749670982361 1.0\n",
      "0.01425457838922739 0.0\n",
      "0.9990069270133972 1.0\n",
      "0.9998667240142822 1.0\n",
      "0.0006379167316481471 0.0\n",
      "0.9991426467895508 1.0\n",
      "0.003165300702676177 0.0\n",
      "0.9997507929801941 1.0\n",
      "0.7719502449035645 1.0\n",
      "0.0005585829494521022 0.0\n",
      "0.8181424736976624 1.0\n",
      "0.0010615213541314006 0.0\n",
      "0.9980549812316895 1.0\n",
      "0.5896283984184265 1.0\n",
      "0.9973745346069336 1.0\n",
      "8.91861564014107e-05 0.0\n",
      "0.001824124134145677 1.0\n",
      "0.0002039773971773684 0.0\n",
      "0.2725403904914856 0.0\n",
      "0.007238597143441439 0.0\n",
      "0.0002491645864211023 0.0\n",
      "0.9990288019180298 1.0\n",
      "0.0013082885416224599 0.0\n",
      "0.9914208650588989 1.0\n",
      "0.609329342842102 1.0\n",
      "0.016748182475566864 0.0\n",
      "0.49777477979660034 1.0\n",
      "0.9794328212738037 1.0\n",
      "0.7472156286239624 1.0\n",
      "0.0001780653983587399 0.0\n",
      "0.9537822008132935 1.0\n",
      "0.9998327493667603 1.0\n",
      "0.0021453918889164925 0.0\n",
      "0.9974005222320557 1.0\n",
      "0.00013458072498906404 0.0\n",
      "0.9994762539863586 1.0\n",
      "0.9557298421859741 1.0\n",
      "0.9993306398391724 1.0\n",
      "0.025451164692640305 0.0\n",
      "0.00010974442557198927 0.0\n",
      "0.0029946903232485056 0.0\n",
      "0.9996881484985352 1.0\n",
      "0.03801165893673897 0.0\n",
      "0.998490571975708 1.0\n",
      "0.004367289133369923 0.0\n",
      "0.9997678399085999 1.0\n",
      "0.999767005443573 1.0\n",
      "0.000896753859706223 0.0\n",
      "0.8812448978424072 1.0\n",
      "5.3944069804856554e-05 0.0\n",
      "0.0005013656336814165 0.0\n",
      "0.0004254765808582306 0.0\n",
      "0.0002826194977387786 0.0\n",
      "0.998155415058136 1.0\n",
      "0.9819443225860596 1.0\n",
      "0.00016937860345933586 0.0\n",
      "TRAIN[steps=4500] loss=0.162001 acc=0.984 P=1.000 R=0.970 F1=0.984615\n",
      "DEV[steps=4500] loss=0.017116 acc=0.693 P=0.690 R=0.676 F1=0.682826 \n",
      "model sim and label tuples:\n",
      "0.06597201526165009 1.0\n",
      "0.0017122965073212981 0.0\n",
      "0.10132647305727005 0.0\n",
      "0.3049481213092804 1.0\n",
      "0.010456368327140808 0.0\n",
      "0.0015450625214725733 0.0\n",
      "0.9889296889305115 1.0\n",
      "0.00034270421019755304 0.0\n",
      "0.9994781613349915 1.0\n",
      "0.0018954977858811617 0.0\n",
      "0.8557833433151245 1.0\n",
      "0.9925767779350281 1.0\n",
      "0.9970026612281799 1.0\n",
      "0.9545411467552185 1.0\n",
      "0.4847443699836731 1.0\n",
      "0.863466739654541 1.0\n",
      "0.004506672732532024 0.0\n",
      "0.0041974796913564205 0.0\n",
      "4.104805339011364e-05 0.0\n",
      "3.3485383028164506e-05 0.0\n",
      "3.2305109925800934e-05 0.0\n",
      "0.9639576077461243 1.0\n",
      "0.9980584979057312 1.0\n",
      "0.0011163806775584817 0.0\n",
      "8.774590969551355e-05 0.0\n",
      "0.9869580864906311 1.0\n",
      "0.1768556833267212 1.0\n",
      "0.999671459197998 1.0\n",
      "7.524576358264312e-05 0.0\n",
      "0.00021633169671986252 0.0\n",
      "0.9029666185379028 1.0\n",
      "0.9997803568840027 1.0\n",
      "0.004660274367779493 0.0\n",
      "0.9997133612632751 1.0\n",
      "5.377165143727325e-05 0.0\n",
      "4.941110091749579e-05 0.0\n",
      "0.9558005332946777 1.0\n",
      "0.9971318244934082 1.0\n",
      "0.9997968077659607 1.0\n",
      "3.9096001273719594e-05 0.0\n",
      "4.046575486427173e-05 0.0\n",
      "0.999941349029541 1.0\n",
      "0.011813582852482796 0.0\n",
      "4.2391973693156615e-05 0.0\n",
      "0.00010460730118211359 0.0\n",
      "0.000260069384239614 0.0\n",
      "0.0018148428061977029 0.0\n",
      "0.998681366443634 1.0\n",
      "0.9998575448989868 1.0\n",
      "0.926029622554779 1.0\n",
      "4.0376708057010546e-05 0.0\n",
      "0.001887578284367919 0.0\n",
      "0.011205402202904224 0.0\n",
      "0.003898611292243004 0.0\n",
      "0.625070333480835 1.0\n",
      "0.00036738195922225714 0.0\n",
      "3.249739529564977e-05 0.0\n",
      "0.3259113132953644 0.0\n",
      "0.9963141083717346 1.0\n",
      "0.0017357391770929098 0.0\n",
      "0.9739634394645691 1.0\n",
      "0.9997590184211731 1.0\n",
      "0.9997590184211731 1.0\n",
      "0.010515240021049976 0.0\n",
      "TRAIN[steps=4600] loss=0.126437 acc=0.953 P=1.000 R=0.900 F1=0.947368\n",
      "DEV[steps=4600] loss=0.018501 acc=0.686 P=0.684 R=0.662 F1=0.673057 \n",
      "model sim and label tuples:\n",
      "0.09418891370296478 0.0\n",
      "0.23038610816001892 0.0\n",
      "0.9901386499404907 1.0\n",
      "0.9991727471351624 1.0\n",
      "0.0025963205844163895 0.0\n",
      "0.9975970387458801 1.0\n",
      "0.0003310036554466933 0.0\n",
      "0.9914602041244507 1.0\n",
      "0.9983800649642944 1.0\n",
      "0.0014105316950008273 0.0\n",
      "0.0001507216802565381 0.0\n",
      "0.00024048601335380226 0.0\n",
      "0.9997566342353821 1.0\n",
      "0.4109005033969879 1.0\n",
      "0.00231572100892663 0.0\n",
      "0.0002075185184367001 0.0\n",
      "0.00048643065383657813 0.0\n",
      "0.9817188382148743 1.0\n",
      "0.9341443777084351 1.0\n",
      "0.019326142966747284 0.0\n",
      "0.012896958738565445 0.0\n",
      "0.9977062940597534 1.0\n",
      "0.0004900030908174813 0.0\n",
      "0.014658004976809025 0.0\n",
      "0.00015177794557530433 0.0\n",
      "0.05408770218491554 0.0\n",
      "0.025962812826037407 0.0\n",
      "0.0007931061554700136 0.0\n",
      "0.9969444870948792 1.0\n",
      "0.021561363711953163 0.0\n",
      "0.9949448704719543 1.0\n",
      "0.0019327731570228934 0.0\n",
      "5.129349301569164e-05 0.0\n",
      "0.002323647728189826 0.0\n",
      "0.9974728226661682 1.0\n",
      "9.828133624978364e-05 0.0\n",
      "0.9997950196266174 1.0\n",
      "0.9436125755310059 1.0\n",
      "0.00046480572200380266 0.0\n",
      "0.9998929500579834 1.0\n",
      "0.07844632118940353 0.0\n",
      "0.00013263191794976592 0.0\n",
      "0.9857450127601624 1.0\n",
      "0.00018123256450053304 0.0\n",
      "0.7596876621246338 1.0\n",
      "0.0007299893768504262 0.0\n",
      "0.6737725138664246 1.0\n",
      "0.997359573841095 1.0\n",
      "0.9957119226455688 1.0\n",
      "0.9996565580368042 1.0\n",
      "0.993449866771698 1.0\n",
      "0.999497652053833 1.0\n",
      "0.9996988773345947 1.0\n",
      "8.886829891707748e-05 0.0\n",
      "0.0074295541271567345 0.0\n",
      "0.9908069968223572 1.0\n",
      "0.0015993736451491714 0.0\n",
      "0.9933591485023499 1.0\n",
      "0.9961215853691101 1.0\n",
      "0.00635661231353879 0.0\n",
      "0.00010389046656200662 0.0\n",
      "5.6757620768621564e-05 0.0\n",
      "0.8525764346122742 0.0\n",
      "0.5727720856666565 0.0\n",
      "TRAIN[steps=4700] loss=0.080914 acc=0.953 P=0.931 R=0.964 F1=0.947368\n",
      "DEV[steps=4700] loss=0.017395 acc=0.690 P=0.690 R=0.665 F1=0.677248 \n",
      "model sim and label tuples:\n",
      "0.9987972974777222 1.0\n",
      "0.9251757264137268 1.0\n",
      "0.9937758445739746 1.0\n",
      "0.0014666441129520535 0.0\n",
      "0.03643227368593216 0.0\n",
      "0.9998695850372314 1.0\n",
      "0.9886260628700256 1.0\n",
      "0.9980803728103638 1.0\n",
      "0.9996329545974731 1.0\n",
      "0.00687091238796711 0.0\n",
      "0.9818304181098938 1.0\n",
      "0.9997419714927673 1.0\n",
      "0.938398540019989 1.0\n",
      "0.12408781051635742 0.0\n",
      "0.0006419729324989021 0.0\n",
      "0.9993113279342651 1.0\n",
      "0.9684624671936035 1.0\n",
      "0.9588646292686462 1.0\n",
      "7.065840327413753e-05 0.0\n",
      "0.00013267606846056879 0.0\n",
      "0.9937905669212341 1.0\n",
      "0.9704283475875854 1.0\n",
      "0.00035122191184200346 0.0\n",
      "0.000178726069862023 0.0\n",
      "0.997772753238678 1.0\n",
      "0.9999542236328125 1.0\n",
      "0.0007239372353069484 0.0\n",
      "0.022702405229210854 0.0\n",
      "0.05658229812979698 0.0\n",
      "0.021694175899028778 0.0\n",
      "0.9967526793479919 1.0\n",
      "0.9999576807022095 1.0\n",
      "0.007991869002580643 0.0\n",
      "0.9998468160629272 1.0\n",
      "0.6868782639503479 1.0\n",
      "0.9149236083030701 1.0\n",
      "0.0007908507832325995 0.0\n",
      "0.9909522533416748 1.0\n",
      "0.0001803477352950722 0.0\n",
      "0.023152757436037064 0.0\n",
      "0.0001358400913886726 0.0\n",
      "0.9977219700813293 1.0\n",
      "0.9993470311164856 1.0\n",
      "0.999956488609314 1.0\n",
      "0.9999463558197021 1.0\n",
      "0.1705399453639984 0.0\n",
      "0.6852437853813171 1.0\n",
      "0.9999468326568604 1.0\n",
      "0.0011325911618769169 0.0\n",
      "0.07851370424032211 1.0\n",
      "0.5218095183372498 0.0\n",
      "0.0009196754544973373 0.0\n",
      "0.9974315762519836 1.0\n",
      "0.18520231544971466 0.0\n",
      "0.0008774327579885721 0.0\n",
      "0.0019213129999116063 0.0\n",
      "0.06481359899044037 0.0\n",
      "0.4607946574687958 0.0\n",
      "0.9994463324546814 1.0\n",
      "0.0004569446900859475 0.0\n",
      "0.9948768019676208 1.0\n",
      "7.549027213826776e-05 0.0\n",
      "0.9999009370803833 1.0\n",
      "0.9890579581260681 1.0\n",
      "TRAIN[steps=4800] loss=0.091435 acc=0.969 P=0.971 R=0.971 F1=0.971429\n",
      "DEV[steps=4800] loss=0.018212 acc=0.690 P=0.667 R=0.728 F1=0.696266 \n",
      "model sim and label tuples:\n",
      "0.0002239181485492736 0.0\n",
      "0.9993846416473389 1.0\n",
      "0.9984545707702637 1.0\n",
      "0.00039791842573322356 0.0\n",
      "0.06143897399306297 0.0\n",
      "0.002714158035814762 0.0\n",
      "0.9574382901191711 1.0\n",
      "0.9965510368347168 1.0\n",
      "0.12379460781812668 0.0\n",
      "0.9514611959457397 1.0\n",
      "0.04791460186243057 0.0\n",
      "0.989594578742981 1.0\n",
      "0.9990480542182922 1.0\n",
      "0.8336671590805054 1.0\n",
      "0.00033023362630046904 0.0\n",
      "0.9927982091903687 1.0\n",
      "0.0009636796894483268 0.0\n",
      "0.2159961462020874 0.0\n",
      "0.09816627204418182 0.0\n",
      "0.0014558661496266723 0.0\n",
      "0.9891188144683838 1.0\n",
      "0.009195409715175629 0.0\n",
      "0.6996529698371887 1.0\n",
      "0.997344434261322 1.0\n",
      "0.8952608108520508 1.0\n",
      "0.0016335913678631186 0.0\n",
      "0.978261411190033 1.0\n",
      "0.9994333386421204 1.0\n",
      "0.8203990459442139 1.0\n",
      "0.00010707067121984437 0.0\n",
      "0.8700331449508667 1.0\n",
      "0.6163592338562012 1.0\n",
      "0.9970537424087524 1.0\n",
      "0.8049421310424805 0.0\n",
      "0.023089876398444176 0.0\n",
      "0.0016871304251253605 0.0\n",
      "0.9998538494110107 1.0\n",
      "0.9822435975074768 1.0\n",
      "0.9997792840003967 1.0\n",
      "0.994517982006073 1.0\n",
      "0.0018953668186441064 0.0\n",
      "0.00037801070720888674 0.0\n",
      "0.9979293346405029 1.0\n",
      "0.00018871969950851053 0.0\n",
      "0.9894271492958069 1.0\n",
      "0.8013156652450562 1.0\n",
      "0.03290350362658501 0.0\n",
      "0.7758209705352783 1.0\n",
      "0.999419093132019 1.0\n",
      "0.05811898410320282 0.0\n",
      "0.016155973076820374 0.0\n",
      "0.08015825599431992 0.0\n",
      "0.525689959526062 1.0\n",
      "0.9851385951042175 1.0\n",
      "0.9968540072441101 1.0\n",
      "0.8849382996559143 1.0\n",
      "0.9886847138404846 1.0\n",
      "0.9997308850288391 1.0\n",
      "0.042608581483364105 0.0\n",
      "0.9929764866828918 1.0\n",
      "0.7269914746284485 1.0\n",
      "0.9901716113090515 1.0\n",
      "0.9703044295310974 1.0\n",
      "0.9949613809585571 1.0\n",
      "TRAIN[steps=4900] loss=0.090827 acc=0.984 P=0.975 R=1.000 F1=0.987342\n",
      "DEV[steps=4900] loss=0.015945 acc=0.696 P=0.691 R=0.684 F1=0.687674 \n",
      "EPOCH:  8\n",
      "model sim and label tuples:\n",
      "0.9863190650939941 1.0\n",
      "0.17518799006938934 0.0\n",
      "0.9745848178863525 1.0\n",
      "0.004832988604903221 1.0\n",
      "0.9997215867042542 1.0\n",
      "9.769863390829414e-05 0.0\n",
      "0.9979130625724792 1.0\n",
      "0.9893150329589844 1.0\n",
      "0.009984993375837803 0.0\n",
      "0.9487125873565674 1.0\n",
      "0.0034321602433919907 0.0\n",
      "0.9988538026809692 1.0\n",
      "0.015658657997846603 0.0\n",
      "0.009061158634722233 0.0\n",
      "0.04000715911388397 0.0\n",
      "0.9643321633338928 1.0\n",
      "0.9924325942993164 1.0\n",
      "0.5055990815162659 0.0\n",
      "0.9996885061264038 1.0\n",
      "0.8786899447441101 0.0\n",
      "0.003279309021309018 0.0\n",
      "0.06630997359752655 0.0\n",
      "0.9915412068367004 1.0\n",
      "0.999505877494812 1.0\n",
      "0.9970724582672119 1.0\n",
      "0.008593512699007988 0.0\n",
      "0.0003649226564448327 0.0\n",
      "0.006445418111979961 0.0\n",
      "0.232092022895813 0.0\n",
      "0.030743058770895004 0.0\n",
      "0.012447367422282696 0.0\n",
      "0.9975333213806152 1.0\n",
      "0.07131775468587875 0.0\n",
      "0.9987369179725647 1.0\n",
      "0.0020475483033806086 0.0\n",
      "0.6958936452865601 1.0\n",
      "0.028318054974079132 0.0\n",
      "0.9994526505470276 1.0\n",
      "0.0008085735025815666 0.0\n",
      "0.9951664209365845 1.0\n",
      "0.9998136162757874 1.0\n",
      "0.0013476611347869039 0.0\n",
      "0.10663352906703949 0.0\n",
      "0.0019589813891798258 0.0\n",
      "0.9960035681724548 1.0\n",
      "0.19724228978157043 0.0\n",
      "0.07742999494075775 0.0\n",
      "0.0003146289964206517 0.0\n",
      "0.9957602620124817 1.0\n",
      "0.9995506405830383 1.0\n",
      "0.9961022138595581 1.0\n",
      "0.9967641830444336 1.0\n",
      "0.5603071451187134 0.0\n",
      "0.003953553736209869 0.0\n",
      "0.9955731630325317 1.0\n",
      "0.5324901938438416 0.0\n",
      "0.011740018613636494 0.0\n",
      "0.9948805570602417 1.0\n",
      "0.00018322395044378936 0.0\n",
      "0.7045372724533081 1.0\n",
      "7.791642565280199e-05 0.0\n",
      "0.8678730130195618 1.0\n",
      "0.005011409055441618 0.0\n",
      "0.00016179522208403796 0.0\n",
      "TRAIN[steps=5000] loss=0.187351 acc=0.922 P=0.875 R=0.966 F1=0.918033\n",
      "DEV[steps=5000] loss=0.017124 acc=0.694 P=0.663 R=0.758 F1=0.707401 *\n",
      "model sim and label tuples:\n",
      "0.0003226394474040717 0.0\n",
      "0.9483676552772522 1.0\n",
      "0.00635506771504879 0.0\n",
      "4.233590152580291e-05 0.0\n",
      "0.9981415271759033 1.0\n",
      "0.0024878261610865593 0.0\n",
      "0.9889779686927795 1.0\n",
      "0.00022886622173245996 0.0\n",
      "4.064609674969688e-05 0.0\n",
      "0.9799200892448425 1.0\n",
      "0.8048710227012634 1.0\n",
      "0.0002092470822390169 0.0\n",
      "0.005647577345371246 0.0\n",
      "0.9996111989021301 1.0\n",
      "0.9999039173126221 1.0\n",
      "0.007960017770528793 0.0\n",
      "0.9009965062141418 0.0\n",
      "0.03205518051981926 0.0\n",
      "5.0545233534649014e-05 0.0\n",
      "0.9998830556869507 1.0\n",
      "0.9997043013572693 1.0\n",
      "0.6331964731216431 0.0\n",
      "0.9998987913131714 1.0\n",
      "0.0002339472557650879 0.0\n",
      "0.8574151992797852 1.0\n",
      "0.9999648332595825 1.0\n",
      "0.00023099601094145328 0.0\n",
      "0.3294372856616974 0.0\n",
      "0.0003329351602587849 0.0\n",
      "0.0031445110216736794 0.0\n",
      "0.9982025623321533 1.0\n",
      "0.9998233914375305 1.0\n",
      "0.9990553259849548 1.0\n",
      "3.63185208698269e-05 0.0\n",
      "0.9892590641975403 1.0\n",
      "0.5997539758682251 1.0\n",
      "3.554581053322181e-05 0.0\n",
      "0.9690789580345154 1.0\n",
      "0.25925490260124207 1.0\n",
      "0.9687173366546631 1.0\n",
      "0.02245762012898922 0.0\n",
      "0.005964307580143213 0.0\n",
      "0.9993230104446411 1.0\n",
      "0.10527048259973526 0.0\n",
      "0.9997312426567078 1.0\n",
      "0.999768078327179 1.0\n",
      "0.00014582682342734188 0.0\n",
      "0.06937925517559052 0.0\n",
      "0.06152952089905739 1.0\n",
      "0.00012575497385114431 0.0\n",
      "0.9995513558387756 1.0\n",
      "0.9999271631240845 1.0\n",
      "0.9995834231376648 1.0\n",
      "7.484366506105289e-05 0.0\n",
      "0.7912824749946594 1.0\n",
      "0.0015141122275963426 0.0\n",
      "0.005813978612422943 0.0\n",
      "0.030686208978295326 0.0\n",
      "8.006783900782466e-05 0.0\n",
      "4.981148595106788e-05 0.0\n",
      "0.999931812286377 1.0\n",
      "0.999854564666748 1.0\n",
      "0.9970179796218872 1.0\n",
      "0.00017644221952650696 0.0\n",
      "TRAIN[steps=5100] loss=0.147658 acc=0.938 P=0.935 R=0.935 F1=0.935484\n",
      "DEV[steps=5100] loss=0.020101 acc=0.697 P=0.677 R=0.726 F1=0.700742 \n",
      "model sim and label tuples:\n",
      "0.9973479509353638 1.0\n",
      "0.00036169480881653726 0.0\n",
      "0.9965500831604004 1.0\n",
      "0.9993247985839844 1.0\n",
      "0.00223311479203403 0.0\n",
      "0.9670163989067078 1.0\n",
      "0.999790608882904 1.0\n",
      "0.999945878982544 1.0\n",
      "4.221531344228424e-05 0.0\n",
      "0.7877257466316223 0.0\n",
      "0.11036323755979538 0.0\n",
      "0.9993459582328796 1.0\n",
      "0.0041658468544483185 0.0\n",
      "0.030641788616776466 0.0\n",
      "0.9798884987831116 1.0\n",
      "0.9996564388275146 1.0\n",
      "0.0006060064188204706 0.0\n",
      "0.0034515303559601307 0.0\n",
      "0.9999654293060303 1.0\n",
      "0.8378574848175049 0.0\n",
      "0.000176960980752483 0.0\n",
      "0.9916525483131409 1.0\n",
      "0.999404788017273 1.0\n",
      "0.0009390196646563709 0.0\n",
      "0.0008531562052667141 0.0\n",
      "0.9885326623916626 1.0\n",
      "0.0001890961721073836 0.0\n",
      "0.0032369771506637335 0.0\n",
      "0.9998775720596313 1.0\n",
      "0.00289134681224823 0.0\n",
      "0.00018603747594170272 0.0\n",
      "5.3627289162250236e-05 0.0\n",
      "0.9988037347793579 1.0\n",
      "0.9719736576080322 1.0\n",
      "0.8720613121986389 1.0\n",
      "0.9996113181114197 1.0\n",
      "0.9140751957893372 1.0\n",
      "0.9999322891235352 1.0\n",
      "0.9975372552871704 1.0\n",
      "0.001727206981740892 0.0\n",
      "5.972447252133861e-05 0.0\n",
      "0.06835847347974777 0.0\n",
      "0.00040596252074465156 0.0\n",
      "0.7763611078262329 1.0\n",
      "0.1111406460404396 0.0\n",
      "0.9999189376831055 1.0\n",
      "0.005088816862553358 0.0\n",
      "0.008327953517436981 0.0\n",
      "3.5922701499657705e-05 0.0\n",
      "0.9993983507156372 1.0\n",
      "0.9918211698532104 1.0\n",
      "0.0024146134965121746 0.0\n",
      "0.02925240248441696 0.0\n",
      "0.7692734599113464 0.0\n",
      "0.9998831748962402 1.0\n",
      "0.9973281621932983 1.0\n",
      "6.066725109121762e-05 0.0\n",
      "0.027071267366409302 0.0\n",
      "0.9937554597854614 1.0\n",
      "0.008138653822243214 0.0\n",
      "0.0008355487370863557 0.0\n",
      "0.004492438863962889 0.0\n",
      "0.009427563287317753 0.0\n",
      "0.0007116703782230616 0.0\n",
      "TRAIN[steps=5200] loss=0.092246 acc=0.953 P=0.903 R=1.000 F1=0.949153\n",
      "DEV[steps=5200] loss=0.019693 acc=0.697 P=0.686 R=0.699 F1=0.692394 \n",
      "model sim and label tuples:\n",
      "4.755404734169133e-05 0.0\n",
      "5.239546953816898e-05 0.0\n",
      "0.9998390674591064 1.0\n",
      "0.9994890689849854 1.0\n",
      "8.510358020430431e-05 0.0\n",
      "0.00241855182684958 0.0\n",
      "0.002647434128448367 0.0\n",
      "0.6219824552536011 1.0\n",
      "0.433718204498291 0.0\n",
      "0.00017520097026135772 0.0\n",
      "0.00014600061695091426 0.0\n",
      "0.8978486657142639 1.0\n",
      "0.5377035140991211 0.0\n",
      "0.996582567691803 1.0\n",
      "5.1773327868431807e-05 0.0\n",
      "0.9978981018066406 1.0\n",
      "0.04884069412946701 0.0\n",
      "0.9938002228736877 1.0\n",
      "0.8991413712501526 0.0\n",
      "4.9375867092749104e-05 0.0\n",
      "0.9994413256645203 1.0\n",
      "0.9875065088272095 1.0\n",
      "0.9794502258300781 1.0\n",
      "0.9999191761016846 1.0\n",
      "0.003796783508732915 0.0\n",
      "0.9943420886993408 1.0\n",
      "0.00022651103790849447 0.0\n",
      "0.823228657245636 1.0\n",
      "0.9881614446640015 1.0\n",
      "0.00014328843099065125 0.0\n",
      "0.0003297472430858761 0.0\n",
      "0.0007601760444231331 0.0\n",
      "0.9120982885360718 1.0\n",
      "0.018406851217150688 0.0\n",
      "0.40053728222846985 0.0\n",
      "0.9926631450653076 1.0\n",
      "0.00016765433247201145 0.0\n",
      "0.9974186420440674 1.0\n",
      "5.221122773946263e-05 0.0\n",
      "0.8187000751495361 1.0\n",
      "0.994222104549408 1.0\n",
      "0.9961276650428772 1.0\n",
      "0.6603605151176453 1.0\n",
      "0.9995562434196472 1.0\n",
      "0.03959943354129791 1.0\n",
      "0.9183788895606995 1.0\n",
      "0.0002570099022705108 0.0\n",
      "0.9836740493774414 1.0\n",
      "0.9999537467956543 1.0\n",
      "0.9989750385284424 1.0\n",
      "0.9822845458984375 1.0\n",
      "0.0008593295933678746 0.0\n",
      "0.9995447993278503 1.0\n",
      "0.0008459186065010726 0.0\n",
      "0.9996053576469421 1.0\n",
      "0.00018855405505746603 0.0\n",
      "0.023911485448479652 0.0\n",
      "0.9968168139457703 1.0\n",
      "0.9999203681945801 1.0\n",
      "0.9069612622261047 1.0\n",
      "0.0005097780958749354 0.0\n",
      "0.011805296875536442 0.0\n",
      "0.09199734777212143 0.0\n",
      "9.34829076868482e-05 0.0\n",
      "TRAIN[steps=5300] loss=0.146571 acc=0.953 P=0.941 R=0.970 F1=0.955224\n",
      "DEV[steps=5300] loss=0.018587 acc=0.696 P=0.693 R=0.681 F1=0.686583 \n",
      "model sim and label tuples:\n",
      "0.9967247843742371 1.0\n",
      "0.9786575436592102 1.0\n",
      "0.9997401833534241 1.0\n",
      "0.936974048614502 1.0\n",
      "0.9879613518714905 1.0\n",
      "0.06789766252040863 1.0\n",
      "0.0014459070516750216 0.0\n",
      "0.0012444675667211413 0.0\n",
      "0.12333624064922333 0.0\n",
      "9.187097748508677e-05 0.0\n",
      "0.9927674531936646 1.0\n",
      "0.999392032623291 1.0\n",
      "0.9375779032707214 1.0\n",
      "0.1618795394897461 0.0\n",
      "0.9976451992988586 1.0\n",
      "0.9998849630355835 1.0\n",
      "0.9996780157089233 1.0\n",
      "0.9795661568641663 1.0\n",
      "0.9982869029045105 1.0\n",
      "0.7629740834236145 0.0\n",
      "0.9315206408500671 1.0\n",
      "7.317043491639197e-05 0.0\n",
      "0.0040431031957268715 0.0\n",
      "0.0006944003980606794 0.0\n",
      "0.9967705011367798 1.0\n",
      "0.99993896484375 1.0\n",
      "0.0007718572160229087 0.0\n",
      "0.0021145078353583813 0.0\n",
      "0.9995239973068237 1.0\n",
      "0.4206545948982239 0.0\n",
      "0.1430637538433075 0.0\n",
      "0.9758597016334534 1.0\n",
      "0.973116397857666 1.0\n",
      "0.9962728023529053 1.0\n",
      "0.031209956854581833 0.0\n",
      "0.0005292972782626748 0.0\n",
      "0.9832783341407776 1.0\n",
      "0.9984893798828125 1.0\n",
      "0.00020710557873826474 0.0\n",
      "6.541377661051229e-05 0.0\n",
      "6.749949534423649e-05 0.0\n",
      "0.000781262235250324 0.0\n",
      "0.31683793663978577 0.0\n",
      "0.00013434096763376147 0.0\n",
      "8.532485662726685e-05 0.0\n",
      "0.0009181913919746876 0.0\n",
      "0.9885708689689636 1.0\n",
      "0.997835099697113 1.0\n",
      "0.8521248698234558 1.0\n",
      "0.9703605771064758 1.0\n",
      "0.0028150074649602175 0.0\n",
      "0.002703492995351553 0.0\n",
      "0.9988991022109985 1.0\n",
      "0.0003603388322517276 0.0\n",
      "0.006460156291723251 0.0\n",
      "0.6974289417266846 0.0\n",
      "0.9973000884056091 1.0\n",
      "0.00011984620505245402 0.0\n",
      "0.9987677335739136 1.0\n",
      "0.9999144077301025 1.0\n",
      "0.00012240244541317225 0.0\n",
      "0.0005952547071501613 0.0\n",
      "0.07139575481414795 0.0\n",
      "0.9986009001731873 1.0\n",
      "TRAIN[steps=5400] loss=0.115705 acc=0.953 P=0.939 R=0.969 F1=0.953846\n",
      "DEV[steps=5400] loss=0.018118 acc=0.695 P=0.689 R=0.684 F1=0.686408 \n",
      "model sim and label tuples:\n",
      "0.14277806878089905 0.0\n",
      "0.8212732076644897 1.0\n",
      "0.0050874678418040276 0.0\n",
      "0.00015793951752129942 0.0\n",
      "0.9993314743041992 1.0\n",
      "0.001563513302244246 0.0\n",
      "0.9976935982704163 1.0\n",
      "0.99868243932724 1.0\n",
      "0.01847810298204422 0.0\n",
      "0.9999336004257202 1.0\n",
      "8.794250607024878e-05 0.0\n",
      "0.9731396436691284 1.0\n",
      "0.0003155084850732237 0.0\n",
      "0.08191391080617905 0.0\n",
      "0.98912113904953 1.0\n",
      "0.8175501823425293 1.0\n",
      "7.092302985256538e-05 0.0\n",
      "0.009532208554446697 0.0\n",
      "0.9868038296699524 1.0\n",
      "0.7151760458946228 1.0\n",
      "0.9693191647529602 1.0\n",
      "0.5286155343055725 0.0\n",
      "0.9932677745819092 1.0\n",
      "0.9888661503791809 1.0\n",
      "0.23208318650722504 0.0\n",
      "0.9991471767425537 1.0\n",
      "0.9869688153266907 1.0\n",
      "0.9991616010665894 1.0\n",
      "0.00014829747669864446 0.0\n",
      "0.00010491478315088898 0.0\n",
      "0.00027071882504969835 0.0\n",
      "0.9747328162193298 1.0\n",
      "6.830272468505427e-05 0.0\n",
      "0.000900755578186363 0.0\n",
      "0.9500616192817688 1.0\n",
      "0.8105370402336121 1.0\n",
      "0.9560027718544006 1.0\n",
      "0.0022229310125112534 0.0\n",
      "0.9609158635139465 1.0\n",
      "0.010828050784766674 0.0\n",
      "0.023535359650850296 0.0\n",
      "0.9349210262298584 1.0\n",
      "0.9993103742599487 1.0\n",
      "0.005455195903778076 0.0\n",
      "0.003169819014146924 0.0\n",
      "0.9983691573143005 1.0\n",
      "0.7283624410629272 1.0\n",
      "7.301945879589766e-05 0.0\n",
      "0.00017833028687164187 0.0\n",
      "0.9618370532989502 1.0\n",
      "0.9803338646888733 1.0\n",
      "0.9533146023750305 1.0\n",
      "0.11154881119728088 0.0\n",
      "0.0011919309617951512 0.0\n",
      "0.9999352693557739 1.0\n",
      "0.6179865002632141 1.0\n",
      "0.8746916055679321 1.0\n",
      "8.15762541606091e-05 0.0\n",
      "0.000353385170456022 0.0\n",
      "0.005534645169973373 0.0\n",
      "0.010187884792685509 0.0\n",
      "0.00035590332117863 0.0\n",
      "0.9971876740455627 1.0\n",
      "0.000185298704309389 0.0\n",
      "TRAIN[steps=5500] loss=0.059547 acc=0.984 P=0.970 R=1.000 F1=0.984615\n",
      "DEV[steps=5500] loss=0.018729 acc=0.695 P=0.704 R=0.649 F1=0.675341 \n",
      "model sim and label tuples:\n",
      "0.002951831091195345 0.0\n",
      "0.6126911640167236 0.0\n",
      "0.9999376535415649 1.0\n",
      "0.1575024425983429 0.0\n",
      "0.9837513566017151 1.0\n",
      "0.02099878340959549 0.0\n",
      "0.9717787504196167 1.0\n",
      "0.08498537540435791 0.0\n",
      "0.9995970129966736 1.0\n",
      "0.9997379183769226 1.0\n",
      "0.032969601452350616 0.0\n",
      "0.9996600151062012 1.0\n",
      "0.0007025914383120835 0.0\n",
      "0.0074748266488313675 0.0\n",
      "0.0016825380735099316 0.0\n",
      "0.907748281955719 1.0\n",
      "0.0004433708090800792 0.0\n",
      "0.0004354026459623128 0.0\n",
      "0.01752840168774128 0.0\n",
      "0.999497652053833 1.0\n",
      "0.747864305973053 1.0\n",
      "0.0001907207624753937 0.0\n",
      "0.9943925738334656 1.0\n",
      "0.006055380217730999 0.0\n",
      "0.0002773331361822784 0.0\n",
      "0.9982836246490479 1.0\n",
      "0.005706410855054855 0.0\n",
      "0.051880158483982086 0.0\n",
      "6.693203613394871e-05 0.0\n",
      "0.9947865009307861 1.0\n",
      "0.9999368190765381 1.0\n",
      "0.1633537858724594 0.0\n",
      "6.619415216846392e-05 0.0\n",
      "0.00010038474283646792 0.0\n",
      "0.594275176525116 1.0\n",
      "0.9518066644668579 1.0\n",
      "0.00018212698341812938 0.0\n",
      "0.00010406983346911147 0.0\n",
      "0.0002258121530758217 0.0\n",
      "0.9975670576095581 1.0\n",
      "0.9978522062301636 1.0\n",
      "0.9872323274612427 1.0\n",
      "0.9969565868377686 1.0\n",
      "0.996240496635437 1.0\n",
      "0.000870631483849138 0.0\n",
      "0.937079131603241 1.0\n",
      "0.030099833384156227 0.0\n",
      "0.9996500015258789 1.0\n",
      "0.4349088966846466 0.0\n",
      "0.9999260902404785 1.0\n",
      "0.9971060156822205 1.0\n",
      "0.9994107484817505 1.0\n",
      "0.9999357461929321 1.0\n",
      "0.0001728220231598243 0.0\n",
      "0.9992401599884033 1.0\n",
      "0.00010614629718475044 0.0\n",
      "0.00028211259632371366 0.0\n",
      "0.0004236666136421263 0.0\n",
      "0.9954884648323059 1.0\n",
      "0.9966848492622375 1.0\n",
      "9.350047184852883e-05 0.0\n",
      "0.00023045089619699866 0.0\n",
      "0.9999266862869263 1.0\n",
      "0.9904860854148865 1.0\n",
      "TRAIN[steps=5600] loss=0.051107 acc=0.984 P=0.969 R=1.000 F1=0.984127\n",
      "DEV[steps=5600] loss=0.019000 acc=0.693 P=0.685 R=0.689 F1=0.686970 \n",
      "EPOCH:  9\n",
      "model sim and label tuples:\n",
      "0.9528561234474182 1.0\n",
      "0.9997883439064026 1.0\n",
      "0.9961677193641663 1.0\n",
      "0.9999347925186157 1.0\n",
      "0.999415397644043 1.0\n",
      "7.307915802812204e-05 0.0\n",
      "0.00018409361655358225 0.0\n",
      "0.805556058883667 1.0\n",
      "0.005563362967222929 1.0\n",
      "0.0004897105391137302 0.0\n",
      "0.998160183429718 1.0\n",
      "0.08465573191642761 0.0\n",
      "7.430525874951854e-05 0.0\n",
      "0.00010908699914580211 0.0\n",
      "0.9962820410728455 1.0\n",
      "0.00139256299007684 0.0\n",
      "0.018166957423090935 0.0\n",
      "0.7442152500152588 1.0\n",
      "0.9948154091835022 1.0\n",
      "0.9957283139228821 1.0\n",
      "0.09004443138837814 0.0\n",
      "0.0003376822860445827 0.0\n",
      "0.0034911194816231728 0.0\n",
      "0.9954291582107544 1.0\n",
      "0.9994755387306213 1.0\n",
      "0.9884051084518433 1.0\n",
      "0.004399966914206743 0.0\n",
      "0.00992597732692957 0.0\n",
      "0.9999301433563232 1.0\n",
      "0.9997528195381165 1.0\n",
      "0.0023397651966661215 0.0\n",
      "0.00034412823151797056 0.0\n",
      "0.0034969716798514128 0.0\n",
      "0.8718307614326477 1.0\n",
      "0.9999388456344604 1.0\n",
      "0.0012989374808967113 0.0\n",
      "0.00010069651762023568 0.0\n",
      "0.006760215852409601 0.0\n",
      "0.9996126294136047 1.0\n",
      "0.00031555991154164076 0.0\n",
      "0.011715928092598915 0.0\n",
      "0.9989959597587585 1.0\n",
      "0.9672521948814392 1.0\n",
      "9.242175292456523e-05 0.0\n",
      "0.0034714622888714075 0.0\n",
      "0.997199535369873 1.0\n",
      "0.2427951693534851 1.0\n",
      "0.9998244643211365 1.0\n",
      "0.043812405318021774 0.0\n",
      "0.00023166078608483076 0.0\n",
      "0.9969770908355713 1.0\n",
      "0.9998470544815063 1.0\n",
      "0.9938268065452576 1.0\n",
      "0.00010325580660719424 0.0\n",
      "0.002093970077112317 0.0\n",
      "0.997642457485199 1.0\n",
      "0.8392170667648315 1.0\n",
      "0.9553236365318298 1.0\n",
      "0.024636495858430862 0.0\n",
      "0.974307119846344 1.0\n",
      "0.9870296716690063 1.0\n",
      "0.005751336924731731 0.0\n",
      "0.9965193271636963 1.0\n",
      "0.998823344707489 1.0\n",
      "TRAIN[steps=5700] loss=0.124764 acc=0.969 P=1.000 R=0.943 F1=0.970588\n",
      "DEV[steps=5700] loss=0.019274 acc=0.698 P=0.689 R=0.696 F1=0.692417 \n",
      "model sim and label tuples:\n",
      "0.999607503414154 1.0\n",
      "0.6760350465774536 1.0\n",
      "0.9998589754104614 1.0\n",
      "0.007065393961966038 0.0\n",
      "0.00013430381659418344 0.0\n",
      "0.00673038000240922 0.0\n",
      "5.7866793213179335e-05 0.0\n",
      "0.9999352693557739 1.0\n",
      "0.0007182621047832072 0.0\n",
      "0.99949049949646 1.0\n",
      "0.00014987069880589843 0.0\n",
      "0.9546643495559692 1.0\n",
      "0.00014908179582562298 0.0\n",
      "0.9964565634727478 1.0\n",
      "0.9583713412284851 1.0\n",
      "0.9975574016571045 1.0\n",
      "0.014579474925994873 0.0\n",
      "5.616107227979228e-05 0.0\n",
      "0.044544242322444916 1.0\n",
      "0.038682300597429276 0.0\n",
      "0.9998884201049805 1.0\n",
      "0.9712533354759216 1.0\n",
      "0.010611200705170631 0.0\n",
      "0.004442078527063131 0.0\n",
      "0.9912281036376953 1.0\n",
      "0.9893456101417542 1.0\n",
      "0.8773996829986572 1.0\n",
      "0.9999494552612305 1.0\n",
      "0.0001844374492065981 0.0\n",
      "0.0001577816146891564 0.0\n",
      "5.73908728256356e-05 0.0\n",
      "0.9868905544281006 1.0\n",
      "0.00033526765764690936 0.0\n",
      "0.9992048144340515 1.0\n",
      "0.00029296951834112406 0.0\n",
      "0.9988527297973633 1.0\n",
      "0.998175859451294 1.0\n",
      "0.9024590849876404 1.0\n",
      "0.00019824922492261976 0.0\n",
      "0.9997966885566711 1.0\n",
      "0.9234375357627869 1.0\n",
      "0.9860562086105347 1.0\n",
      "6.730345194227993e-05 0.0\n",
      "5.794045500806533e-05 0.0\n",
      "0.011442537419497967 0.0\n",
      "7.386793731711805e-05 0.0\n",
      "0.9973461627960205 1.0\n",
      "0.08945658802986145 0.0\n",
      "0.9971206188201904 1.0\n",
      "0.9999490976333618 1.0\n",
      "0.0030282035004347563 0.0\n",
      "0.0037770154885947704 0.0\n",
      "0.001872635679319501 0.0\n",
      "0.032584771513938904 0.0\n",
      "0.00016714580124244094 0.0\n",
      "0.03793025389313698 0.0\n",
      "0.9992567896842957 1.0\n",
      "0.00012992734264116734 0.0\n",
      "0.0010506800608709455 0.0\n",
      "0.007646532729268074 0.0\n",
      "0.943166971206665 1.0\n",
      "0.00022214993077795953 0.0\n",
      "0.9990811347961426 1.0\n",
      "0.017031317576766014 0.0\n",
      "TRAIN[steps=5800] loss=0.068057 acc=0.984 P=1.000 R=0.967 F1=0.983051\n",
      "DEV[steps=5800] loss=0.019910 acc=0.696 P=0.687 R=0.695 F1=0.690779 \n",
      "model sim and label tuples:\n",
      "0.9997608065605164 1.0\n",
      "4.4982738472754136e-05 0.0\n",
      "0.999907374382019 1.0\n",
      "0.0007641729316674173 0.0\n",
      "0.005176527425646782 0.0\n",
      "5.214082557358779e-05 0.0\n",
      "0.9999353885650635 1.0\n",
      "7.142862159525976e-05 0.0\n",
      "0.04949789494276047 0.0\n",
      "5.5978747695917264e-05 0.0\n",
      "0.0007024438236840069 0.0\n",
      "0.07875088602304459 1.0\n",
      "0.9999568462371826 1.0\n",
      "0.00014420980005525053 0.0\n",
      "0.00034196008346043527 0.0\n",
      "5.04224663018249e-05 0.0\n",
      "6.705604755552486e-05 0.0\n",
      "8.358000923180953e-05 0.0\n",
      "9.338258678326383e-05 0.0\n",
      "6.03002481511794e-05 0.0\n",
      "0.9990156888961792 1.0\n",
      "0.9969289898872375 1.0\n",
      "0.9983853101730347 1.0\n",
      "0.995306670665741 1.0\n",
      "0.9969700574874878 1.0\n",
      "7.476583414245397e-05 0.0\n",
      "0.0006541315233334899 0.0\n",
      "0.0023365796077996492 0.0\n",
      "7.560344965895638e-05 0.0\n",
      "0.9999250173568726 1.0\n",
      "0.999902606010437 1.0\n",
      "0.9999337196350098 1.0\n",
      "0.010218830779194832 0.0\n",
      "0.0904136672616005 0.0\n",
      "0.9999567270278931 1.0\n",
      "5.573993985308334e-05 0.0\n",
      "0.01823710836470127 0.0\n",
      "6.865889008622617e-05 0.0\n",
      "0.9994120597839355 1.0\n",
      "0.9999314546585083 1.0\n",
      "0.9930579662322998 1.0\n",
      "0.8355165719985962 1.0\n",
      "0.9945303797721863 1.0\n",
      "0.0341094508767128 0.0\n",
      "0.0001165535650216043 0.0\n",
      "0.9994587302207947 1.0\n",
      "0.0004877343599218875 0.0\n",
      "0.9996515512466431 1.0\n",
      "0.9705599546432495 1.0\n",
      "0.0006816942477598786 0.0\n",
      "0.99962317943573 1.0\n",
      "0.9957594275474548 1.0\n",
      "0.9993163347244263 1.0\n",
      "0.022018173709511757 0.0\n",
      "0.809086799621582 0.0\n",
      "6.0880236560478806e-05 0.0\n",
      "0.9989200830459595 1.0\n",
      "0.9983803033828735 1.0\n",
      "0.9989715814590454 1.0\n",
      "0.13601437211036682 0.0\n",
      "4.494702079682611e-05 0.0\n",
      "0.0015927576459944248 0.0\n",
      "0.38427606225013733 0.0\n",
      "0.9682895541191101 1.0\n",
      "TRAIN[steps=5900] loss=0.083636 acc=0.969 P=0.966 R=0.966 F1=0.965517\n",
      "DEV[steps=5900] loss=0.020532 acc=0.695 P=0.683 R=0.702 F1=0.692129 \n",
      "model sim and label tuples:\n",
      "6.321852561086416e-05 0.0\n",
      "0.9987553358078003 1.0\n",
      "0.0010565746342763305 0.0\n",
      "5.7573419326217845e-05 0.0\n",
      "0.019078882411122322 0.0\n",
      "0.9997138381004333 1.0\n",
      "0.9994295239448547 1.0\n",
      "0.9988347887992859 1.0\n",
      "0.9999595880508423 1.0\n",
      "0.9987769722938538 1.0\n",
      "4.171734690316953e-05 0.0\n",
      "0.0001283143210457638 0.0\n",
      "0.007271387614309788 0.0\n",
      "8.70747389853932e-05 0.0\n",
      "0.9999598264694214 1.0\n",
      "0.01898142322897911 0.0\n",
      "0.00045686954399570823 0.0\n",
      "0.9999605417251587 1.0\n",
      "0.003132912563160062 0.0\n",
      "4.049628842039965e-05 0.0\n",
      "0.9646663069725037 1.0\n",
      "0.009603595361113548 0.0\n",
      "0.9999600648880005 1.0\n",
      "0.9992042183876038 1.0\n",
      "8.475896902382374e-05 0.0\n",
      "0.9998699426651001 1.0\n",
      "0.9998313188552856 1.0\n",
      "5.639194569084793e-05 0.0\n",
      "0.0029805521480739117 0.0\n",
      "0.022199846804142 0.0\n",
      "8.529157639713958e-05 0.0\n",
      "0.9524543285369873 1.0\n",
      "0.012749341316521168 0.0\n",
      "4.0489492675988004e-05 0.0\n",
      "0.9998323917388916 1.0\n",
      "0.979622483253479 1.0\n",
      "0.9997245669364929 1.0\n",
      "6.918897270224988e-05 0.0\n",
      "0.0005879253149032593 0.0\n",
      "0.9998339414596558 1.0\n",
      "0.005589543376117945 0.0\n",
      "0.9650033116340637 1.0\n",
      "0.9985364675521851 1.0\n",
      "0.9997504353523254 1.0\n",
      "0.9989141225814819 1.0\n",
      "0.0001682415750110522 0.0\n",
      "0.00031527757528238 0.0\n",
      "0.9999469518661499 1.0\n",
      "5.270858673611656e-05 0.0\n",
      "0.9998733997344971 1.0\n",
      "0.9931069016456604 1.0\n",
      "0.0002487622550688684 0.0\n",
      "0.983693540096283 1.0\n",
      "0.03262721374630928 0.0\n",
      "0.9844937920570374 1.0\n",
      "0.0001716338738333434 0.0\n",
      "0.5277606844902039 1.0\n",
      "4.6338929678313434e-05 0.0\n",
      "0.9957093000411987 1.0\n",
      "0.9464632868766785 1.0\n",
      "9.368485916638747e-05 0.0\n",
      "0.01586904004216194 0.0\n",
      "0.99991774559021 1.0\n",
      "0.9934822916984558 1.0\n",
      "TRAIN[steps=6000] loss=0.016403 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=6000] loss=0.020702 acc=0.696 P=0.686 R=0.696 F1=0.690891 \n",
      "model sim and label tuples:\n",
      "0.00020102756388951093 0.0\n",
      "0.0013523097150027752 0.0\n",
      "3.780313636525534e-05 0.0\n",
      "0.9984583854675293 1.0\n",
      "8.433553739450872e-05 0.0\n",
      "0.9999006986618042 1.0\n",
      "6.778059469070286e-05 0.0\n",
      "0.20088151097297668 0.0\n",
      "7.347382052103058e-05 0.0\n",
      "9.270755253965035e-05 0.0\n",
      "0.9994157552719116 1.0\n",
      "0.9987748265266418 1.0\n",
      "0.07840477675199509 0.0\n",
      "0.0007118830690160394 0.0\n",
      "0.0009514009580016136 0.0\n",
      "0.0006636675680056214 0.0\n",
      "0.998839795589447 1.0\n",
      "0.05272512137889862 0.0\n",
      "0.0003892936510965228 0.0\n",
      "0.9999417066574097 1.0\n",
      "0.9723977446556091 1.0\n",
      "0.9547495245933533 1.0\n",
      "0.0001949829893419519 0.0\n",
      "0.9814136028289795 1.0\n",
      "0.9875237345695496 1.0\n",
      "0.00522620091214776 0.0\n",
      "0.9999527931213379 1.0\n",
      "3.908124199369922e-05 0.0\n",
      "0.999808132648468 1.0\n",
      "0.9999649524688721 1.0\n",
      "0.0021837116219103336 0.0\n",
      "0.9999650716781616 1.0\n",
      "3.8648900954285637e-05 0.0\n",
      "0.002996809547767043 0.0\n",
      "9.840820712270215e-05 0.0\n",
      "6.266757554840297e-05 0.0\n",
      "0.9993417859077454 1.0\n",
      "0.417553573846817 0.0\n",
      "0.999937891960144 1.0\n",
      "0.0001710589276626706 0.0\n",
      "0.9965656399726868 1.0\n",
      "0.0009017525590024889 0.0\n",
      "0.9996943473815918 1.0\n",
      "0.9998887777328491 1.0\n",
      "0.0013560742372646928 0.0\n",
      "0.009267667308449745 0.0\n",
      "0.00012709840666502714 0.0\n",
      "0.9997499585151672 1.0\n",
      "0.9998331069946289 1.0\n",
      "4.115160118089989e-05 0.0\n",
      "0.04434715583920479 0.0\n",
      "0.0007299775606952608 0.0\n",
      "0.999783456325531 1.0\n",
      "0.00043613999150693417 0.0\n",
      "0.9998985528945923 1.0\n",
      "0.00903436541557312 0.0\n",
      "0.9997898936271667 1.0\n",
      "3.845720857498236e-05 0.0\n",
      "0.9995428323745728 1.0\n",
      "0.9999440908432007 1.0\n",
      "0.9993808269500732 0.0\n",
      "0.09841091930866241 0.0\n",
      "0.9997479319572449 1.0\n",
      "0.9993417859077454 1.0\n",
      "TRAIN[steps=6100] loss=0.134248 acc=0.984 P=0.966 R=1.000 F1=0.982456\n",
      "DEV[steps=6100] loss=0.021217 acc=0.697 P=0.683 R=0.708 F1=0.695390 \n",
      "model sim and label tuples:\n",
      "0.9921482801437378 1.0\n",
      "0.9993070363998413 1.0\n",
      "0.9999628067016602 1.0\n",
      "0.99928218126297 1.0\n",
      "0.0010128450812771916 0.0\n",
      "0.999852180480957 1.0\n",
      "4.787205034517683e-05 0.0\n",
      "0.9999154806137085 1.0\n",
      "0.9892071485519409 1.0\n",
      "0.0007731941295787692 0.0\n",
      "0.00010101707448484376 0.0\n",
      "0.9983289837837219 1.0\n",
      "0.9922648668289185 1.0\n",
      "0.9766935706138611 1.0\n",
      "0.9985780715942383 1.0\n",
      "3.815663148998283e-05 0.0\n",
      "0.00013274022785481066 0.0\n",
      "0.999950647354126 1.0\n",
      "0.9998273849487305 1.0\n",
      "9.891664376482368e-05 0.0\n",
      "0.9705161452293396 1.0\n",
      "0.1931045800447464 0.0\n",
      "0.01013281662017107 0.0\n",
      "0.9885426759719849 1.0\n",
      "4.526174234342761e-05 0.0\n",
      "0.0011099475668743253 0.0\n",
      "0.9988659620285034 1.0\n",
      "0.9997462630271912 1.0\n",
      "0.9980288147926331 1.0\n",
      "0.00013554780161939561 0.0\n",
      "0.9998632669448853 1.0\n",
      "0.0026616505347192287 0.0\n",
      "0.9988963603973389 1.0\n",
      "0.9986953139305115 1.0\n",
      "0.999921441078186 1.0\n",
      "0.9996644258499146 1.0\n",
      "8.592703670728952e-05 0.0\n",
      "0.9934080839157104 1.0\n",
      "0.9999338388442993 1.0\n",
      "0.016321245580911636 0.0\n",
      "0.00028157036285847425 0.0\n",
      "0.07179734110832214 0.0\n",
      "3.553947317413986e-05 0.0\n",
      "0.9947879314422607 1.0\n",
      "0.9994274377822876 1.0\n",
      "4.6545053919544443e-05 0.0\n",
      "3.622384610935114e-05 0.0\n",
      "0.9999624490737915 1.0\n",
      "0.9998866319656372 1.0\n",
      "0.004889320116490126 0.0\n",
      "0.04979359358549118 0.0\n",
      "0.0002278511819895357 0.0\n",
      "0.01767699047923088 0.0\n",
      "3.8445217796834186e-05 0.0\n",
      "0.9740779995918274 1.0\n",
      "0.001306383404880762 0.0\n",
      "0.0003045409685000777 0.0\n",
      "3.367698809597641e-05 0.0\n",
      "0.07218818366527557 0.0\n",
      "0.9990887641906738 1.0\n",
      "3.6094126699026674e-05 0.0\n",
      "5.358184716897085e-05 0.0\n",
      "6.959108577575535e-05 0.0\n",
      "3.654030297184363e-05 0.0\n",
      "TRAIN[steps=6200] loss=0.009623 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=6200] loss=0.021416 acc=0.697 P=0.684 R=0.706 F1=0.694713 \n",
      "EPOCH:  10\n",
      "model sim and label tuples:\n",
      "0.9989157915115356 1.0\n",
      "0.9968602657318115 1.0\n",
      "0.010799646377563477 0.0\n",
      "0.909070611000061 1.0\n",
      "0.7225667238235474 1.0\n",
      "0.0002646102220751345 0.0\n",
      "0.01775652915239334 0.0\n",
      "0.9964266419410706 1.0\n",
      "4.7690700739622116e-05 0.0\n",
      "0.0007811376126483083 0.0\n",
      "0.8642817735671997 1.0\n",
      "0.005149385426193476 0.0\n",
      "0.0007655920926481485 0.0\n",
      "0.05596378445625305 0.0\n",
      "0.00018185852968599647 0.0\n",
      "0.0006758676609024405 0.0\n",
      "0.9999591112136841 1.0\n",
      "4.7685101890238e-05 0.0\n",
      "0.9996551275253296 1.0\n",
      "0.6869312524795532 1.0\n",
      "0.9996964931488037 1.0\n",
      "0.008870105259120464 0.0\n",
      "0.00034014758421108127 0.0\n",
      "8.911644545150921e-05 0.0\n",
      "0.9976697564125061 1.0\n",
      "0.9975185394287109 1.0\n",
      "0.0008022816618904471 0.0\n",
      "0.000462897151010111 0.0\n",
      "0.9999614953994751 1.0\n",
      "0.04828488826751709 0.0\n",
      "0.34363824129104614 1.0\n",
      "5.477371087181382e-05 0.0\n",
      "0.93877774477005 1.0\n",
      "0.9991655349731445 1.0\n",
      "0.9990977048873901 1.0\n",
      "6.960402970435098e-05 0.0\n",
      "0.9999669790267944 1.0\n",
      "9.817361569730565e-05 0.0\n",
      "0.00015074525435920805 0.0\n",
      "0.9938353300094604 1.0\n",
      "0.9945429563522339 1.0\n",
      "0.998293936252594 1.0\n",
      "0.9995569586753845 1.0\n",
      "0.0007903124787844718 0.0\n",
      "0.050677016377449036 0.0\n",
      "0.984609842300415 1.0\n",
      "0.9997164607048035 1.0\n",
      "0.999904990196228 1.0\n",
      "0.9997379183769226 1.0\n",
      "3.592835855670273e-05 0.0\n",
      "0.04027705639600754 0.0\n",
      "0.0033649171236902475 0.0\n",
      "0.9115654230117798 1.0\n",
      "0.07719793170690536 0.0\n",
      "0.008182552643120289 0.0\n",
      "0.10693609714508057 0.0\n",
      "4.2076895624632016e-05 0.0\n",
      "0.006582240574061871 0.0\n",
      "2.853714977391064e-05 0.0\n",
      "0.9308241605758667 1.0\n",
      "2.908227725129109e-05 0.0\n",
      "0.9999450445175171 1.0\n",
      "0.0001754348340909928 0.0\n",
      "0.9999358654022217 1.0\n",
      "TRAIN[steps=6300] loss=0.042861 acc=0.984 P=1.000 R=0.967 F1=0.983051\n",
      "DEV[steps=6300] loss=0.022120 acc=0.698 P=0.687 R=0.699 F1=0.693285 \n",
      "model sim and label tuples:\n",
      "0.00021582530462183058 0.0\n",
      "0.9988794922828674 1.0\n",
      "0.9878709316253662 1.0\n",
      "0.8659521341323853 0.0\n",
      "0.9999165534973145 1.0\n",
      "0.9999728202819824 1.0\n",
      "0.9997702240943909 1.0\n",
      "0.999337375164032 1.0\n",
      "2.733923975029029e-05 0.0\n",
      "0.9994099140167236 1.0\n",
      "0.9998584985733032 1.0\n",
      "0.999976634979248 1.0\n",
      "0.00018885848112404346 0.0\n",
      "5.212998439674266e-05 0.0\n",
      "2.5565441319486126e-05 0.0\n",
      "0.9997268319129944 1.0\n",
      "8.038042869884521e-05 0.0\n",
      "0.9992806315422058 1.0\n",
      "0.9999767541885376 1.0\n",
      "3.903267133864574e-05 0.0\n",
      "0.9977163076400757 1.0\n",
      "0.4783201217651367 0.0\n",
      "0.9999715089797974 1.0\n",
      "0.00019243564747739583 0.0\n",
      "2.4256622054963373e-05 0.0\n",
      "0.9935684204101562 1.0\n",
      "0.00012230491847731173 0.0\n",
      "3.545991421560757e-05 0.0\n",
      "0.9999293088912964 1.0\n",
      "0.9996248483657837 1.0\n",
      "0.00010247901809634641 0.0\n",
      "3.259637742303312e-05 0.0\n",
      "0.9999741315841675 1.0\n",
      "3.081339309574105e-05 0.0\n",
      "3.399604975129478e-05 0.0\n",
      "0.9993299245834351 1.0\n",
      "0.0002295217855134979 0.0\n",
      "0.00019102287478744984 0.0\n",
      "2.8031785404891707e-05 0.0\n",
      "0.9962154030799866 1.0\n",
      "0.9983752965927124 1.0\n",
      "0.9034696221351624 1.0\n",
      "0.9994090795516968 1.0\n",
      "0.00028774046222679317 0.0\n",
      "0.99342280626297 1.0\n",
      "0.998684823513031 1.0\n",
      "0.00022367850760929286 0.0\n",
      "0.9998961687088013 1.0\n",
      "0.999976634979248 1.0\n",
      "3.094360727118328e-05 0.0\n",
      "0.9999614953994751 1.0\n",
      "0.001142853987403214 0.0\n",
      "0.9993675351142883 1.0\n",
      "0.9996927976608276 1.0\n",
      "0.9997987151145935 1.0\n",
      "0.0013634966453537345 0.0\n",
      "0.9998679161071777 1.0\n",
      "0.9997414946556091 1.0\n",
      "0.0019311686046421528 0.0\n",
      "0.0005692013655789196 0.0\n",
      "0.999772846698761 1.0\n",
      "0.9999758005142212 1.0\n",
      "0.9077135920524597 1.0\n",
      "0.9868258833885193 1.0\n",
      "TRAIN[steps=6400] loss=0.045640 acc=0.984 P=0.974 R=1.000 F1=0.986667\n",
      "DEV[steps=6400] loss=0.022836 acc=0.700 P=0.695 R=0.687 F1=0.690935 \n",
      "model sim and label tuples:\n",
      "6.722057878505439e-05 0.0\n",
      "0.004644147120416164 0.0\n",
      "0.9990436434745789 1.0\n",
      "0.00014169224596116692 0.0\n",
      "8.533095387974754e-05 0.0\n",
      "5.5192915169755e-05 0.0\n",
      "0.9798250794410706 1.0\n",
      "0.999901533126831 1.0\n",
      "2.5003782866406254e-05 0.0\n",
      "2.2936232198844664e-05 0.0\n",
      "2.4526374545530416e-05 0.0\n",
      "0.9999785423278809 1.0\n",
      "0.9874075651168823 1.0\n",
      "2.480809598637279e-05 0.0\n",
      "2.5157696654787287e-05 0.0\n",
      "0.9993643164634705 1.0\n",
      "0.9999665021896362 1.0\n",
      "0.001101543428376317 0.0\n",
      "0.09952308237552643 0.0\n",
      "2.915986806328874e-05 0.0\n",
      "0.00012804994184989482 0.0\n",
      "0.9860256314277649 1.0\n",
      "2.1948695575702004e-05 0.0\n",
      "0.999945878982544 1.0\n",
      "2.229683923360426e-05 0.0\n",
      "5.299264012137428e-05 0.0\n",
      "0.9997599720954895 1.0\n",
      "2.166974991268944e-05 0.0\n",
      "7.049681153148413e-05 0.0\n",
      "3.90357272408437e-05 0.0\n",
      "0.0003294766938779503 0.0\n",
      "0.9746920466423035 1.0\n",
      "0.9999560117721558 1.0\n",
      "0.9999785423278809 1.0\n",
      "9.123306517722085e-05 0.0\n",
      "0.9962427616119385 1.0\n",
      "0.9691242575645447 1.0\n",
      "0.9993060827255249 1.0\n",
      "0.999323844909668 1.0\n",
      "2.2301901481114328e-05 0.0\n",
      "0.999884843826294 1.0\n",
      "0.9966071844100952 1.0\n",
      "0.025393258780241013 0.0\n",
      "2.417915857222397e-05 0.0\n",
      "0.9995019435882568 1.0\n",
      "0.0018546917708590627 0.0\n",
      "0.9999134540557861 1.0\n",
      "0.0017664647893980145 0.0\n",
      "2.167189995816443e-05 0.0\n",
      "6.256224878598005e-05 0.0\n",
      "0.9998512268066406 1.0\n",
      "0.0009758784435689449 0.0\n",
      "0.000607931287959218 0.0\n",
      "0.0002117993135470897 0.0\n",
      "0.010155162774026394 0.0\n",
      "0.9998465776443481 1.0\n",
      "2.8847352950833738e-05 0.0\n",
      "0.0028738665860146284 0.0\n",
      "0.00012651282304432243 0.0\n",
      "0.0005776156322099268 0.0\n",
      "0.9998579025268555 1.0\n",
      "0.00021980643214192241 0.0\n",
      "0.9740129709243774 1.0\n",
      "2.2008558516972698e-05 0.0\n",
      "TRAIN[steps=6500] loss=0.004679 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=6500] loss=0.023332 acc=0.698 P=0.688 R=0.699 F1=0.693521 \n",
      "model sim and label tuples:\n",
      "0.9980936646461487 1.0\n",
      "0.9990348815917969 1.0\n",
      "2.0134610167588107e-05 0.0\n",
      "0.9998651742935181 1.0\n",
      "0.0005879743839614093 0.0\n",
      "2.9469423679984175e-05 0.0\n",
      "0.9979021549224854 1.0\n",
      "2.92155073111644e-05 0.0\n",
      "0.9186622500419617 1.0\n",
      "0.00025485717924311757 0.0\n",
      "5.834131661686115e-05 0.0\n",
      "0.9911905527114868 1.0\n",
      "9.039582073455676e-05 0.0\n",
      "0.0030112494714558125 0.0\n",
      "1.9906832676497288e-05 0.0\n",
      "9.085947385756299e-05 0.0\n",
      "0.9999768733978271 1.0\n",
      "0.9991924166679382 1.0\n",
      "0.00016160510131157935 0.0\n",
      "0.0008071855409070849 0.0\n",
      "2.134307760570664e-05 0.0\n",
      "0.9182888269424438 1.0\n",
      "0.0004314961552154273 0.0\n",
      "0.9472479224205017 1.0\n",
      "0.9985629916191101 1.0\n",
      "0.9998262524604797 1.0\n",
      "0.0008814831380732358 0.0\n",
      "3.010843829542864e-05 0.0\n",
      "2.6753305064630695e-05 0.0\n",
      "0.8812839388847351 1.0\n",
      "0.9999715089797974 1.0\n",
      "0.9999793767929077 1.0\n",
      "2.022596527240239e-05 0.0\n",
      "0.0046615502797067165 0.0\n",
      "0.999910831451416 1.0\n",
      "2.6675372282625176e-05 0.0\n",
      "0.028691770508885384 0.0\n",
      "2.583787136245519e-05 0.0\n",
      "0.00017994630616158247 0.0\n",
      "0.9909365773200989 1.0\n",
      "0.031908340752124786 0.0\n",
      "0.03194672614336014 0.0\n",
      "2.754835077212192e-05 0.0\n",
      "2.1413579815998673e-05 0.0\n",
      "0.9999423027038574 1.0\n",
      "0.999974250793457 1.0\n",
      "0.9999713897705078 1.0\n",
      "0.005212339106947184 0.0\n",
      "2.5131605070782825e-05 0.0\n",
      "0.00014871478197164834 0.0\n",
      "0.9999805688858032 1.0\n",
      "0.9994756579399109 1.0\n",
      "2.675386713235639e-05 0.0\n",
      "2.0184170352877118e-05 0.0\n",
      "0.000168231621501036 0.0\n",
      "2.0318433598731644e-05 0.0\n",
      "7.976606138981879e-05 0.0\n",
      "0.9993342757225037 1.0\n",
      "0.9670278429985046 1.0\n",
      "9.295808558817953e-05 0.0\n",
      "0.9850693345069885 1.0\n",
      "0.9999725818634033 1.0\n",
      "0.9971774816513062 1.0\n",
      "0.9918404817581177 1.0\n",
      "TRAIN[steps=6600] loss=0.008572 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=6600] loss=0.023423 acc=0.699 P=0.691 R=0.694 F1=0.692559 \n",
      "model sim and label tuples:\n",
      "0.9993703961372375 1.0\n",
      "0.9963963627815247 1.0\n",
      "1.81392279046122e-05 0.0\n",
      "0.9998941421508789 1.0\n",
      "0.9990221261978149 1.0\n",
      "0.00013912930444348603 0.0\n",
      "0.9999803304672241 1.0\n",
      "2.1540585294133052e-05 0.0\n",
      "0.9933285713195801 1.0\n",
      "0.0467844121158123 0.0\n",
      "0.9999725818634033 1.0\n",
      "0.9999734163284302 1.0\n",
      "0.00023644919565413147 0.0\n",
      "1.8209264453616925e-05 0.0\n",
      "0.00012089919619029388 0.0\n",
      "0.9992904663085938 1.0\n",
      "0.0007737432024441659 0.0\n",
      "0.0018385391449555755 0.0\n",
      "0.999134361743927 1.0\n",
      "2.427122126391623e-05 0.0\n",
      "0.9997000694274902 1.0\n",
      "0.9999754428863525 1.0\n",
      "2.6605506718624383e-05 0.0\n",
      "0.0006670553702861071 0.0\n",
      "0.03399941697716713 0.0\n",
      "0.9997139573097229 1.0\n",
      "0.08451341092586517 1.0\n",
      "0.00020434842735994607 0.0\n",
      "0.0008788866107352078 0.0\n",
      "0.9999545812606812 1.0\n",
      "0.9999483823776245 1.0\n",
      "0.9999743700027466 1.0\n",
      "0.00021811244369018823 0.0\n",
      "0.9967999458312988 1.0\n",
      "3.4898162994068116e-05 0.0\n",
      "0.9998588562011719 1.0\n",
      "0.0015196900349110365 0.0\n",
      "9.0210342023056e-05 0.0\n",
      "8.277374581666663e-05 0.0\n",
      "0.006719255354255438 0.0\n",
      "0.02062634937465191 0.0\n",
      "0.9976606369018555 1.0\n",
      "0.0010284577729180455 0.0\n",
      "7.829970854800195e-05 0.0\n",
      "0.9996383190155029 1.0\n",
      "0.9996588230133057 1.0\n",
      "0.00015527983487118036 0.0\n",
      "0.00036222953349351883 0.0\n",
      "0.0058151776902377605 0.0\n",
      "0.9974955916404724 1.0\n",
      "0.9992013573646545 1.0\n",
      "0.000142016593599692 0.0\n",
      "0.9999822378158569 1.0\n",
      "0.9999790191650391 1.0\n",
      "0.00010232414933852851 0.0\n",
      "0.9973491430282593 1.0\n",
      "0.9999128580093384 1.0\n",
      "0.00016498740296810865 0.0\n",
      "6.611025310121477e-05 0.0\n",
      "0.996956467628479 1.0\n",
      "0.9999620914459229 1.0\n",
      "1.9494906155159697e-05 0.0\n",
      "6.279020453803241e-05 0.0\n",
      "0.9994089603424072 1.0\n",
      "TRAIN[steps=6700] loss=0.041038 acc=0.984 P=1.000 R=0.968 F1=0.983607\n",
      "DEV[steps=6700] loss=0.023790 acc=0.701 P=0.690 R=0.706 F1=0.698065 \n",
      "model sim and label tuples:\n",
      "0.002058460609987378 0.0\n",
      "0.9997687935829163 1.0\n",
      "0.10338550060987473 0.0\n",
      "0.9999277591705322 1.0\n",
      "0.9997326731681824 1.0\n",
      "0.0002651031536515802 0.0\n",
      "1.8884025848819874e-05 0.0\n",
      "0.0037524604704231024 0.0\n",
      "0.02850012667477131 0.0\n",
      "0.9999816417694092 1.0\n",
      "1.5741690731374547e-05 0.0\n",
      "0.00045697734458371997 0.0\n",
      "0.8318679332733154 0.0\n",
      "0.9998390674591064 1.0\n",
      "0.00048118529957719147 0.0\n",
      "0.9999748468399048 1.0\n",
      "0.9999836683273315 1.0\n",
      "0.00044570397585630417 0.0\n",
      "0.9999802112579346 1.0\n",
      "0.001370657584629953 0.0\n",
      "1.5985222489689477e-05 0.0\n",
      "2.672342088771984e-05 0.0\n",
      "0.9996272325515747 1.0\n",
      "0.9652155041694641 1.0\n",
      "0.0014823221135884523 0.0\n",
      "1.805965439416468e-05 0.0\n",
      "0.00480355741456151 0.0\n",
      "0.9999834299087524 1.0\n",
      "1.882262404251378e-05 0.0\n",
      "0.9999700784683228 1.0\n",
      "0.9891353845596313 1.0\n",
      "2.74745198112214e-05 0.0\n",
      "3.614593151723966e-05 0.0\n",
      "2.946141285065096e-05 0.0\n",
      "1.9543993403203785e-05 0.0\n",
      "0.00019499563495628536 0.0\n",
      "0.9998812675476074 1.0\n",
      "0.9999241828918457 1.0\n",
      "0.9979128241539001 1.0\n",
      "1.576007707626559e-05 0.0\n",
      "0.9848781228065491 1.0\n",
      "0.9979501366615295 1.0\n",
      "0.00010122419917024672 0.0\n",
      "0.9979307651519775 1.0\n",
      "0.9998935461044312 1.0\n",
      "0.9998725652694702 1.0\n",
      "0.0016726776957511902 0.0\n",
      "0.00032365485094487667 0.0\n",
      "0.8884780406951904 1.0\n",
      "0.9999445676803589 1.0\n",
      "0.9988987445831299 1.0\n",
      "0.16704308986663818 0.0\n",
      "0.0004630719486158341 0.0\n",
      "1.809910281735938e-05 0.0\n",
      "0.02657121978700161 0.0\n",
      "0.3672887086868286 1.0\n",
      "0.0011068539461120963 0.0\n",
      "7.029528205748647e-05 0.0\n",
      "0.9997350573539734 1.0\n",
      "0.003608537605032325 0.0\n",
      "0.00010927149560302496 0.0\n",
      "0.9999159574508667 1.0\n",
      "2.064910404442344e-05 0.0\n",
      "0.0002882094122469425 0.0\n",
      "TRAIN[steps=6800] loss=0.052264 acc=0.969 P=0.963 R=0.963 F1=0.962963\n",
      "DEV[steps=6800] loss=0.024185 acc=0.698 P=0.686 R=0.703 F1=0.694543 \n",
      "EPOCH:  11\n",
      "model sim and label tuples:\n",
      "0.005296335089951754 0.0\n",
      "1.4017005923960824e-05 0.0\n",
      "0.003855621675029397 0.0\n",
      "2.6339399482822046e-05 0.0\n",
      "0.03899876028299332 0.0\n",
      "1.8912860468844883e-05 0.0\n",
      "7.378669397439808e-05 0.0\n",
      "0.007364396471530199 0.0\n",
      "0.9879468083381653 1.0\n",
      "0.9999768733978271 1.0\n",
      "6.26578985247761e-05 0.0\n",
      "1.3997701898915693e-05 0.0\n",
      "0.0005543742445297539 0.0\n",
      "3.720472886925563e-05 0.0\n",
      "5.045175566920079e-05 0.0\n",
      "0.0008657516445964575 0.0\n",
      "7.055747119011357e-05 0.0\n",
      "1.4630686564487405e-05 0.0\n",
      "8.564260497223586e-05 0.0\n",
      "1.7514948922325857e-05 0.0\n",
      "0.002345103770494461 0.0\n",
      "0.999962568283081 1.0\n",
      "0.989828884601593 1.0\n",
      "1.539153527119197e-05 0.0\n",
      "5.087522367830388e-05 0.0\n",
      "2.6206156690022908e-05 0.0\n",
      "3.54081166733522e-05 0.0\n",
      "2.4229753762483597e-05 0.0\n",
      "0.9999253749847412 1.0\n",
      "0.000568730931263417 0.0\n",
      "9.828207839746028e-05 0.0\n",
      "0.03760114684700966 0.0\n",
      "0.0018059691647067666 0.0\n",
      "0.9956450462341309 1.0\n",
      "1.4728555470355786e-05 0.0\n",
      "6.170777487568557e-05 0.0\n",
      "0.9747769832611084 1.0\n",
      "2.5420811653020792e-05 0.0\n",
      "0.00013697573740500957 0.0\n",
      "0.9993187189102173 1.0\n",
      "0.0008080820553004742 0.0\n",
      "0.9999575614929199 1.0\n",
      "1.747148962749634e-05 0.0\n",
      "0.9999171495437622 1.0\n",
      "4.916804391541518e-05 0.0\n",
      "1.8342387193115428e-05 0.0\n",
      "0.9999741315841675 1.0\n",
      "0.9984838366508484 1.0\n",
      "0.0002902117557823658 0.0\n",
      "2.0910583771183155e-05 0.0\n",
      "0.00018550960521679372 0.0\n",
      "1.3783492249785922e-05 0.0\n",
      "1.4933852071408182e-05 0.0\n",
      "0.9999805688858032 1.0\n",
      "0.9995920062065125 1.0\n",
      "1.4122416359896306e-05 0.0\n",
      "1.6726973626646213e-05 0.0\n",
      "2.4416200176347047e-05 0.0\n",
      "0.0002488151367288083 0.0\n",
      "0.9999332427978516 1.0\n",
      "1.3964755453343969e-05 0.0\n",
      "0.013959450647234917 0.0\n",
      "0.999683141708374 1.0\n",
      "0.9897360801696777 1.0\n",
      "TRAIN[steps=6900] loss=0.002867 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=6900] loss=0.024741 acc=0.699 P=0.691 R=0.694 F1=0.692449 \n",
      "model sim and label tuples:\n",
      "0.9999699592590332 1.0\n",
      "0.0008699126192368567 0.0\n",
      "1.2488373613450676e-05 0.0\n",
      "0.9999434947967529 1.0\n",
      "0.00019556166080292314 0.0\n",
      "0.9983622431755066 1.0\n",
      "0.9999797344207764 1.0\n",
      "1.8656372049008496e-05 0.0\n",
      "0.466931015253067 1.0\n",
      "0.9999029636383057 1.0\n",
      "0.989006757736206 1.0\n",
      "0.999675989151001 1.0\n",
      "0.00043139231274835765 0.0\n",
      "9.795143705559894e-05 0.0\n",
      "0.9975232481956482 1.0\n",
      "0.0006155017763376236 0.0\n",
      "0.004179482348263264 0.0\n",
      "0.2699858844280243 0.0\n",
      "1.3668301107827574e-05 0.0\n",
      "0.9996292591094971 1.0\n",
      "1.735114165057894e-05 0.0\n",
      "0.9989098310470581 1.0\n",
      "1.4818867384747136e-05 0.0\n",
      "4.423882273840718e-05 0.0\n",
      "0.9996908903121948 1.0\n",
      "0.9999873638153076 1.0\n",
      "0.9999570846557617 1.0\n",
      "1.4440984159591608e-05 0.0\n",
      "3.8442503864644095e-05 0.0\n",
      "1.2709736438409891e-05 0.0\n",
      "0.999977707862854 1.0\n",
      "0.9999713897705078 1.0\n",
      "1.3678093637281563e-05 0.0\n",
      "1.3172780199965928e-05 0.0\n",
      "0.9999462366104126 1.0\n",
      "0.007134501822292805 0.0\n",
      "0.9865133166313171 1.0\n",
      "0.9996144771575928 1.0\n",
      "0.9961666464805603 1.0\n",
      "0.00019130558939650655 0.0\n",
      "1.7313835996901616e-05 0.0\n",
      "4.7029148845467716e-05 0.0\n",
      "0.0001467052206862718 0.0\n",
      "2.3361179046332836e-05 0.0\n",
      "0.9992702603340149 1.0\n",
      "0.9998689889907837 1.0\n",
      "0.0001253522204933688 0.0\n",
      "0.9999082088470459 1.0\n",
      "0.0020980690605938435 0.0\n",
      "0.0006692637107335031 0.0\n",
      "0.9998193383216858 1.0\n",
      "1.544429142086301e-05 0.0\n",
      "0.9991915822029114 1.0\n",
      "0.9995125532150269 1.0\n",
      "0.9999426603317261 1.0\n",
      "1.3293273696035612e-05 0.0\n",
      "9.183437214232981e-05 0.0\n",
      "0.9999592304229736 1.0\n",
      "0.9999741315841675 1.0\n",
      "0.9999632835388184 1.0\n",
      "0.9999150037765503 1.0\n",
      "0.9999594688415527 1.0\n",
      "0.0010690093040466309 0.0\n",
      "2.235023202956654e-05 0.0\n",
      "TRAIN[steps=7000] loss=0.017699 acc=0.984 P=1.000 R=0.969 F1=0.984127\n",
      "DEV[steps=7000] loss=0.025109 acc=0.700 P=0.693 R=0.693 F1=0.692962 \n",
      "model sim and label tuples:\n",
      "1.2750407222483773e-05 0.0\n",
      "0.9999208450317383 1.0\n",
      "2.0120980479987338e-05 0.0\n",
      "0.96833336353302 1.0\n",
      "0.9998615980148315 1.0\n",
      "0.9999599456787109 1.0\n",
      "0.9999306201934814 1.0\n",
      "0.01407085545361042 0.0\n",
      "0.0017701933393254876 0.0\n",
      "2.5639223167672753e-05 0.0\n",
      "0.005676512140780687 0.0\n",
      "0.0004915660829283297 0.0\n",
      "0.005360033828765154 0.0\n",
      "0.9952131509780884 1.0\n",
      "0.9998310804367065 1.0\n",
      "0.9493361115455627 1.0\n",
      "3.374211519258097e-05 0.0\n",
      "1.5233415979309939e-05 0.0\n",
      "0.1382349580526352 0.0\n",
      "0.03320982679724693 0.0\n",
      "0.9992607235908508 1.0\n",
      "0.999869704246521 1.0\n",
      "0.999875545501709 1.0\n",
      "0.9997715353965759 1.0\n",
      "1.0731606380431913e-05 0.0\n",
      "0.9994264841079712 1.0\n",
      "3.221014776499942e-05 0.0\n",
      "1.1444981282693334e-05 0.0\n",
      "9.11647075554356e-05 0.0\n",
      "0.004137239884585142 0.0\n",
      "1.071635870175669e-05 0.0\n",
      "0.9999768733978271 1.0\n",
      "4.565253766486421e-05 0.0\n",
      "0.9983362555503845 1.0\n",
      "0.9992287158966064 1.0\n",
      "0.9997521042823792 1.0\n",
      "1.2169708497822285e-05 0.0\n",
      "0.9999713897705078 1.0\n",
      "0.99964439868927 1.0\n",
      "0.006633532699197531 0.0\n",
      "0.7590715885162354 1.0\n",
      "0.9969391822814941 1.0\n",
      "0.9999816417694092 1.0\n",
      "3.716302308021113e-05 0.0\n",
      "0.9999861717224121 1.0\n",
      "0.9931573271751404 1.0\n",
      "1.1189908946107607e-05 0.0\n",
      "0.9999887943267822 1.0\n",
      "1.0650551303115208e-05 0.0\n",
      "0.000692626170348376 0.0\n",
      "4.676764729083516e-05 0.0\n",
      "0.9999743700027466 1.0\n",
      "0.9999892711639404 1.0\n",
      "0.9999477863311768 1.0\n",
      "0.00019956637697760016 0.0\n",
      "0.00026116741355508566 0.0\n",
      "0.0002654633135534823 0.0\n",
      "1.9564748072298244e-05 0.0\n",
      "2.3077638616086915e-05 0.0\n",
      "2.734703775786329e-05 0.0\n",
      "0.9989897608757019 1.0\n",
      "0.9998906850814819 1.0\n",
      "0.003841760801151395 0.0\n",
      "1.0848842066479847e-05 0.0\n",
      "TRAIN[steps=7100] loss=0.009497 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7100] loss=0.025999 acc=0.699 P=0.686 R=0.708 F1=0.696863 \n",
      "model sim and label tuples:\n",
      "2.9520475436584093e-05 0.0\n",
      "0.0012795719085261226 0.0\n",
      "0.9997023940086365 1.0\n",
      "0.0006926951464265585 0.0\n",
      "0.9887163639068604 1.0\n",
      "1.0878773537115194e-05 0.0\n",
      "1.8323959011468105e-05 0.0\n",
      "0.00014015486522112042 0.0\n",
      "0.9997187256813049 1.0\n",
      "0.0001009469706332311 0.0\n",
      "0.9999626874923706 1.0\n",
      "0.999924898147583 1.0\n",
      "9.448469063499942e-05 0.0\n",
      "1.7516167645226233e-05 0.0\n",
      "0.006726298481225967 0.0\n",
      "0.08683226257562637 0.0\n",
      "0.9996601343154907 1.0\n",
      "0.9999861717224121 1.0\n",
      "1.140990116255125e-05 0.0\n",
      "0.0016241843113675714 0.0\n",
      "1.1134949090774171e-05 0.0\n",
      "0.9980775117874146 1.0\n",
      "0.9994058609008789 1.0\n",
      "0.9991796612739563 1.0\n",
      "0.9999781847000122 1.0\n",
      "0.9999661445617676 1.0\n",
      "0.9999897480010986 1.0\n",
      "9.86496434052242e-06 0.0\n",
      "0.9999364614486694 1.0\n",
      "1.5137100490392186e-05 0.0\n",
      "0.9999819993972778 1.0\n",
      "0.9935415983200073 1.0\n",
      "0.0020125790033489466 0.0\n",
      "0.9999804496765137 1.0\n",
      "0.00014166197797749192 0.0\n",
      "0.9922728538513184 1.0\n",
      "0.9999831914901733 1.0\n",
      "0.9676206111907959 1.0\n",
      "0.9974314570426941 1.0\n",
      "0.0004493860760703683 0.0\n",
      "0.9998999834060669 1.0\n",
      "4.314579928177409e-05 0.0\n",
      "0.9998738765716553 1.0\n",
      "0.9999899864196777 1.0\n",
      "1.0506752005312592e-05 0.0\n",
      "0.044534750282764435 0.0\n",
      "0.9999086856842041 1.0\n",
      "9.496913662587758e-06 0.0\n",
      "0.9901201725006104 1.0\n",
      "0.9999574422836304 1.0\n",
      "1.9522143702488393e-05 0.0\n",
      "0.9999741315841675 1.0\n",
      "0.0016459438484162092 0.0\n",
      "5.1853225158993155e-05 0.0\n",
      "9.61889872996835e-06 0.0\n",
      "0.9999434947967529 1.0\n",
      "0.9851058721542358 1.0\n",
      "0.0001367760414723307 0.0\n",
      "0.0003948885132558644 0.0\n",
      "0.9995585083961487 1.0\n",
      "1.6606574718025513e-05 0.0\n",
      "0.9982642531394958 1.0\n",
      "3.589250263758004e-05 0.0\n",
      "0.9999768733978271 1.0\n",
      "TRAIN[steps=7200] loss=0.003835 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7200] loss=0.026237 acc=0.700 P=0.691 R=0.697 F1=0.694011 \n",
      "model sim and label tuples:\n",
      "0.9998424053192139 1.0\n",
      "0.00010239148832624778 0.0\n",
      "0.9954420328140259 1.0\n",
      "0.9999270439147949 1.0\n",
      "1.808207707654219e-05 0.0\n",
      "5.7203629694413394e-05 0.0\n",
      "0.997735857963562 1.0\n",
      "2.735989801294636e-05 0.0\n",
      "0.9999605417251587 1.0\n",
      "0.9969244599342346 1.0\n",
      "0.9995323419570923 1.0\n",
      "0.9999167919158936 1.0\n",
      "1.4594956155633554e-05 0.0\n",
      "9.597090866009239e-06 0.0\n",
      "1.2361658264126163e-05 0.0\n",
      "0.998866081237793 1.0\n",
      "3.144938455079682e-05 0.0\n",
      "2.070032678602729e-05 0.0\n",
      "0.9997178912162781 1.0\n",
      "1.0523658602323849e-05 0.0\n",
      "8.732268724997994e-06 0.0\n",
      "0.0003501072060316801 0.0\n",
      "0.9999915361404419 1.0\n",
      "0.00019545743998605758 0.0\n",
      "0.0003446526825428009 0.0\n",
      "0.9974743723869324 1.0\n",
      "1.5242310837493278e-05 0.0\n",
      "0.9999895095825195 1.0\n",
      "0.9999563694000244 1.0\n",
      "0.02762250043451786 0.0\n",
      "0.9933868050575256 1.0\n",
      "0.9999784231185913 1.0\n",
      "6.725174898747355e-05 0.0\n",
      "0.00733080692589283 0.0\n",
      "0.0008354428573511541 0.0\n",
      "1.6210660760407336e-05 0.0\n",
      "0.9997419714927673 1.0\n",
      "0.999976634979248 1.0\n",
      "0.9933388829231262 1.0\n",
      "0.999991774559021 1.0\n",
      "0.999824583530426 1.0\n",
      "0.9999903440475464 1.0\n",
      "5.0861497584264725e-05 0.0\n",
      "0.0012498939177021384 0.0\n",
      "1.3634283277497161e-05 0.0\n",
      "0.9989161491394043 1.0\n",
      "0.0005720488843508065 0.0\n",
      "0.9998383522033691 1.0\n",
      "0.999972939491272 1.0\n",
      "0.9999285936355591 1.0\n",
      "0.0002459841198287904 0.0\n",
      "0.9998812675476074 1.0\n",
      "0.9996910095214844 1.0\n",
      "0.9915959239006042 1.0\n",
      "0.003351208521053195 0.0\n",
      "0.001398822758346796 0.0\n",
      "0.9999803304672241 1.0\n",
      "1.689690543571487e-05 0.0\n",
      "0.947859525680542 1.0\n",
      "0.9254363775253296 1.0\n",
      "0.9936864972114563 1.0\n",
      "2.0970453988411464e-05 0.0\n",
      "0.9999109506607056 1.0\n",
      "8.587702723161783e-06 0.0\n",
      "TRAIN[steps=7300] loss=0.003448 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7300] loss=0.026558 acc=0.703 P=0.701 R=0.685 F1=0.692802 \n",
      "model sim and label tuples:\n",
      "1.1685533536365256e-05 0.0\n",
      "0.9999914169311523 1.0\n",
      "2.2981314032222144e-05 0.0\n",
      "0.9998618364334106 1.0\n",
      "1.07687401396106e-05 0.0\n",
      "0.9994993209838867 1.0\n",
      "0.9993612170219421 1.0\n",
      "0.0010620802640914917 0.0\n",
      "1.2193152542749885e-05 0.0\n",
      "8.270407852251083e-05 0.0\n",
      "8.585131581639871e-06 0.0\n",
      "0.9999885559082031 1.0\n",
      "0.9949117302894592 1.0\n",
      "0.9999655485153198 1.0\n",
      "0.013109365478157997 0.0\n",
      "0.9996486902236938 1.0\n",
      "0.9963955283164978 1.0\n",
      "0.00038283347385004163 0.0\n",
      "0.9999923706054688 1.0\n",
      "0.9998595714569092 1.0\n",
      "0.9994051456451416 1.0\n",
      "0.00019198920927010477 0.0\n",
      "1.235693252965575e-05 0.0\n",
      "0.0005000336095690727 0.0\n",
      "0.9984226226806641 1.0\n",
      "0.9989664554595947 1.0\n",
      "0.00012202718062326312 0.0\n",
      "2.3790777049725875e-05 0.0\n",
      "0.9853532910346985 1.0\n",
      "0.9999876022338867 1.0\n",
      "9.324229722551536e-06 0.0\n",
      "0.0001605398574611172 0.0\n",
      "0.9978197813034058 1.0\n",
      "1.2085177331755403e-05 0.0\n",
      "0.999804675579071 1.0\n",
      "1.0942817425529938e-05 0.0\n",
      "1.712802804831881e-05 0.0\n",
      "6.075849523767829e-05 0.0\n",
      "0.9997631907463074 1.0\n",
      "0.9998486042022705 1.0\n",
      "2.615068297018297e-05 0.0\n",
      "1.2927070201840252e-05 0.0\n",
      "0.9999805688858032 1.0\n",
      "0.0008953443611972034 0.0\n",
      "8.997128134069499e-06 0.0\n",
      "0.9994776844978333 1.0\n",
      "0.0013630164321511984 0.0\n",
      "0.9999743700027466 1.0\n",
      "0.9996466636657715 1.0\n",
      "1.9221943148295395e-05 0.0\n",
      "4.7391386033268645e-05 0.0\n",
      "1.2276660527277272e-05 0.0\n",
      "0.99977046251297 1.0\n",
      "0.06143083795905113 0.0\n",
      "0.9998736381530762 1.0\n",
      "0.9998642206192017 1.0\n",
      "0.9999086856842041 1.0\n",
      "8.984814485302195e-05 0.0\n",
      "1.9515795429470018e-05 0.0\n",
      "0.9999842643737793 1.0\n",
      "0.999945878982544 1.0\n",
      "0.9999243021011353 1.0\n",
      "6.849546480225399e-05 0.0\n",
      "0.0003659830254036933 0.0\n",
      "TRAIN[steps=7400] loss=0.001800 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7400] loss=0.026755 acc=0.699 P=0.692 R=0.691 F1=0.691424 \n",
      "EPOCH:  12\n",
      "model sim and label tuples:\n",
      "0.0015334922354668379 0.0\n",
      "5.580381548497826e-05 0.0\n",
      "0.9999487400054932 1.0\n",
      "0.010061502456665039 0.0\n",
      "1.3930386558058672e-05 0.0\n",
      "0.9999926090240479 1.0\n",
      "0.0002298318868270144 0.0\n",
      "0.9995484948158264 1.0\n",
      "0.9996410608291626 1.0\n",
      "0.999617338180542 1.0\n",
      "6.735539500368759e-05 0.0\n",
      "9.443298040423542e-05 0.0\n",
      "0.9996167421340942 1.0\n",
      "0.9999685287475586 1.0\n",
      "8.760060154600069e-06 0.0\n",
      "5.6049895647447556e-05 0.0\n",
      "0.9995638728141785 1.0\n",
      "0.9999713897705078 1.0\n",
      "1.2551379768410698e-05 0.0\n",
      "0.9872855544090271 1.0\n",
      "0.999832034111023 1.0\n",
      "0.9999933242797852 1.0\n",
      "7.180363354564179e-06 0.0\n",
      "0.9887075424194336 1.0\n",
      "2.736289934546221e-05 0.0\n",
      "0.004995947238057852 0.0\n",
      "0.012570245191454887 0.0\n",
      "0.9999572038650513 1.0\n",
      "1.6309548300341703e-05 0.0\n",
      "0.9999933242797852 1.0\n",
      "7.1369986471836455e-06 0.0\n",
      "8.286154297820758e-06 0.0\n",
      "1.3472165846906137e-05 0.0\n",
      "0.9994623064994812 1.0\n",
      "0.9998979568481445 1.0\n",
      "0.9996185302734375 1.0\n",
      "4.240911584929563e-05 0.0\n",
      "0.999982476234436 1.0\n",
      "0.0006568089593201876 0.0\n",
      "5.7974557421403006e-05 0.0\n",
      "8.391221490455791e-05 0.0\n",
      "0.9977610111236572 1.0\n",
      "0.999930739402771 1.0\n",
      "0.999606192111969 1.0\n",
      "0.9999895095825195 1.0\n",
      "0.9997825026512146 1.0\n",
      "0.0005253436975181103 0.0\n",
      "0.9999070167541504 1.0\n",
      "3.824172745225951e-05 0.0\n",
      "0.020882895216345787 0.0\n",
      "0.9999284744262695 1.0\n",
      "2.29906272579683e-05 0.0\n",
      "0.999744713306427 1.0\n",
      "0.00015036610420793295 0.0\n",
      "0.9986748695373535 1.0\n",
      "0.9905092716217041 1.0\n",
      "0.9999650716781616 1.0\n",
      "1.524507206340786e-05 0.0\n",
      "0.9997546076774597 1.0\n",
      "8.154914212354925e-06 0.0\n",
      "0.9996994733810425 1.0\n",
      "2.8675976864178665e-05 0.0\n",
      "0.003274796763435006 0.0\n",
      "0.0316876657307148 0.0\n",
      "TRAIN[steps=7500] loss=0.002039 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7500] loss=0.027364 acc=0.701 P=0.690 R=0.703 F1=0.696588 \n",
      "model sim and label tuples:\n",
      "0.9999945163726807 1.0\n",
      "1.0191017281613313e-05 0.0\n",
      "0.999870777130127 1.0\n",
      "0.0008538369438610971 0.0\n",
      "0.9999076128005981 1.0\n",
      "0.9989897608757019 1.0\n",
      "0.9999822378158569 1.0\n",
      "0.9997959733009338 1.0\n",
      "0.00012753593910019845 0.0\n",
      "6.442372978199273e-06 0.0\n",
      "0.0023194686509668827 0.0\n",
      "0.9983483552932739 1.0\n",
      "1.2321160284045618e-05 0.0\n",
      "6.223735908861272e-06 0.0\n",
      "6.033190402376931e-06 0.0\n",
      "0.00011694921704474837 0.0\n",
      "0.9989674091339111 1.0\n",
      "0.9998935461044312 1.0\n",
      "0.007637171074748039 0.0\n",
      "6.526362267322838e-05 0.0\n",
      "0.9855925440788269 1.0\n",
      "9.456713996769395e-06 0.0\n",
      "0.9998501539230347 1.0\n",
      "0.9999638795852661 1.0\n",
      "0.9997422099113464 1.0\n",
      "0.00036742223892360926 0.0\n",
      "8.522051757609006e-06 0.0\n",
      "0.9998725652694702 1.0\n",
      "8.623575922683813e-06 0.0\n",
      "1.0715131793403998e-05 0.0\n",
      "0.9984205961227417 1.0\n",
      "0.9998825788497925 1.0\n",
      "0.0007775967824272811 0.0\n",
      "0.9994685053825378 1.0\n",
      "0.9999879598617554 1.0\n",
      "0.014568434096872807 0.0\n",
      "0.00011854457261506468 0.0\n",
      "0.9999680519104004 1.0\n",
      "0.9999909400939941 1.0\n",
      "1.6394864360336214e-05 0.0\n",
      "6.937898433534428e-05 0.0\n",
      "0.9999934434890747 1.0\n",
      "5.765989044448361e-06 0.0\n",
      "1.3625457540911157e-05 0.0\n",
      "0.9995589852333069 1.0\n",
      "0.999862790107727 1.0\n",
      "1.2949786651006434e-05 0.0\n",
      "6.600565939152148e-06 0.0\n",
      "7.576013103971491e-06 0.0\n",
      "0.9999213218688965 1.0\n",
      "5.844575753144454e-06 0.0\n",
      "0.999985933303833 1.0\n",
      "1.649015575821977e-05 0.0\n",
      "6.114491952757817e-06 0.0\n",
      "0.00013314968964550644 0.0\n",
      "0.9997453093528748 1.0\n",
      "0.9999884366989136 1.0\n",
      "0.9999886751174927 1.0\n",
      "0.999991774559021 1.0\n",
      "0.9999490976333618 1.0\n",
      "0.9997954964637756 1.0\n",
      "0.9972296357154846 1.0\n",
      "0.9998924732208252 1.0\n",
      "0.9998100399971008 1.0\n",
      "TRAIN[steps=7600] loss=0.000834 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7600] loss=0.027948 acc=0.701 P=0.690 R=0.703 F1=0.696517 \n",
      "model sim and label tuples:\n",
      "0.9943833351135254 1.0\n",
      "0.9999945163726807 1.0\n",
      "1.3159534319129307e-05 0.0\n",
      "0.9999945163726807 1.0\n",
      "0.0015744382981210947 0.0\n",
      "0.003483304986730218 0.0\n",
      "0.00011903220001840964 0.0\n",
      "0.9999171495437622 1.0\n",
      "1.3465794836520217e-05 0.0\n",
      "8.012299986148719e-06 0.0\n",
      "8.542279829271138e-06 0.0\n",
      "0.0014168916968628764 0.0\n",
      "0.9999852180480957 1.0\n",
      "1.369871915812837e-05 0.0\n",
      "0.0004471838765311986 0.0\n",
      "4.941289444104768e-05 0.0\n",
      "0.9999841451644897 1.0\n",
      "6.020689397701062e-06 0.0\n",
      "0.9999533891677856 1.0\n",
      "0.9998669624328613 1.0\n",
      "8.628618161310442e-06 0.0\n",
      "5.956621862424072e-06 0.0\n",
      "6.327813025563955e-06 0.0\n",
      "0.9442033171653748 1.0\n",
      "0.9992594122886658 1.0\n",
      "0.9967408776283264 1.0\n",
      "0.00011437263310654089 0.0\n",
      "0.9998688697814941 1.0\n",
      "8.018460903258529e-06 0.0\n",
      "0.9989379048347473 1.0\n",
      "0.00029070107848383486 0.0\n",
      "1.4909361198078841e-05 0.0\n",
      "0.999559223651886 1.0\n",
      "9.513200348010287e-05 0.0\n",
      "0.9999947547912598 1.0\n",
      "0.0001290621148655191 0.0\n",
      "0.9787523150444031 1.0\n",
      "0.9992338418960571 1.0\n",
      "0.9996715784072876 1.0\n",
      "1.613844324310776e-05 0.0\n",
      "0.9999474287033081 1.0\n",
      "0.9999948740005493 1.0\n",
      "7.274702056747628e-06 0.0\n",
      "3.499868034850806e-05 0.0\n",
      "0.000990599044598639 0.0\n",
      "0.9999873638153076 1.0\n",
      "0.1642308384180069 0.0\n",
      "0.9999947547912598 1.0\n",
      "1.175384568341542e-05 0.0\n",
      "9.56356234382838e-05 0.0\n",
      "0.9995840191841125 1.0\n",
      "0.9999884366989136 1.0\n",
      "0.00019310304196551442 0.0\n",
      "0.9961991906166077 1.0\n",
      "1.4750849913980346e-05 0.0\n",
      "2.0274244889151305e-05 0.0\n",
      "1.050047194439685e-05 0.0\n",
      "0.00016208707529585809 0.0\n",
      "0.999947190284729 1.0\n",
      "0.0001277705014217645 0.0\n",
      "0.9999904632568359 1.0\n",
      "0.9999936819076538 1.0\n",
      "0.999985933303833 1.0\n",
      "0.9999904632568359 1.0\n",
      "TRAIN[steps=7700] loss=0.004451 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7700] loss=0.028082 acc=0.699 P=0.692 R=0.692 F1=0.692095 \n",
      "model sim and label tuples:\n",
      "0.9992573857307434 1.0\n",
      "0.002668884117156267 0.0\n",
      "0.9998303651809692 1.0\n",
      "5.557852546189679e-06 0.0\n",
      "8.79584695212543e-06 0.0\n",
      "0.9998065829277039 1.0\n",
      "0.9999885559082031 1.0\n",
      "0.9995300769805908 1.0\n",
      "0.00016164456610567868 0.0\n",
      "8.932744094636291e-05 0.0\n",
      "1.8800701582222246e-05 0.0\n",
      "7.153351816668874e-06 0.0\n",
      "0.9995720982551575 1.0\n",
      "0.9999910593032837 1.0\n",
      "0.000470254773972556 0.0\n",
      "5.775267709395848e-05 0.0\n",
      "0.9999908208847046 1.0\n",
      "0.9993815422058105 1.0\n",
      "0.999957799911499 1.0\n",
      "2.5284018192905933e-05 0.0\n",
      "0.9998537302017212 1.0\n",
      "0.9999878406524658 1.0\n",
      "7.167630246840417e-05 0.0\n",
      "0.9865357279777527 1.0\n",
      "0.9999741315841675 1.0\n",
      "0.9999772310256958 1.0\n",
      "6.621340162382694e-06 0.0\n",
      "3.159325206070207e-05 0.0\n",
      "5.6946196309581865e-06 0.0\n",
      "0.9998006224632263 1.0\n",
      "7.406369149975944e-06 0.0\n",
      "0.9999833106994629 1.0\n",
      "0.9986205101013184 1.0\n",
      "1.313189113716362e-05 0.0\n",
      "6.537517765536904e-05 0.0\n",
      "0.9994586110115051 1.0\n",
      "7.664285476494115e-06 0.0\n",
      "0.9999843835830688 1.0\n",
      "0.999984860420227 1.0\n",
      "0.9999746084213257 1.0\n",
      "7.471942808479071e-06 0.0\n",
      "0.9999330043792725 1.0\n",
      "0.9999775886535645 1.0\n",
      "0.9999691247940063 1.0\n",
      "9.260419028578326e-05 0.0\n",
      "6.84372980686021e-06 0.0\n",
      "5.9707263062591664e-06 0.0\n",
      "0.9997503161430359 1.0\n",
      "0.9999847412109375 1.0\n",
      "1.5424981029354967e-05 0.0\n",
      "0.9999140501022339 1.0\n",
      "0.9999836683273315 1.0\n",
      "0.9710784554481506 1.0\n",
      "0.998581051826477 1.0\n",
      "1.0225138794339728e-05 0.0\n",
      "0.99991774559021 1.0\n",
      "9.031609806697816e-06 0.0\n",
      "0.9999821186065674 1.0\n",
      "8.961582352640107e-06 0.0\n",
      "0.9999940395355225 1.0\n",
      "9.758146916283295e-06 0.0\n",
      "0.9999047517776489 1.0\n",
      "0.0005869734450243413 0.0\n",
      "0.0031899227760732174 0.0\n",
      "TRAIN[steps=7800] loss=0.000903 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7800] loss=0.028214 acc=0.701 P=0.689 R=0.706 F1=0.697651 \n",
      "model sim and label tuples:\n",
      "0.984965443611145 1.0\n",
      "3.297065995866433e-05 0.0\n",
      "0.999967098236084 1.0\n",
      "0.9999672174453735 1.0\n",
      "5.8226733017363586e-06 0.0\n",
      "0.999626636505127 1.0\n",
      "1.00255711004138e-05 0.0\n",
      "0.9998672008514404 1.0\n",
      "0.9994365572929382 1.0\n",
      "0.008340727537870407 0.0\n",
      "0.9998683929443359 1.0\n",
      "9.25805125007173e-06 0.0\n",
      "1.2712113857560325e-05 0.0\n",
      "6.8817475948890205e-06 0.0\n",
      "0.9972538352012634 1.0\n",
      "6.148515967652202e-05 0.0\n",
      "0.9999865293502808 1.0\n",
      "0.0010558064095675945 0.0\n",
      "0.999977707862854 1.0\n",
      "0.9999897480010986 1.0\n",
      "0.9993607401847839 1.0\n",
      "0.0025630348827689886 0.0\n",
      "0.9982377290725708 1.0\n",
      "0.0025020965840667486 0.0\n",
      "0.9990369081497192 1.0\n",
      "6.4359678617620375e-06 0.0\n",
      "0.9999011754989624 1.0\n",
      "0.00014218298019841313 0.0\n",
      "0.9999250173568726 1.0\n",
      "0.0001827201049309224 0.0\n",
      "0.9994789958000183 1.0\n",
      "0.9980539083480835 1.0\n",
      "0.0005043189157731831 0.0\n",
      "5.6373755796812475e-06 0.0\n",
      "0.9999396800994873 1.0\n",
      "0.006951392162591219 0.0\n",
      "0.999963641166687 1.0\n",
      "1.11977024062071e-05 0.0\n",
      "0.9998430013656616 1.0\n",
      "1.1779857231886126e-05 0.0\n",
      "0.0020653060637414455 0.0\n",
      "0.9878585338592529 1.0\n",
      "0.006825590040534735 0.0\n",
      "5.4233150876825675e-06 0.0\n",
      "0.00023034171317704022 0.0\n",
      "0.9999942779541016 1.0\n",
      "1.2552636690088548e-05 0.0\n",
      "0.9999123811721802 1.0\n",
      "4.335342600825243e-05 0.0\n",
      "5.653731932397932e-06 0.0\n",
      "8.018674634513445e-06 0.0\n",
      "0.0003736919315997511 0.0\n",
      "0.9997640252113342 1.0\n",
      "9.410346137883607e-06 0.0\n",
      "0.9999291896820068 1.0\n",
      "0.9992259740829468 1.0\n",
      "0.0008149690693244338 0.0\n",
      "0.9985718727111816 1.0\n",
      "6.358257905958453e-06 0.0\n",
      "0.9999569654464722 1.0\n",
      "1.1107394129794557e-05 0.0\n",
      "0.9998937845230103 1.0\n",
      "2.2693657228956e-05 0.0\n",
      "5.291297384246718e-06 0.0\n",
      "TRAIN[steps=7900] loss=0.001147 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=7900] loss=0.028158 acc=0.699 P=0.696 R=0.683 F1=0.689349 \n",
      "model sim and label tuples:\n",
      "0.9765749573707581 1.0\n",
      "1.6015099390642717e-05 0.0\n",
      "5.563113518292084e-06 0.0\n",
      "3.537088923621923e-05 0.0\n",
      "0.992207944393158 1.0\n",
      "4.802293551620096e-05 0.0\n",
      "5.114650775794871e-06 0.0\n",
      "0.9997267127037048 1.0\n",
      "0.0003858136187773198 0.0\n",
      "5.429178600024898e-06 0.0\n",
      "3.6403060221346095e-05 0.0\n",
      "6.1003620430710725e-06 0.0\n",
      "0.0003695383493322879 0.0\n",
      "0.9999721050262451 1.0\n",
      "0.00012104257621103898 0.0\n",
      "6.262423994485289e-06 0.0\n",
      "6.373321957653388e-05 0.0\n",
      "0.9999852180480957 1.0\n",
      "0.00023119212710298598 0.0\n",
      "0.9999778270721436 1.0\n",
      "5.3945932450005785e-06 0.0\n",
      "0.9851399064064026 1.0\n",
      "0.00016383669571951032 0.0\n",
      "0.00044908872223459184 0.0\n",
      "6.527183359139599e-06 0.0\n",
      "7.86568762123352e-06 0.0\n",
      "1.6743668311391957e-05 0.0\n",
      "0.9940499067306519 1.0\n",
      "0.9997361302375793 1.0\n",
      "0.002012636512517929 0.0\n",
      "0.9969332218170166 1.0\n",
      "8.132112270686775e-06 0.0\n",
      "0.9998728036880493 1.0\n",
      "1.1259843631705735e-05 0.0\n",
      "8.62246670294553e-05 0.0\n",
      "0.999994158744812 1.0\n",
      "0.7525501847267151 1.0\n",
      "0.9998660087585449 1.0\n",
      "0.9999934434890747 1.0\n",
      "0.9988239407539368 1.0\n",
      "9.215025784214959e-05 0.0\n",
      "0.9999022483825684 1.0\n",
      "0.9965106844902039 1.0\n",
      "0.9999312162399292 1.0\n",
      "0.00020419263455551118 0.0\n",
      "0.006037647370249033 0.0\n",
      "0.999850869178772 1.0\n",
      "9.341987606603652e-06 0.0\n",
      "1.5571909898426384e-05 0.0\n",
      "5.177447746973485e-06 0.0\n",
      "1.7376758478349075e-05 0.0\n",
      "5.274334853311302e-06 0.0\n",
      "0.9985671043395996 1.0\n",
      "0.0001151438700617291 0.0\n",
      "0.9996695518493652 1.0\n",
      "0.999674916267395 1.0\n",
      "0.9999662637710571 1.0\n",
      "0.9885402321815491 1.0\n",
      "8.779480594967026e-06 0.0\n",
      "0.9998733997344971 1.0\n",
      "5.264922037895303e-06 0.0\n",
      "0.9999549388885498 1.0\n",
      "0.000579073850531131 0.0\n",
      "7.4136901275778655e-06 0.0\n",
      "TRAIN[steps=8000] loss=0.005793 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8000] loss=0.028253 acc=0.701 P=0.702 R=0.675 F1=0.688257 \n",
      "model sim and label tuples:\n",
      "3.824825762421824e-05 0.0\n",
      "0.9918206930160522 1.0\n",
      "0.4298601746559143 0.0\n",
      "0.0003586601815186441 0.0\n",
      "0.9997588992118835 1.0\n",
      "2.7504434910952114e-05 0.0\n",
      "0.9999954700469971 1.0\n",
      "0.999995231628418 1.0\n",
      "0.00023027168936096132 0.0\n",
      "0.9999880790710449 1.0\n",
      "0.00035545165883377194 0.0\n",
      "0.9999566078186035 1.0\n",
      "5.638644779537572e-06 0.0\n",
      "0.9999527931213379 1.0\n",
      "5.486856480274582e-06 0.0\n",
      "9.101018804358318e-05 0.0\n",
      "0.9999594688415527 1.0\n",
      "0.9998828172683716 1.0\n",
      "1.9174614863004535e-05 0.0\n",
      "0.9998062252998352 1.0\n",
      "5.447132934932597e-06 0.0\n",
      "5.7574948186811525e-06 0.0\n",
      "0.9980090260505676 1.0\n",
      "1.252759375347523e-05 0.0\n",
      "0.000841624743770808 0.0\n",
      "0.000436435075243935 0.0\n",
      "3.961255060858093e-05 0.0\n",
      "3.806664244621061e-05 0.0\n",
      "0.9963098168373108 1.0\n",
      "0.9999934434890747 1.0\n",
      "0.999976634979248 1.0\n",
      "4.665228334488347e-05 0.0\n",
      "7.226315938169137e-05 0.0\n",
      "0.1884063184261322 0.0\n",
      "5.134722869115649e-06 0.0\n",
      "6.888383359182626e-05 0.0\n",
      "5.044233148510102e-06 0.0\n",
      "0.005253789946436882 0.0\n",
      "0.9999431371688843 1.0\n",
      "2.9124879802111536e-05 0.0\n",
      "0.9999878406524658 1.0\n",
      "1.4639995242760051e-05 0.0\n",
      "0.00020036047499161214 0.0\n",
      "0.9999756813049316 1.0\n",
      "0.001803453080356121 0.0\n",
      "0.9999831914901733 1.0\n",
      "1.2273359061509836e-05 0.0\n",
      "2.665955798875075e-05 0.0\n",
      "1.1842709682241548e-05 0.0\n",
      "0.005241085775196552 0.0\n",
      "0.00012548771337606013 0.0\n",
      "5.15309366164729e-05 0.0\n",
      "0.9969596862792969 1.0\n",
      "0.9999215602874756 1.0\n",
      "4.811043709196383e-06 0.0\n",
      "9.854818927124143e-06 0.0\n",
      "0.9942343831062317 1.0\n",
      "0.9999951124191284 1.0\n",
      "7.035107046249323e-06 0.0\n",
      "2.5552009901730344e-05 0.0\n",
      "0.9987334609031677 1.0\n",
      "0.9999603033065796 1.0\n",
      "2.9901982998126186e-05 0.0\n",
      "0.9999759197235107 1.0\n",
      "TRAIN[steps=8100] loss=0.012675 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8100] loss=0.028485 acc=0.701 P=0.697 R=0.686 F1=0.691672 \n",
      "EPOCH:  13\n",
      "model sim and label tuples:\n",
      "4.876373623119434e-06 0.0\n",
      "0.997530996799469 1.0\n",
      "0.007540022488683462 0.0\n",
      "4.555201485345606e-06 0.0\n",
      "5.24067127116723e-06 0.0\n",
      "1.205671014758991e-05 0.0\n",
      "0.0007845841464586556 0.0\n",
      "0.999995231628418 1.0\n",
      "4.4939988583792e-05 0.0\n",
      "0.003399055451154709 0.0\n",
      "0.9999921321868896 1.0\n",
      "0.9998563528060913 1.0\n",
      "0.9999159574508667 1.0\n",
      "0.9978823065757751 1.0\n",
      "4.33836294178036e-06 0.0\n",
      "0.9999814033508301 1.0\n",
      "4.3705131247406825e-06 0.0\n",
      "0.0019014558056369424 0.0\n",
      "5.578110176429618e-06 0.0\n",
      "0.999975323677063 1.0\n",
      "0.999994158744812 1.0\n",
      "0.9999856948852539 1.0\n",
      "0.035541702061891556 0.0\n",
      "0.9999905824661255 1.0\n",
      "0.00038371444679796696 0.0\n",
      "0.9972233772277832 1.0\n",
      "0.9999539852142334 1.0\n",
      "0.9999502897262573 1.0\n",
      "4.726916358777089e-06 0.0\n",
      "0.999933123588562 1.0\n",
      "0.0008286923984996974 0.0\n",
      "0.00030374445486813784 0.0\n",
      "0.9410668611526489 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999845027923584 1.0\n",
      "0.0016258042305707932 0.0\n",
      "6.7523519646783825e-06 0.0\n",
      "2.0951023543602787e-05 0.0\n",
      "1.2230968422954902e-05 0.0\n",
      "0.0002164523903047666 0.0\n",
      "0.00014086066221352667 0.0\n",
      "0.8073460459709167 0.0\n",
      "0.9999947547912598 1.0\n",
      "0.9999926090240479 1.0\n",
      "0.0007319936412386596 0.0\n",
      "8.08542699815007e-06 0.0\n",
      "4.929320311930496e-06 0.0\n",
      "0.9998557567596436 1.0\n",
      "0.9999659061431885 1.0\n",
      "0.9996757507324219 1.0\n",
      "0.00016575104382354766 0.0\n",
      "0.0008671592804603279 0.0\n",
      "5.098657311464194e-06 0.0\n",
      "0.0003302059485577047 0.0\n",
      "0.0026477237697690725 0.0\n",
      "5.213295025896514e-06 0.0\n",
      "0.0027664429508149624 0.0\n",
      "3.120598921668716e-05 0.0\n",
      "0.9908745288848877 1.0\n",
      "4.377696313895285e-06 0.0\n",
      "0.0006544454954564571 0.0\n",
      "2.490366568963509e-05 0.0\n",
      "4.568626991385827e-06 0.0\n",
      "0.0003414780949242413 0.0\n",
      "TRAIN[steps=8200] loss=0.027926 acc=0.984 P=0.960 R=1.000 F1=0.979592\n",
      "DEV[steps=8200] loss=0.028915 acc=0.699 P=0.683 R=0.716 F1=0.699099 \n",
      "model sim and label tuples:\n",
      "0.9999392032623291 1.0\n",
      "7.197129889391363e-05 0.0\n",
      "0.9999046325683594 1.0\n",
      "0.9999203681945801 1.0\n",
      "7.156149422371527e-06 0.0\n",
      "9.252553718397394e-05 0.0\n",
      "0.005470407661050558 0.0\n",
      "0.9999642372131348 1.0\n",
      "0.9998526573181152 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.006435952614992857 0.0\n",
      "0.9305060505867004 1.0\n",
      "3.252517490182072e-05 0.0\n",
      "0.9999947547912598 1.0\n",
      "0.01135663315653801 0.0\n",
      "5.946512828813866e-06 0.0\n",
      "5.505185072252061e-06 0.0\n",
      "0.999976396560669 1.0\n",
      "0.9967193007469177 1.0\n",
      "0.9965847730636597 1.0\n",
      "0.999923586845398 1.0\n",
      "1.986386087082792e-05 0.0\n",
      "0.14439305663108826 0.0\n",
      "4.711277142632753e-06 0.0\n",
      "0.9985047578811646 1.0\n",
      "6.4570799622742925e-06 0.0\n",
      "0.0003146784729324281 0.0\n",
      "4.486885245569283e-06 0.0\n",
      "2.2880194592289627e-05 0.0\n",
      "0.9999829530715942 1.0\n",
      "0.9907048344612122 1.0\n",
      "0.9992964267730713 1.0\n",
      "0.9954690933227539 1.0\n",
      "1.0126726920134388e-05 0.0\n",
      "0.05305008590221405 0.0\n",
      "0.0010353639954701066 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.9669103026390076 1.0\n",
      "0.999995231628418 1.0\n",
      "4.090448328497587e-06 0.0\n",
      "4.303321020415751e-06 0.0\n",
      "0.9997838139533997 1.0\n",
      "0.999972939491272 1.0\n",
      "6.024606591381598e-06 0.0\n",
      "0.9996101260185242 1.0\n",
      "5.18961860507261e-06 0.0\n",
      "4.660543709178455e-06 0.0\n",
      "3.928151636500843e-05 0.0\n",
      "0.9999910593032837 1.0\n",
      "5.0950316108355764e-06 0.0\n",
      "0.9998278617858887 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.9949979782104492 1.0\n",
      "5.053776021668455e-06 0.0\n",
      "0.9998618364334106 1.0\n",
      "3.200041828677058e-05 0.0\n",
      "4.230582999298349e-05 0.0\n",
      "7.080026534822537e-06 0.0\n",
      "5.158113708603196e-06 0.0\n",
      "0.9981774091720581 1.0\n",
      "0.9999136924743652 1.0\n",
      "0.9994590878486633 1.0\n",
      "0.0001714526442810893 0.0\n",
      "0.028985943645238876 0.0\n",
      "TRAIN[steps=8300] loss=0.006291 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8300] loss=0.029150 acc=0.700 P=0.689 R=0.704 F1=0.696490 \n",
      "model sim and label tuples:\n",
      "0.00018219159392174333 0.0\n",
      "0.9999926090240479 1.0\n",
      "0.9989327788352966 1.0\n",
      "0.9999926090240479 1.0\n",
      "0.003923350945115089 0.0\n",
      "1.1414897016948089e-05 0.0\n",
      "9.592937203706242e-06 0.0\n",
      "2.8192587706143968e-05 0.0\n",
      "0.9997010827064514 1.0\n",
      "0.9991716146469116 1.0\n",
      "0.9999483823776245 1.0\n",
      "0.002148513915017247 0.0\n",
      "0.9999905824661255 1.0\n",
      "2.669418063305784e-05 0.0\n",
      "4.019907009933377e-06 0.0\n",
      "0.9999872446060181 1.0\n",
      "3.995015958935255e-06 0.0\n",
      "1.7732174455886707e-05 0.0\n",
      "0.9999827146530151 1.0\n",
      "6.5206204453716055e-06 0.0\n",
      "4.5727825636276975e-05 0.0\n",
      "5.972702638246119e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.9987336993217468 1.0\n",
      "4.409927441884065e-06 0.0\n",
      "6.465614205808379e-06 0.0\n",
      "0.0006601721397601068 0.0\n",
      "5.983449682389619e-06 0.0\n",
      "0.9999549388885498 1.0\n",
      "0.9994692206382751 1.0\n",
      "0.9999874830245972 1.0\n",
      "8.836253982735798e-06 0.0\n",
      "0.9921785593032837 1.0\n",
      "0.9834386706352234 1.0\n",
      "9.439652785658836e-05 0.0\n",
      "0.00017795235908124596 0.0\n",
      "7.649096914974507e-06 0.0\n",
      "0.9999830722808838 1.0\n",
      "0.999995231628418 1.0\n",
      "0.9998766183853149 1.0\n",
      "3.6325240216683596e-05 0.0\n",
      "0.01748049259185791 0.0\n",
      "0.002182464115321636 0.0\n",
      "0.9957751631736755 1.0\n",
      "0.9941564798355103 1.0\n",
      "0.9953011274337769 1.0\n",
      "0.9999938011169434 1.0\n",
      "0.00012383506691548973 0.0\n",
      "0.9999865293502808 1.0\n",
      "0.0016687901224941015 0.0\n",
      "7.595335773658007e-06 0.0\n",
      "4.0582103792985436e-06 0.0\n",
      "3.274130722275004e-05 0.0\n",
      "0.99998939037323 1.0\n",
      "4.655425982491579e-06 0.0\n",
      "0.999968409538269 1.0\n",
      "5.170106305740774e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.00373103772290051 0.0\n",
      "0.9999313354492188 1.0\n",
      "0.9999724626541138 1.0\n",
      "0.00031953456345945597 0.0\n",
      "0.0010852643754333258 0.0\n",
      "0.9995986819267273 1.0\n",
      "TRAIN[steps=8400] loss=0.001226 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8400] loss=0.029304 acc=0.700 P=0.688 R=0.706 F1=0.696672 \n",
      "model sim and label tuples:\n",
      "1.3541062799049541e-05 0.0\n",
      "0.000573742960114032 0.0\n",
      "1.188705391541589e-05 0.0\n",
      "0.9987137317657471 1.0\n",
      "0.000823767448309809 0.0\n",
      "0.41967108845710754 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999853372573853 1.0\n",
      "1.1016732969437726e-05 0.0\n",
      "1.9626415451057255e-05 0.0\n",
      "8.448197149846237e-06 0.0\n",
      "0.999715268611908 1.0\n",
      "0.002789984690025449 0.0\n",
      "0.9999614953994751 1.0\n",
      "0.9996716976165771 1.0\n",
      "0.9998657703399658 1.0\n",
      "8.198971954698209e-06 0.0\n",
      "0.9983799457550049 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.9999771118164062 1.0\n",
      "0.9999949932098389 1.0\n",
      "0.9985798597335815 1.0\n",
      "0.999994158744812 1.0\n",
      "0.9999263286590576 1.0\n",
      "5.124195467942627e-06 0.0\n",
      "6.141971880424535e-06 0.0\n",
      "4.153061126999091e-06 0.0\n",
      "1.346435510640731e-05 0.0\n",
      "0.0006310528260655701 0.0\n",
      "8.07763535703998e-06 0.0\n",
      "0.9984307885169983 1.0\n",
      "4.5879737626819406e-06 0.0\n",
      "0.00010700657003326342 0.0\n",
      "5.7431065215496346e-05 0.0\n",
      "0.9998136162757874 1.0\n",
      "1.1310197805869393e-05 0.0\n",
      "0.9998985528945923 1.0\n",
      "5.543924999074079e-06 0.0\n",
      "1.2567993508127984e-05 0.0\n",
      "0.9999755620956421 1.0\n",
      "0.9999780654907227 1.0\n",
      "0.999421238899231 1.0\n",
      "0.9999847412109375 1.0\n",
      "0.9999511241912842 1.0\n",
      "0.9999476671218872 1.0\n",
      "4.830149555345997e-05 0.0\n",
      "0.9967948794364929 1.0\n",
      "0.9999959468841553 1.0\n",
      "0.999991774559021 1.0\n",
      "0.9999914169311523 1.0\n",
      "4.954511041432852e-06 0.0\n",
      "0.9954391121864319 1.0\n",
      "7.5661564551410265e-06 0.0\n",
      "0.00013307212793733925 0.0\n",
      "0.0006114509887993336 0.0\n",
      "0.996636152267456 1.0\n",
      "1.214507210534066e-05 0.0\n",
      "1.5384213838842697e-05 0.0\n",
      "8.124592568492517e-06 0.0\n",
      "0.9999624490737915 1.0\n",
      "0.998445451259613 1.0\n",
      "1.3555495570471976e-05 0.0\n",
      "0.0012078160652890801 0.0\n",
      "0.999994158744812 1.0\n",
      "TRAIN[steps=8500] loss=0.008937 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8500] loss=0.029221 acc=0.700 P=0.689 R=0.704 F1=0.696349 \n",
      "model sim and label tuples:\n",
      "0.9977684020996094 1.0\n",
      "4.144262220506789e-06 0.0\n",
      "0.9999864101409912 1.0\n",
      "0.9999804496765137 1.0\n",
      "0.9999853372573853 1.0\n",
      "1.2420613529684488e-05 0.0\n",
      "4.910893949272577e-06 0.0\n",
      "0.9979135394096375 1.0\n",
      "4.134629762120312e-06 0.0\n",
      "4.223223641020013e-06 0.0\n",
      "4.230592821841128e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "5.41315384907648e-05 0.0\n",
      "0.004101851489394903 0.0\n",
      "3.2169347832677886e-05 0.0\n",
      "1.2202389370941091e-05 0.0\n",
      "3.64422048733104e-05 0.0\n",
      "5.4924512369325384e-05 0.0\n",
      "4.065861503477208e-06 0.0\n",
      "0.9997838139533997 1.0\n",
      "4.859567980020074e-06 0.0\n",
      "0.999883770942688 1.0\n",
      "5.922403033764567e-06 0.0\n",
      "0.9999858140945435 1.0\n",
      "0.00039769921568222344 0.0\n",
      "1.470795086788712e-05 0.0\n",
      "0.9999864101409912 1.0\n",
      "0.9999735355377197 1.0\n",
      "0.9999951124191284 1.0\n",
      "0.0005825445405207574 0.0\n",
      "0.9999833106994629 1.0\n",
      "0.9997075200080872 1.0\n",
      "0.9999947547912598 1.0\n",
      "0.9999898672103882 1.0\n",
      "4.9297718760499265e-06 0.0\n",
      "0.9148261547088623 1.0\n",
      "0.9999949932098389 1.0\n",
      "1.1223371984669939e-05 0.0\n",
      "0.0006884030299261212 0.0\n",
      "3.893193479598267e-06 0.0\n",
      "0.0002762793155852705 0.0\n",
      "0.9999915361404419 1.0\n",
      "0.9999915361404419 1.0\n",
      "2.1653597286785953e-05 0.0\n",
      "0.9999923706054688 1.0\n",
      "0.9995530247688293 1.0\n",
      "6.162706995382905e-06 0.0\n",
      "0.0028397156856954098 0.0\n",
      "8.034036909521092e-06 0.0\n",
      "0.9945993423461914 1.0\n",
      "3.984844624937978e-06 0.0\n",
      "0.998803973197937 1.0\n",
      "0.0008227978250943124 0.0\n",
      "5.607429102383321e-06 0.0\n",
      "0.9999402761459351 1.0\n",
      "0.0013345254119485617 0.0\n",
      "0.9999891519546509 1.0\n",
      "0.9999258518218994 1.0\n",
      "0.0003243687388021499 0.0\n",
      "7.62017089073197e-06 0.0\n",
      "0.9988601207733154 1.0\n",
      "1.180579056381248e-05 0.0\n",
      "1.6317340850946493e-05 0.0\n",
      "0.0004674204683396965 0.0\n",
      "TRAIN[steps=8600] loss=0.001792 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8600] loss=0.029329 acc=0.700 P=0.689 R=0.704 F1=0.696260 \n",
      "model sim and label tuples:\n",
      "0.00013970999862067401 0.0\n",
      "4.8747788241598755e-06 0.0\n",
      "0.9999641180038452 1.0\n",
      "0.9995261430740356 1.0\n",
      "1.3633035450766329e-05 0.0\n",
      "4.9941521865548566e-05 0.0\n",
      "0.9994946718215942 1.0\n",
      "3.3150467061204836e-05 0.0\n",
      "0.993485152721405 1.0\n",
      "0.0073834811337292194 0.0\n",
      "0.9999954700469971 1.0\n",
      "3.964501047448721e-06 0.0\n",
      "0.0005570703069679439 0.0\n",
      "0.9998834133148193 1.0\n",
      "0.0051238820888102055 0.0\n",
      "0.9999231100082397 1.0\n",
      "0.9999938011169434 1.0\n",
      "5.078711637906963e-06 0.0\n",
      "0.9999961853027344 1.0\n",
      "1.930604594235774e-05 0.0\n",
      "4.763217020808952e-06 0.0\n",
      "0.9913774728775024 1.0\n",
      "4.05025730287889e-06 0.0\n",
      "0.9999840259552002 1.0\n",
      "0.99997878074646 1.0\n",
      "4.2542545998003334e-05 0.0\n",
      "0.9999376535415649 1.0\n",
      "0.00012817454989999533 0.0\n",
      "0.000824327755253762 0.0\n",
      "4.32327578892e-06 0.0\n",
      "0.9999942779541016 1.0\n",
      "0.9895520806312561 1.0\n",
      "1.4187750821292866e-05 0.0\n",
      "1.0192960871791001e-05 0.0\n",
      "0.9999916553497314 1.0\n",
      "0.00028118726913817227 0.0\n",
      "0.001605395576916635 0.0\n",
      "0.9999920129776001 1.0\n",
      "0.9999880790710449 1.0\n",
      "0.00679321913048625 0.0\n",
      "1.3930133718531579e-05 0.0\n",
      "0.9999942779541016 1.0\n",
      "0.9999841451644897 1.0\n",
      "0.9999594688415527 1.0\n",
      "0.00028921657940372825 0.0\n",
      "4.223212272336241e-06 0.0\n",
      "0.9976657629013062 1.0\n",
      "0.9999943971633911 1.0\n",
      "0.0016943184891715646 0.0\n",
      "0.9999661445617676 1.0\n",
      "0.9999912977218628 1.0\n",
      "0.993744432926178 1.0\n",
      "6.583748472621664e-06 0.0\n",
      "3.966974418290192e-06 0.0\n",
      "0.9999836683273315 1.0\n",
      "0.9999641180038452 1.0\n",
      "0.0012281184317544103 0.0\n",
      "5.512816278496757e-05 0.0\n",
      "0.0013769390061497688 0.0\n",
      "0.9999059438705444 1.0\n",
      "0.00017515603394713253 0.0\n",
      "0.9999887943267822 1.0\n",
      "0.0018550290260463953 0.0\n",
      "9.428051271243021e-06 0.0\n",
      "TRAIN[steps=8700] loss=0.001028 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8700] loss=0.029281 acc=0.700 P=0.688 R=0.705 F1=0.696497 \n",
      "EPOCH:  14\n",
      "model sim and label tuples:\n",
      "0.998957633972168 1.0\n",
      "9.278220204578247e-06 0.0\n",
      "4.358725163911004e-06 0.0\n",
      "1.4795792594668455e-05 0.0\n",
      "0.9999696016311646 1.0\n",
      "0.001864142483100295 0.0\n",
      "0.999987006187439 1.0\n",
      "1.2907212294521742e-05 0.0\n",
      "0.9995365142822266 1.0\n",
      "0.9999479055404663 1.0\n",
      "0.8881155848503113 1.0\n",
      "2.2167021597852e-05 0.0\n",
      "0.9999881982803345 1.0\n",
      "0.9999279975891113 1.0\n",
      "5.641581537929596e-06 0.0\n",
      "5.2462119128904305e-06 0.0\n",
      "0.999964714050293 1.0\n",
      "4.816915406991029e-06 0.0\n",
      "1.6274789231829345e-05 0.0\n",
      "0.9999947547912598 1.0\n",
      "4.393974904814968e-06 0.0\n",
      "0.9996331930160522 1.0\n",
      "4.437901952769607e-05 0.0\n",
      "1.3009980648348574e-05 0.0\n",
      "0.999656081199646 1.0\n",
      "0.9998859167098999 1.0\n",
      "0.9999403953552246 1.0\n",
      "6.513988773804158e-05 0.0\n",
      "0.00023016237537376583 0.0\n",
      "3.0905888706911355e-05 0.0\n",
      "0.9995713829994202 1.0\n",
      "0.0004026140086352825 0.0\n",
      "0.9999830722808838 1.0\n",
      "1.0790596206788905e-05 0.0\n",
      "0.9996227025985718 1.0\n",
      "0.999991774559021 1.0\n",
      "0.0006605160888284445 0.0\n",
      "0.999954104423523 1.0\n",
      "0.03473318740725517 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999963045120239 1.0\n",
      "5.305207650962984e-06 0.0\n",
      "3.558298203643062e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9998766183853149 1.0\n",
      "0.9999961853027344 1.0\n",
      "0.9998550415039062 1.0\n",
      "0.9999842643737793 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.9865990877151489 1.0\n",
      "0.00019885275105480105 0.0\n",
      "0.9999947547912598 1.0\n",
      "0.995479941368103 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.9999924898147583 1.0\n",
      "0.9431174397468567 1.0\n",
      "0.8764739632606506 1.0\n",
      "0.0004368113004602492 0.0\n",
      "0.9983426332473755 1.0\n",
      "0.9999803304672241 1.0\n",
      "5.927986876486102e-06 0.0\n",
      "1.532397436676547e-05 0.0\n",
      "5.8608864492271096e-05 0.0\n",
      "0.9996569156646729 1.0\n",
      "TRAIN[steps=8800] loss=0.057768 acc=0.984 P=1.000 R=0.974 F1=0.986667\n",
      "DEV[steps=8800] loss=0.029707 acc=0.701 P=0.691 R=0.701 F1=0.695980 \n",
      "model sim and label tuples:\n",
      "0.9999959468841553 1.0\n",
      "0.9999182224273682 1.0\n",
      "0.001432211953215301 0.0\n",
      "0.0003994089493062347 0.0\n",
      "0.0004267444892320782 0.0\n",
      "5.201996373216389e-06 0.0\n",
      "0.999966025352478 1.0\n",
      "0.9999393224716187 1.0\n",
      "8.098507350950968e-06 0.0\n",
      "0.008830524049699306 0.0\n",
      "8.522035386704374e-06 0.0\n",
      "0.9990701079368591 1.0\n",
      "5.8742487453855574e-05 0.0\n",
      "0.9999955892562866 1.0\n",
      "4.924908353132196e-06 0.0\n",
      "1.57365884660976e-05 0.0\n",
      "4.614644240064081e-06 0.0\n",
      "0.9999154806137085 1.0\n",
      "0.017226947471499443 0.0\n",
      "0.9993242025375366 1.0\n",
      "7.75262469687732e-06 0.0\n",
      "0.9999934434890747 1.0\n",
      "0.9999961853027344 1.0\n",
      "0.9996984004974365 1.0\n",
      "3.745233698282391e-06 0.0\n",
      "0.9999127388000488 1.0\n",
      "2.3762526325299405e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.0006061877938918769 0.0\n",
      "5.045806028647348e-05 0.0\n",
      "0.9999943971633911 1.0\n",
      "1.170825635199435e-05 0.0\n",
      "4.052594704262447e-06 0.0\n",
      "4.296530460123904e-05 0.0\n",
      "2.1548949007410556e-05 0.0\n",
      "9.263279025617521e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.9993089437484741 1.0\n",
      "3.839614691969473e-06 0.0\n",
      "2.279414729855489e-05 0.0\n",
      "0.9823926687240601 1.0\n",
      "0.9995347261428833 1.0\n",
      "0.9999873638153076 1.0\n",
      "0.0011840774677693844 0.0\n",
      "0.0015568517846986651 0.0\n",
      "0.9924975037574768 1.0\n",
      "0.0029277713038027287 0.0\n",
      "0.00029001259827055037 0.0\n",
      "4.923752840113593e-06 0.0\n",
      "3.0127968784654513e-05 0.0\n",
      "1.550592241983395e-05 0.0\n",
      "3.6064034247829113e-06 0.0\n",
      "0.9999849796295166 1.0\n",
      "3.6444837405724684e-06 0.0\n",
      "6.756506536476081e-06 0.0\n",
      "0.9995028972625732 1.0\n",
      "2.145666985597927e-05 0.0\n",
      "0.9820947051048279 1.0\n",
      "0.9999934434890747 1.0\n",
      "8.433843504462857e-06 0.0\n",
      "2.4105138436425477e-05 0.0\n",
      "4.223143605486257e-06 0.0\n",
      "2.53366852120962e-05 0.0\n",
      "4.403724233270623e-06 0.0\n",
      "TRAIN[steps=8900] loss=0.001295 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=8900] loss=0.029756 acc=0.702 P=0.697 R=0.690 F1=0.693528 \n",
      "model sim and label tuples:\n",
      "6.452967681980226e-06 0.0\n",
      "0.0039002317935228348 0.0\n",
      "0.981663703918457 1.0\n",
      "0.0007935514440760016 0.0\n",
      "0.9999929666519165 1.0\n",
      "3.459048912191065e-06 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.9999884366989136 1.0\n",
      "3.7428949326567817e-06 0.0\n",
      "0.00020097906235605478 0.0\n",
      "3.962173286709003e-05 0.0\n",
      "8.095557859633118e-06 0.0\n",
      "5.575865361606702e-05 0.0\n",
      "0.999990701675415 1.0\n",
      "0.9998745918273926 1.0\n",
      "0.999991774559021 1.0\n",
      "3.484446551738074e-06 0.0\n",
      "0.004013812635093927 0.0\n",
      "8.123864063236397e-06 0.0\n",
      "4.5311549001780804e-06 0.0\n",
      "0.0002274665457662195 0.0\n",
      "3.3292678836005507e-06 0.0\n",
      "0.017412004992365837 0.0\n",
      "0.99997878074646 1.0\n",
      "0.00018173544958699495 0.0\n",
      "0.9999306201934814 1.0\n",
      "0.9999313354492188 1.0\n",
      "3.968438704760047e-06 0.0\n",
      "0.9999805688858032 1.0\n",
      "0.002463447628542781 0.0\n",
      "9.474272519582883e-05 0.0\n",
      "3.964099960285239e-06 0.0\n",
      "3.446782784521929e-06 0.0\n",
      "0.9999079704284668 1.0\n",
      "4.030997843074147e-06 0.0\n",
      "0.9999864101409912 1.0\n",
      "4.528372755885357e-06 0.0\n",
      "0.9999939203262329 1.0\n",
      "0.9999542236328125 1.0\n",
      "5.6448370742145926e-06 0.0\n",
      "0.0034910847898572683 0.0\n",
      "0.9999939203262329 1.0\n",
      "3.6246587114874274e-05 0.0\n",
      "0.0003533514973241836 0.0\n",
      "4.877596893493319e-06 0.0\n",
      "0.9992815852165222 1.0\n",
      "0.0003473065735306591 0.0\n",
      "0.9997374415397644 1.0\n",
      "0.20606835186481476 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9998817443847656 1.0\n",
      "0.9999926090240479 1.0\n",
      "4.400970283313654e-06 0.0\n",
      "0.0002635054988786578 0.0\n",
      "2.5043802452273667e-05 0.0\n",
      "3.466919224592857e-06 0.0\n",
      "0.00014638411812484264 0.0\n",
      "1.0003527677326929e-05 0.0\n",
      "0.9999954700469971 1.0\n",
      "0.9999955892562866 1.0\n",
      "4.340129635238554e-06 0.0\n",
      "0.9999879598617554 1.0\n",
      "0.0014512062771245837 0.0\n",
      "1.2063816029694863e-05 0.0\n",
      "TRAIN[steps=9000] loss=0.004480 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9000] loss=0.029914 acc=0.701 P=0.689 R=0.708 F1=0.698413 \n",
      "model sim and label tuples:\n",
      "5.256929398456123e-06 0.0\n",
      "1.2590149708557874e-05 0.0\n",
      "2.4159335225704126e-05 0.0\n",
      "1.2235344001965132e-05 0.0\n",
      "3.512040166242514e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9962246417999268 1.0\n",
      "0.9949989318847656 1.0\n",
      "4.4464140955824405e-06 0.0\n",
      "0.0014633449027314782 0.0\n",
      "0.9994243383407593 1.0\n",
      "0.00013035061419941485 0.0\n",
      "0.00011037052900064737 0.0\n",
      "0.001081508700735867 0.0\n",
      "0.06126181036233902 0.0\n",
      "2.6167574105784297e-05 0.0\n",
      "0.996606707572937 1.0\n",
      "0.012643812224268913 0.0\n",
      "0.9999704360961914 1.0\n",
      "8.844315016176552e-05 0.0\n",
      "0.9999312162399292 1.0\n",
      "3.880357326124795e-06 0.0\n",
      "6.440578999900026e-06 0.0\n",
      "2.7009891709894873e-05 0.0\n",
      "3.956984528485918e-06 0.0\n",
      "0.000159627350512892 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.9036627411842346 1.0\n",
      "0.0052562132477760315 0.0\n",
      "0.9987006187438965 1.0\n",
      "3.3944165807042737e-06 0.0\n",
      "5.663904630637262e-06 0.0\n",
      "0.0001506122061982751 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.9999748468399048 1.0\n",
      "1.843281825131271e-05 0.0\n",
      "0.9999803304672241 1.0\n",
      "1.967791649803985e-05 0.0\n",
      "1.6289526683976874e-05 0.0\n",
      "7.481005013687536e-05 0.0\n",
      "0.9998816251754761 1.0\n",
      "0.00023284993949346244 0.0\n",
      "1.921787225001026e-05 0.0\n",
      "0.99974125623703 1.0\n",
      "2.129840686393436e-05 0.0\n",
      "0.000828741816803813 0.0\n",
      "0.0037894409615546465 0.0\n",
      "0.9982220530509949 1.0\n",
      "0.9999947547912598 1.0\n",
      "4.091575829079375e-05 0.0\n",
      "0.9998286962509155 1.0\n",
      "0.9999746084213257 1.0\n",
      "0.0001735957630444318 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9997137188911438 1.0\n",
      "7.436115538439481e-06 0.0\n",
      "1.4082659617997706e-05 0.0\n",
      "8.575761967222206e-06 0.0\n",
      "0.00018146222282666713 0.0\n",
      "0.9999685287475586 1.0\n",
      "0.9999890327453613 1.0\n",
      "0.9999947547912598 1.0\n",
      "3.877734343404882e-06 0.0\n",
      "4.677940978581319e-06 0.0\n",
      "TRAIN[steps=9100] loss=0.003254 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9100] loss=0.030182 acc=0.697 P=0.680 R=0.719 F1=0.699064 \n",
      "model sim and label tuples:\n",
      "0.741647481918335 1.0\n",
      "0.00027985882479697466 0.0\n",
      "0.1446571946144104 0.0\n",
      "0.006880775559693575 0.0\n",
      "0.9999954700469971 1.0\n",
      "7.442081823683111e-06 0.0\n",
      "0.999954342842102 1.0\n",
      "5.159402280696668e-05 0.0\n",
      "0.9996022582054138 1.0\n",
      "0.9999357461929321 1.0\n",
      "0.9870333671569824 1.0\n",
      "0.999945878982544 1.0\n",
      "0.999038815498352 1.0\n",
      "5.74527393837343e-06 0.0\n",
      "0.9999767541885376 1.0\n",
      "3.1227395083988085e-05 0.0\n",
      "1.5199678273347672e-05 0.0\n",
      "3.509358293740661e-06 0.0\n",
      "6.307063358690357e-06 0.0\n",
      "0.9999877214431763 1.0\n",
      "0.9999415874481201 1.0\n",
      "8.971158422355074e-06 0.0\n",
      "0.9999958276748657 1.0\n",
      "8.653941040392965e-05 0.0\n",
      "0.9978214502334595 1.0\n",
      "0.9999204874038696 1.0\n",
      "0.0006844654562883079 0.0\n",
      "0.9850900173187256 1.0\n",
      "0.9999743700027466 1.0\n",
      "5.5939679441507906e-05 0.0\n",
      "7.048806764942128e-06 0.0\n",
      "0.9900429248809814 1.0\n",
      "1.3180458154238295e-05 0.0\n",
      "0.000124260681332089 0.0\n",
      "0.9998893737792969 1.0\n",
      "0.9999908208847046 1.0\n",
      "4.585100214171689e-06 0.0\n",
      "0.9995854496955872 1.0\n",
      "0.0002662361366674304 0.0\n",
      "1.9856172002619132e-05 0.0\n",
      "0.999993085861206 1.0\n",
      "0.9973022937774658 1.0\n",
      "2.823550494213123e-05 0.0\n",
      "4.909762355964631e-05 0.0\n",
      "0.0024372346233576536 0.0\n",
      "0.9995662569999695 1.0\n",
      "0.9999947547912598 1.0\n",
      "0.0010247603058815002 0.0\n",
      "3.294666385045275e-06 0.0\n",
      "3.309805151729961e-06 0.0\n",
      "0.0001714026293484494 0.0\n",
      "0.9999605417251587 1.0\n",
      "5.058863735030172e-06 0.0\n",
      "0.9999589920043945 1.0\n",
      "0.9999735355377197 1.0\n",
      "0.999991774559021 1.0\n",
      "0.9999918937683105 1.0\n",
      "6.14331338510965e-06 0.0\n",
      "0.9999614953994751 1.0\n",
      "0.9999821186065674 1.0\n",
      "3.3793555758165894e-06 0.0\n",
      "0.9866647720336914 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9997639060020447 1.0\n",
      "TRAIN[steps=9200] loss=0.008234 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9200] loss=0.030211 acc=0.701 P=0.691 R=0.702 F1=0.696351 \n",
      "model sim and label tuples:\n",
      "0.990195095539093 1.0\n",
      "7.085592642397387e-06 0.0\n",
      "0.9999442100524902 1.0\n",
      "0.9996658563613892 1.0\n",
      "0.9999948740005493 1.0\n",
      "0.00010016682062996551 0.0\n",
      "0.998421311378479 1.0\n",
      "7.851396730984561e-06 0.0\n",
      "0.023761479184031487 0.0\n",
      "5.3189569371170364e-06 0.0\n",
      "7.8114544521668e-06 0.0\n",
      "1.6662588677718304e-05 0.0\n",
      "4.704627826868091e-06 0.0\n",
      "0.9999611377716064 1.0\n",
      "0.9999932050704956 1.0\n",
      "0.999980092048645 1.0\n",
      "0.9998600482940674 1.0\n",
      "1.5114265806914773e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.000201981863938272 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9811229109764099 1.0\n",
      "1.192268064187374e-05 0.0\n",
      "7.678295332880225e-06 0.0\n",
      "0.9999940395355225 1.0\n",
      "0.9999161958694458 1.0\n",
      "8.577357220929116e-06 0.0\n",
      "0.9999932050704956 1.0\n",
      "0.9998936653137207 1.0\n",
      "1.5104164049262181e-05 0.0\n",
      "0.9999536275863647 1.0\n",
      "0.0001748176582623273 0.0\n",
      "0.9998880624771118 1.0\n",
      "0.9745225310325623 1.0\n",
      "8.87686383066466e-06 0.0\n",
      "7.285519131983165e-06 0.0\n",
      "1.2555354260257445e-05 0.0\n",
      "5.292968126013875e-06 0.0\n",
      "0.9999948740005493 1.0\n",
      "1.0337767889723182e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9986358284950256 1.0\n",
      "0.999729573726654 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999814033508301 1.0\n",
      "0.020946234464645386 1.0\n",
      "0.9999645948410034 1.0\n",
      "0.9999381303787231 1.0\n",
      "0.0015198542969301343 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.9998600482940674 1.0\n",
      "0.00011154294043080881 0.0\n",
      "0.9999921321868896 1.0\n",
      "1.0306011972716078e-05 0.0\n",
      "3.4764175325108226e-06 0.0\n",
      "0.9999767541885376 1.0\n",
      "0.9659950137138367 0.0\n",
      "0.9999326467514038 1.0\n",
      "4.233728759572841e-06 0.0\n",
      "1.2484158105507959e-05 0.0\n",
      "4.26428186983685e-06 0.0\n",
      "1.8138101950171404e-05 0.0\n",
      "0.9984884262084961 1.0\n",
      "0.005131060723215342 0.0\n",
      "TRAIN[steps=9300] loss=0.114677 acc=0.969 P=0.971 R=0.971 F1=0.970588\n",
      "DEV[steps=9300] loss=0.029812 acc=0.699 P=0.690 R=0.695 F1=0.692873 \n",
      "EPOCH:  15\n",
      "model sim and label tuples:\n",
      "0.9999892711639404 1.0\n",
      "0.9991815686225891 1.0\n",
      "0.9704194068908691 1.0\n",
      "0.9999954700469971 1.0\n",
      "0.0002850873570423573 0.0\n",
      "9.918753676174674e-06 0.0\n",
      "0.9979210495948792 1.0\n",
      "0.9999963045120239 1.0\n",
      "1.7782553186407313e-05 0.0\n",
      "0.9999964237213135 1.0\n",
      "2.7985286578768864e-05 0.0\n",
      "0.9999881982803345 1.0\n",
      "0.007653664331883192 0.0\n",
      "3.798995066972566e-06 0.0\n",
      "0.9999911785125732 1.0\n",
      "6.730814493494108e-05 0.0\n",
      "0.999188244342804 1.0\n",
      "0.0016097985208034515 0.0\n",
      "0.9859850406646729 1.0\n",
      "0.0007032693247310817 0.0\n",
      "3.7200525184744038e-06 0.0\n",
      "7.69457528804196e-06 0.0\n",
      "0.9999148845672607 1.0\n",
      "4.401612750370987e-05 0.0\n",
      "1.4853545508231036e-05 0.0\n",
      "5.103453986521345e-06 0.0\n",
      "5.979935394861968e-06 0.0\n",
      "0.9999923706054688 1.0\n",
      "5.566838808590546e-06 0.0\n",
      "0.9991670846939087 1.0\n",
      "0.999992847442627 1.0\n",
      "0.9994062185287476 1.0\n",
      "0.9998902082443237 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.9995185136795044 1.0\n",
      "4.218260073685087e-05 0.0\n",
      "8.115044693113305e-06 0.0\n",
      "0.9999964237213135 1.0\n",
      "1.8261835066368803e-05 0.0\n",
      "0.9998061060905457 1.0\n",
      "0.9999885559082031 1.0\n",
      "4.091240498382831e-06 0.0\n",
      "0.00022962050570640713 0.0\n",
      "0.9999663829803467 1.0\n",
      "0.9999209642410278 1.0\n",
      "0.999954342842102 1.0\n",
      "0.9999375343322754 1.0\n",
      "0.9991772770881653 1.0\n",
      "0.9999371767044067 1.0\n",
      "0.9999902248382568 1.0\n",
      "0.00010416435543447733 0.0\n",
      "4.7666030695836525e-06 0.0\n",
      "0.999610960483551 1.0\n",
      "0.9989339709281921 1.0\n",
      "0.00013483143993653357 0.0\n",
      "0.9999963045120239 1.0\n",
      "2.7342213797965087e-05 0.0\n",
      "0.9999922513961792 1.0\n",
      "0.9999434947967529 1.0\n",
      "8.189578693418298e-06 0.0\n",
      "4.704223556473153e-06 0.0\n",
      "0.0009149370016530156 0.0\n",
      "3.853958787658485e-06 0.0\n",
      "3.420667553655221e-06 0.0\n",
      "TRAIN[steps=9400] loss=0.001014 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9400] loss=0.029415 acc=0.700 P=0.689 R=0.704 F1=0.696481 \n",
      "model sim and label tuples:\n",
      "0.9999655485153198 1.0\n",
      "2.973821210616734e-05 0.0\n",
      "0.000123073419672437 0.0\n",
      "0.9998565912246704 1.0\n",
      "0.9999703168869019 1.0\n",
      "0.9948981404304504 1.0\n",
      "0.008114171214401722 0.0\n",
      "0.9999852180480957 1.0\n",
      "2.6348643586970866e-05 0.0\n",
      "0.9999814033508301 1.0\n",
      "4.314757461543195e-06 0.0\n",
      "0.9999896287918091 1.0\n",
      "0.9998327493667603 1.0\n",
      "0.9995424747467041 1.0\n",
      "0.9999512434005737 1.0\n",
      "3.6658273074863246e-06 0.0\n",
      "0.999864935874939 1.0\n",
      "1.3054202099738177e-05 0.0\n",
      "9.683604002930224e-05 0.0\n",
      "0.9999722242355347 1.0\n",
      "0.9999936819076538 1.0\n",
      "0.9998798370361328 1.0\n",
      "0.9999926090240479 1.0\n",
      "0.9997866749763489 1.0\n",
      "3.6379042285261676e-05 0.0\n",
      "0.999846339225769 1.0\n",
      "8.400322258239612e-06 0.0\n",
      "3.973464117734693e-06 0.0\n",
      "5.009324013371952e-06 0.0\n",
      "0.00023329640680458397 0.0\n",
      "2.0027768186992034e-05 0.0\n",
      "0.9996469020843506 1.0\n",
      "0.9999576807022095 1.0\n",
      "4.664912194130011e-05 0.0\n",
      "0.9955471754074097 1.0\n",
      "0.001607901300303638 0.0\n",
      "0.001384709496051073 0.0\n",
      "0.9997739195823669 1.0\n",
      "1.9089791749138385e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9995131492614746 1.0\n",
      "4.958892986905994e-06 0.0\n",
      "0.9999954700469971 1.0\n",
      "0.9999202489852905 1.0\n",
      "0.9998311996459961 1.0\n",
      "5.559262717724778e-06 0.0\n",
      "0.9999877214431763 1.0\n",
      "0.0005978018743917346 0.0\n",
      "0.9999880790710449 1.0\n",
      "0.9998220801353455 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.9979006052017212 1.0\n",
      "0.9999963045120239 1.0\n",
      "0.9997997879981995 1.0\n",
      "0.0017640163423493505 0.0\n",
      "0.9999779462814331 1.0\n",
      "0.9995869994163513 1.0\n",
      "0.00036008318420499563 0.0\n",
      "0.0005573591915890574 0.0\n",
      "3.878570169035811e-06 0.0\n",
      "7.5607254075293895e-06 0.0\n",
      "4.658130819734652e-06 0.0\n",
      "1.3115846741129644e-05 0.0\n",
      "0.998843789100647 1.0\n",
      "TRAIN[steps=9500] loss=0.000496 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9500] loss=0.029521 acc=0.700 P=0.693 R=0.693 F1=0.693175 \n",
      "model sim and label tuples:\n",
      "1.2436673387128394e-05 0.0\n",
      "0.9920676946640015 1.0\n",
      "7.193961664597737e-06 0.0\n",
      "0.0006342224660329521 0.0\n",
      "0.0008494374342262745 0.0\n",
      "0.9999755620956421 1.0\n",
      "0.9999376535415649 1.0\n",
      "0.0025815165136009455 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9998413324356079 1.0\n",
      "5.269251687423093e-06 0.0\n",
      "3.800951162702404e-05 0.0\n",
      "0.00500733545050025 0.0\n",
      "1.4183570783643518e-05 0.0\n",
      "5.4382107919082046e-06 0.0\n",
      "0.9997188448905945 1.0\n",
      "0.9999817609786987 1.0\n",
      "6.786347512388602e-06 0.0\n",
      "0.9999955892562866 1.0\n",
      "0.9999942779541016 1.0\n",
      "4.444336809683591e-06 0.0\n",
      "0.9998537302017212 1.0\n",
      "3.3523723686812446e-05 0.0\n",
      "0.9989497065544128 1.0\n",
      "1.692531986918766e-05 0.0\n",
      "0.9988008737564087 1.0\n",
      "2.336278157599736e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.0005689879762940109 0.0\n",
      "0.9999885559082031 1.0\n",
      "0.9999232292175293 1.0\n",
      "4.520372749539092e-05 0.0\n",
      "0.9992096424102783 1.0\n",
      "0.9999932050704956 1.0\n",
      "0.00018487301713321358 0.0\n",
      "0.999996542930603 1.0\n",
      "0.00011883748084073886 0.0\n",
      "1.840953882492613e-05 0.0\n",
      "0.9999884366989136 1.0\n",
      "0.9999897480010986 1.0\n",
      "0.9999946355819702 1.0\n",
      "3.808728024523589e-06 0.0\n",
      "4.35170568380272e-06 0.0\n",
      "3.036153611901682e-05 0.0\n",
      "0.99998939037323 1.0\n",
      "0.00010094418394146487 0.0\n",
      "3.668016461233492e-06 0.0\n",
      "0.002032834105193615 0.0\n",
      "0.9999908208847046 1.0\n",
      "0.001007086830213666 0.0\n",
      "2.1482837837538682e-05 0.0\n",
      "0.9999772310256958 1.0\n",
      "0.9999836683273315 1.0\n",
      "0.9998830556869507 1.0\n",
      "9.897928066493478e-06 0.0\n",
      "0.9999865293502808 1.0\n",
      "0.9999712705612183 1.0\n",
      "1.2048629287164658e-05 0.0\n",
      "0.01684599556028843 0.0\n",
      "0.01260851789265871 0.0\n",
      "0.9998878240585327 1.0\n",
      "4.068587713845773e-06 0.0\n",
      "0.999984860420227 1.0\n",
      "9.521080755803268e-06 0.0\n",
      "TRAIN[steps=9600] loss=0.000864 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9600] loss=0.029660 acc=0.700 P=0.690 R=0.700 F1=0.694936 \n",
      "model sim and label tuples:\n",
      "1.4065505638427567e-05 0.0\n",
      "0.9998880624771118 1.0\n",
      "6.589624535990879e-05 0.0\n",
      "0.0017151470528915524 0.0\n",
      "5.913576387683861e-06 0.0\n",
      "0.0005783587694168091 0.0\n",
      "1.1092669410572853e-05 0.0\n",
      "0.999993085861206 1.0\n",
      "1.8191942217526957e-05 0.0\n",
      "0.9998680353164673 1.0\n",
      "0.9684891104698181 1.0\n",
      "0.9951619505882263 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.9991627931594849 1.0\n",
      "0.9905886650085449 1.0\n",
      "0.9939674139022827 1.0\n",
      "0.0003858437412418425 0.0\n",
      "0.9995365142822266 1.0\n",
      "4.402746071718866e-06 0.0\n",
      "4.367288056528196e-06 0.0\n",
      "0.00198766659013927 0.0\n",
      "0.9999090433120728 1.0\n",
      "0.9999845027923584 1.0\n",
      "3.6736985293828184e-06 0.0\n",
      "0.0003480979357846081 0.0\n",
      "3.978560471296078e-06 0.0\n",
      "0.9999945163726807 1.0\n",
      "0.9999814033508301 1.0\n",
      "0.9999274015426636 1.0\n",
      "0.06313289701938629 0.0\n",
      "3.5796522297459887e-06 0.0\n",
      "0.002722029574215412 0.0\n",
      "0.9999725818634033 1.0\n",
      "0.024765241891145706 0.0\n",
      "0.999996542930603 1.0\n",
      "7.477117833332159e-06 0.0\n",
      "0.9992146492004395 1.0\n",
      "9.369834879180416e-05 0.0\n",
      "0.9999721050262451 1.0\n",
      "0.00020761153427883983 0.0\n",
      "0.0010558180510997772 0.0\n",
      "0.999995231628418 1.0\n",
      "0.9953930377960205 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.9996336698532104 1.0\n",
      "0.9999349117279053 1.0\n",
      "3.815242962446064e-06 0.0\n",
      "0.9999308586120605 1.0\n",
      "0.9999078512191772 1.0\n",
      "0.9999701976776123 1.0\n",
      "7.070236279105302e-06 0.0\n",
      "0.003618876449763775 0.0\n",
      "0.000684881117194891 0.0\n",
      "4.366492703411495e-06 0.0\n",
      "0.9999639987945557 1.0\n",
      "5.058854185335804e-06 0.0\n",
      "1.1006221029674634e-05 0.0\n",
      "1.6222429621848278e-05 0.0\n",
      "1.3506616596714593e-05 0.0\n",
      "0.9769162535667419 1.0\n",
      "0.0006583569338545203 0.0\n",
      "5.298255928209983e-06 0.0\n",
      "8.905774302547798e-05 0.0\n",
      "2.982511796290055e-05 0.0\n",
      "TRAIN[steps=9700] loss=0.002942 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9700] loss=0.029813 acc=0.700 P=0.689 R=0.702 F1=0.695308 \n",
      "model sim and label tuples:\n",
      "3.955060037696967e-06 0.0\n",
      "5.565028459386667e-06 0.0\n",
      "0.9982444047927856 1.0\n",
      "3.510765600367449e-05 0.0\n",
      "1.2930743650940713e-05 0.0\n",
      "1.4635179468314163e-05 0.0\n",
      "4.369720954855438e-06 0.0\n",
      "0.00019501811766531318 0.0\n",
      "4.1954049265768845e-06 0.0\n",
      "6.3565062191628385e-06 0.0\n",
      "0.0005392351886257529 0.0\n",
      "0.9972130656242371 1.0\n",
      "0.9999963045120239 1.0\n",
      "6.528782978421077e-05 0.0\n",
      "0.9955173134803772 1.0\n",
      "0.9985458850860596 1.0\n",
      "0.0009620338096283376 0.0\n",
      "5.747011528001167e-06 0.0\n",
      "2.0484292690525763e-05 0.0\n",
      "0.9999843835830688 1.0\n",
      "3.5026744171773316e-06 0.0\n",
      "0.9999009370803833 1.0\n",
      "0.9999951124191284 1.0\n",
      "3.8571834011236206e-06 0.0\n",
      "3.818635377683677e-06 0.0\n",
      "1.569528285472188e-05 0.0\n",
      "7.542385719716549e-05 0.0\n",
      "0.9999927282333374 1.0\n",
      "0.9999834299087524 1.0\n",
      "0.001251581939868629 0.0\n",
      "0.9999951124191284 1.0\n",
      "0.999955415725708 1.0\n",
      "0.9999651908874512 1.0\n",
      "0.9994004964828491 1.0\n",
      "0.0001473939628340304 0.0\n",
      "4.5522680011345074e-05 0.0\n",
      "4.546482159639709e-06 0.0\n",
      "0.975597083568573 1.0\n",
      "0.0001562309480505064 0.0\n",
      "5.937585228821263e-05 0.0\n",
      "0.9991483688354492 1.0\n",
      "3.4441477509972174e-06 0.0\n",
      "3.358215963089606e-06 0.0\n",
      "6.456806295318529e-05 0.0\n",
      "0.9920777678489685 1.0\n",
      "0.999967098236084 1.0\n",
      "2.339856837352272e-05 0.0\n",
      "4.270606041245628e-06 0.0\n",
      "2.0026851416332647e-05 0.0\n",
      "0.9999793767929077 1.0\n",
      "4.796461780642858e-06 0.0\n",
      "6.867554020573152e-06 0.0\n",
      "2.6400532078696415e-05 0.0\n",
      "2.8879194360342808e-05 0.0\n",
      "0.9998167157173157 1.0\n",
      "0.9993979930877686 1.0\n",
      "0.999821126461029 1.0\n",
      "4.392048140289262e-06 0.0\n",
      "0.9993681311607361 1.0\n",
      "6.896953709656373e-05 0.0\n",
      "0.00011522757267812267 0.0\n",
      "7.823291525710374e-05 0.0\n",
      "0.0004401484620757401 0.0\n",
      "0.0008617684943601489 0.0\n",
      "TRAIN[steps=9800] loss=0.000811 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9800] loss=0.030170 acc=0.703 P=0.696 R=0.696 F1=0.695634 \n",
      "model sim and label tuples:\n",
      "0.0004306742630433291 0.0\n",
      "0.9987805485725403 1.0\n",
      "3.970619218307547e-06 0.0\n",
      "4.66751771455165e-06 0.0\n",
      "0.9999949932098389 1.0\n",
      "0.9999666213989258 1.0\n",
      "3.827786713372916e-06 0.0\n",
      "0.0005678096204064786 0.0\n",
      "7.0807082011015154e-06 0.0\n",
      "3.397232649149373e-05 0.0\n",
      "0.999994158744812 1.0\n",
      "0.999894380569458 1.0\n",
      "0.00022363821335602552 0.0\n",
      "0.9988018274307251 1.0\n",
      "0.9999843835830688 1.0\n",
      "0.0012744627892971039 0.0\n",
      "0.009237864054739475 0.0\n",
      "0.9999232292175293 1.0\n",
      "3.947743243770674e-05 0.0\n",
      "8.225521924032364e-06 0.0\n",
      "7.084862318151863e-06 0.0\n",
      "0.9999921321868896 1.0\n",
      "2.4490756914019585e-05 0.0\n",
      "6.91524064677651e-06 0.0\n",
      "6.784826382499887e-06 0.0\n",
      "0.9990895986557007 1.0\n",
      "4.066671863256488e-06 0.0\n",
      "2.8573338568094186e-05 0.0\n",
      "3.4612960462254705e-06 0.0\n",
      "0.9998371601104736 1.0\n",
      "9.41785219765734e-06 0.0\n",
      "0.9740009903907776 1.0\n",
      "3.6749004266312113e-06 0.0\n",
      "5.001263343729079e-05 0.0\n",
      "0.997986912727356 1.0\n",
      "0.9999167919158936 1.0\n",
      "3.7027373309683753e-06 0.0\n",
      "0.0001722529559629038 0.0\n",
      "3.518711082506343e-06 0.0\n",
      "0.9998328685760498 1.0\n",
      "6.612152446905384e-06 0.0\n",
      "2.5605162591091357e-05 0.0\n",
      "3.477081008895766e-06 0.0\n",
      "0.002811107551679015 0.0\n",
      "4.2136965930694714e-05 0.0\n",
      "0.00044622752466239035 0.0\n",
      "0.0003124728100374341 0.0\n",
      "0.9999942779541016 1.0\n",
      "2.3836970285628922e-05 0.0\n",
      "0.9999339580535889 1.0\n",
      "0.9999829530715942 1.0\n",
      "0.998646080493927 1.0\n",
      "0.9999901056289673 1.0\n",
      "3.953125997213647e-06 0.0\n",
      "0.9999198913574219 1.0\n",
      "0.9999964237213135 1.0\n",
      "2.5713734430610202e-05 0.0\n",
      "0.9999940395355225 1.0\n",
      "0.9997307658195496 1.0\n",
      "0.08054488152265549 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.999985933303833 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.024812035262584686 0.0\n",
      "TRAIN[steps=9900] loss=0.002487 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=9900] loss=0.030171 acc=0.700 P=0.685 R=0.714 F1=0.699398 \n",
      "EPOCH:  16\n",
      "model sim and label tuples:\n",
      "3.521286316754413e-06 0.0\n",
      "0.999829888343811 1.0\n",
      "9.471932571614161e-05 0.0\n",
      "0.996887743473053 1.0\n",
      "1.79614780790871e-05 0.0\n",
      "0.9755523800849915 1.0\n",
      "0.00016776355914771557 0.0\n",
      "0.9999072551727295 1.0\n",
      "4.240359885443468e-06 0.0\n",
      "9.52414939092705e-06 0.0\n",
      "0.999991774559021 1.0\n",
      "0.00038402475183829665 0.0\n",
      "1.3918739568907768e-05 0.0\n",
      "0.9999873638153076 1.0\n",
      "6.627840775763616e-05 0.0\n",
      "0.9836803674697876 1.0\n",
      "2.8051550543750636e-05 0.0\n",
      "6.0531833696586546e-06 0.0\n",
      "7.927530532469973e-06 0.0\n",
      "0.9997043013572693 1.0\n",
      "1.3208984455559403e-05 0.0\n",
      "0.9996984004974365 1.0\n",
      "0.9951070547103882 1.0\n",
      "0.01494205929338932 0.0\n",
      "0.000223390175960958 0.0\n",
      "1.7214162653544918e-05 0.0\n",
      "4.371559498395072e-06 0.0\n",
      "0.9999818801879883 1.0\n",
      "0.9999816417694092 1.0\n",
      "0.999975323677063 1.0\n",
      "0.0003075588901992887 0.0\n",
      "1.1100764822913334e-05 0.0\n",
      "0.9999967813491821 1.0\n",
      "0.09507713466882706 0.0\n",
      "3.6652784274338046e-06 0.0\n",
      "0.0008109384798444808 0.0\n",
      "1.2817459719371982e-05 0.0\n",
      "1.0819323506439105e-05 0.0\n",
      "1.2380676707834937e-05 0.0\n",
      "0.9999959468841553 1.0\n",
      "8.253283340309281e-06 0.0\n",
      "0.8656586408615112 1.0\n",
      "0.9999953508377075 1.0\n",
      "4.548156084638322e-06 0.0\n",
      "0.00023630114446859807 0.0\n",
      "0.9995790123939514 1.0\n",
      "3.287623485448421e-06 0.0\n",
      "3.6145224839856382e-06 0.0\n",
      "0.9852749109268188 1.0\n",
      "0.9997246861457825 1.0\n",
      "3.936917892133351e-06 0.0\n",
      "3.199415459675947e-06 0.0\n",
      "0.9999856948852539 1.0\n",
      "0.9985041618347168 1.0\n",
      "0.00033730457653291523 0.0\n",
      "0.9999899864196777 1.0\n",
      "0.9999881982803345 1.0\n",
      "0.9999960660934448 1.0\n",
      "0.9999892711639404 1.0\n",
      "1.4955571714381222e-05 0.0\n",
      "9.90807075140765e-06 0.0\n",
      "0.0011059738462790847 0.0\n",
      "0.9999876022338867 1.0\n",
      "0.999982476234436 1.0\n",
      "TRAIN[steps=10000] loss=0.005164 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10000] loss=0.030102 acc=0.702 P=0.692 R=0.703 F1=0.697235 \n",
      "model sim and label tuples:\n",
      "5.724818947783206e-06 0.0\n",
      "4.846696811000584e-06 0.0\n",
      "6.126403604866937e-05 0.0\n",
      "0.000351961818523705 0.0\n",
      "0.9999744892120361 1.0\n",
      "0.9995900988578796 1.0\n",
      "3.914246008207556e-06 0.0\n",
      "0.014486495405435562 0.0\n",
      "0.9998306035995483 1.0\n",
      "2.3255573978531174e-05 0.0\n",
      "0.999985933303833 1.0\n",
      "1.2199770026199985e-05 0.0\n",
      "4.930006980430335e-06 0.0\n",
      "3.5801645026367623e-06 0.0\n",
      "0.000314624747261405 0.0\n",
      "0.9998482465744019 1.0\n",
      "0.00010736471449490637 0.0\n",
      "3.431410505072563e-06 0.0\n",
      "0.9971734285354614 1.0\n",
      "2.7762291210819967e-05 0.0\n",
      "4.2076171666849405e-05 0.0\n",
      "3.3150579383800505e-06 0.0\n",
      "0.9999070167541504 1.0\n",
      "1.099844666896388e-05 0.0\n",
      "3.6109638585912762e-06 0.0\n",
      "0.9999798536300659 1.0\n",
      "6.246497832762543e-06 0.0\n",
      "0.9999861717224121 1.0\n",
      "7.586445917695528e-06 0.0\n",
      "0.00011488559539429843 0.0\n",
      "0.0034573182929307222 0.0\n",
      "0.9998233914375305 1.0\n",
      "0.9999967813491821 1.0\n",
      "7.881230885686819e-06 0.0\n",
      "3.1118374863581266e-06 0.0\n",
      "3.3467667890363373e-06 0.0\n",
      "5.604959369520657e-06 0.0\n",
      "0.999881386756897 1.0\n",
      "0.00024272411246784031 0.0\n",
      "0.9999960660934448 1.0\n",
      "4.516167791734915e-06 0.0\n",
      "3.107362772425404e-06 0.0\n",
      "5.32498143002158e-06 0.0\n",
      "0.9999666213989258 1.0\n",
      "0.9999490976333618 1.0\n",
      "3.2425546123704407e-06 0.0\n",
      "4.00961798732169e-05 0.0\n",
      "3.7417919429572066e-06 0.0\n",
      "3.062629048145027e-06 0.0\n",
      "1.0815950190590229e-05 0.0\n",
      "0.9999969005584717 1.0\n",
      "3.278006033724523e-06 0.0\n",
      "0.9998723268508911 1.0\n",
      "0.00026184809394180775 0.0\n",
      "0.9999935626983643 1.0\n",
      "0.9621811509132385 1.0\n",
      "3.263762891947408e-06 0.0\n",
      "0.9999876022338867 1.0\n",
      "0.00014074170030653477 0.0\n",
      "0.999994158744812 1.0\n",
      "0.9999809265136719 1.0\n",
      "0.9999953508377075 1.0\n",
      "4.9860391300171614e-05 0.0\n",
      "0.05756853148341179 0.0\n",
      "TRAIN[steps=10100] loss=0.001908 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10100] loss=0.030499 acc=0.704 P=0.696 R=0.700 F1=0.698211 \n",
      "model sim and label tuples:\n",
      "0.9999511241912842 1.0\n",
      "7.5909333645540755e-06 0.0\n",
      "0.9999918937683105 1.0\n",
      "3.107395286860992e-06 0.0\n",
      "4.182022621534998e-06 0.0\n",
      "0.9999943971633911 1.0\n",
      "0.9981619715690613 1.0\n",
      "3.494698466965929e-05 0.0\n",
      "5.381624305300647e-06 0.0\n",
      "4.4724924919137266e-06 0.0\n",
      "5.3981995733920485e-05 0.0\n",
      "3.638868292910047e-06 0.0\n",
      "3.0384219371626386e-06 0.0\n",
      "0.9999939203262329 1.0\n",
      "2.5084555090870708e-05 0.0\n",
      "1.1828227798105218e-05 0.0\n",
      "2.9164539228077047e-06 0.0\n",
      "7.62484432925703e-06 0.0\n",
      "3.191714313288685e-06 0.0\n",
      "9.831866918830201e-06 0.0\n",
      "4.2562778617138974e-06 0.0\n",
      "2.6830177830561297e-06 0.0\n",
      "1.1146996257593855e-05 0.0\n",
      "0.9999963045120239 1.0\n",
      "0.9999693632125854 1.0\n",
      "0.9999971389770508 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999967813491821 1.0\n",
      "0.995662271976471 1.0\n",
      "1.581948163220659e-05 0.0\n",
      "0.9999799728393555 1.0\n",
      "0.999964714050293 1.0\n",
      "3.2767304674052866e-06 0.0\n",
      "3.0708829399372917e-06 0.0\n",
      "0.006943061947822571 0.0\n",
      "0.9999926090240479 1.0\n",
      "2.6873591195908375e-05 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.999683141708374 1.0\n",
      "0.9999970197677612 1.0\n",
      "3.090231075475458e-06 0.0\n",
      "0.9999864101409912 1.0\n",
      "0.9969960451126099 1.0\n",
      "3.3438762329751626e-05 0.0\n",
      "0.999995231628418 1.0\n",
      "0.9997155070304871 1.0\n",
      "0.00043736424413509667 0.0\n",
      "2.813607807183871e-06 0.0\n",
      "0.0005809213616885245 0.0\n",
      "5.302269528328907e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.9998443126678467 1.0\n",
      "4.401709247758845e-06 0.0\n",
      "0.00020697548461612314 0.0\n",
      "0.9999961853027344 1.0\n",
      "3.6032301977684256e-06 0.0\n",
      "0.9996942281723022 1.0\n",
      "0.9999934434890747 1.0\n",
      "0.0005540744750760496 0.0\n",
      "0.9999958276748657 1.0\n",
      "6.5058925429184455e-06 0.0\n",
      "1.909143065859098e-05 0.0\n",
      "0.9994487166404724 1.0\n",
      "0.9980735778808594 1.0\n",
      "TRAIN[steps=10200] loss=0.000344 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10200] loss=0.030976 acc=0.703 P=0.693 R=0.706 F1=0.699299 \n",
      "model sim and label tuples:\n",
      "0.9999575614929199 1.0\n",
      "3.210657496310887e-06 0.0\n",
      "0.0009525002096779644 0.0\n",
      "0.00048473678180016577 0.0\n",
      "0.9571107625961304 1.0\n",
      "0.9999972581863403 1.0\n",
      "8.674547279952094e-05 0.0\n",
      "4.8763711674837396e-05 0.0\n",
      "0.9999587535858154 1.0\n",
      "3.8461489566543605e-06 0.0\n",
      "0.9999847412109375 1.0\n",
      "5.107348897581687e-06 0.0\n",
      "0.9999749660491943 1.0\n",
      "0.9999916553497314 1.0\n",
      "3.0936869279685197e-06 0.0\n",
      "0.0016919681802392006 0.0\n",
      "0.9999920129776001 1.0\n",
      "5.9784924815176055e-06 0.0\n",
      "9.051779488800094e-05 0.0\n",
      "3.3801836707425537e-06 0.0\n",
      "0.0002222154289484024 0.0\n",
      "1.2115076970076188e-05 0.0\n",
      "0.011512516997754574 0.0\n",
      "0.999980092048645 1.0\n",
      "0.9990467429161072 1.0\n",
      "0.9995299577713013 1.0\n",
      "0.9999839067459106 1.0\n",
      "1.0900042070716154e-05 0.0\n",
      "0.9999929666519165 1.0\n",
      "2.7637530365609564e-05 0.0\n",
      "3.393423185116262e-06 0.0\n",
      "0.999854326248169 1.0\n",
      "2.7939518076891545e-06 0.0\n",
      "0.00941593386232853 0.0\n",
      "0.9999779462814331 1.0\n",
      "0.9999921321868896 1.0\n",
      "0.9998911619186401 1.0\n",
      "1.6743635569582693e-05 0.0\n",
      "1.4765500964131206e-05 0.0\n",
      "9.143117495113984e-06 0.0\n",
      "1.1062053999921773e-05 0.0\n",
      "0.9999749660491943 1.0\n",
      "0.9999510049819946 1.0\n",
      "1.0961806765408255e-05 0.0\n",
      "5.44555450687767e-06 0.0\n",
      "0.9999972581863403 1.0\n",
      "4.649980110116303e-05 0.0\n",
      "0.9995425939559937 1.0\n",
      "0.9999973773956299 1.0\n",
      "4.1034827518160455e-06 0.0\n",
      "2.9938244097138522e-06 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.9999839067459106 1.0\n",
      "0.0002956471871584654 0.0\n",
      "0.9998090863227844 1.0\n",
      "0.9999866485595703 1.0\n",
      "0.00021715959883295 0.0\n",
      "0.9980850219726562 1.0\n",
      "3.0712312764080707e-06 0.0\n",
      "0.9997953772544861 1.0\n",
      "0.9977163076400757 1.0\n",
      "0.9997060894966125 1.0\n",
      "0.9999951124191284 1.0\n",
      "5.856549705640646e-06 0.0\n",
      "TRAIN[steps=10300] loss=0.001196 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10300] loss=0.030934 acc=0.702 P=0.694 R=0.697 F1=0.695910 \n",
      "model sim and label tuples:\n",
      "0.000126949671539478 0.0\n",
      "2.0536948795779608e-05 0.0\n",
      "3.437319537624717e-05 0.0\n",
      "0.9999949932098389 1.0\n",
      "0.999955415725708 1.0\n",
      "5.542508006328717e-06 0.0\n",
      "3.6456517591432203e-06 0.0\n",
      "0.9999949932098389 1.0\n",
      "0.9493942260742188 1.0\n",
      "4.337302016210742e-05 0.0\n",
      "0.000145785539643839 0.0\n",
      "0.00011296216689515859 0.0\n",
      "3.881212251144461e-06 0.0\n",
      "0.00016731083451304585 0.0\n",
      "0.00011995137901976705 0.0\n",
      "0.9999825954437256 1.0\n",
      "0.9999018907546997 1.0\n",
      "0.9998847246170044 1.0\n",
      "0.0171916875988245 0.0\n",
      "0.9997648596763611 1.0\n",
      "0.9999971389770508 1.0\n",
      "1.737319507810753e-05 0.0\n",
      "8.333902769663837e-06 0.0\n",
      "0.9999914169311523 1.0\n",
      "0.9999825954437256 1.0\n",
      "3.675832886074204e-06 0.0\n",
      "2.5241542971343733e-05 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.0001701904839137569 0.0\n",
      "0.9974579215049744 1.0\n",
      "0.0019550221040844917 0.0\n",
      "0.9974601864814758 1.0\n",
      "0.9999858140945435 1.0\n",
      "6.551224032591563e-06 0.0\n",
      "0.0001344352785963565 0.0\n",
      "0.9964936375617981 1.0\n",
      "3.873010791721754e-05 0.0\n",
      "0.99910968542099 1.0\n",
      "0.9998364448547363 1.0\n",
      "4.168878149357624e-06 0.0\n",
      "0.9999608993530273 1.0\n",
      "0.011173994280397892 0.0\n",
      "0.9999954700469971 1.0\n",
      "4.177251867076848e-06 0.0\n",
      "0.999241828918457 1.0\n",
      "0.9996612071990967 1.0\n",
      "5.05709285789635e-05 0.0\n",
      "1.349823287455365e-05 0.0\n",
      "0.9999969005584717 1.0\n",
      "5.963840521872044e-06 0.0\n",
      "0.9962156414985657 1.0\n",
      "2.2002452169544995e-05 0.0\n",
      "5.4670079407515004e-05 0.0\n",
      "0.999052107334137 1.0\n",
      "4.644840373657644e-06 0.0\n",
      "0.00013214525824878365 0.0\n",
      "0.9999914169311523 1.0\n",
      "0.9999947547912598 1.0\n",
      "3.17222338708234e-06 0.0\n",
      "0.008340233936905861 0.0\n",
      "4.6684679546160623e-05 0.0\n",
      "0.9999971389770508 1.0\n",
      "0.9972924590110779 1.0\n",
      "0.001909758779220283 0.0\n",
      "TRAIN[steps=10400] loss=0.001767 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10400] loss=0.030272 acc=0.702 P=0.691 R=0.704 F1=0.697674 \n",
      "model sim and label tuples:\n",
      "4.6958924940554425e-05 0.0\n",
      "0.9892982244491577 1.0\n",
      "4.262631136953132e-06 0.0\n",
      "0.0015173836145550013 0.0\n",
      "0.0006430561770685017 0.0\n",
      "0.00012757061631418765 0.0\n",
      "4.749794698000187e-06 0.0\n",
      "0.9999885559082031 1.0\n",
      "0.999599039554596 1.0\n",
      "6.377194949891418e-05 0.0\n",
      "2.954801675514318e-05 0.0\n",
      "0.9994232654571533 1.0\n",
      "0.00010949647548841313 0.0\n",
      "4.643984357244335e-05 0.0\n",
      "0.9998618364334106 1.0\n",
      "3.2132334126799833e-06 0.0\n",
      "1.8415667000226676e-05 0.0\n",
      "6.03180978941964e-06 0.0\n",
      "1.0411893526907079e-05 0.0\n",
      "0.009437511675059795 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.012514830566942692 0.0\n",
      "0.9999884366989136 1.0\n",
      "6.345428118947893e-06 0.0\n",
      "6.94080290486454e-06 0.0\n",
      "0.999985933303833 1.0\n",
      "0.9992368221282959 1.0\n",
      "0.0005064565339125693 0.0\n",
      "0.0005825681728310883 0.0\n",
      "0.9999966621398926 1.0\n",
      "2.312370270374231e-05 0.0\n",
      "0.996131181716919 1.0\n",
      "0.998874843120575 1.0\n",
      "0.0013893037103116512 0.0\n",
      "1.8060276488540694e-05 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.9994074106216431 1.0\n",
      "0.998111367225647 1.0\n",
      "0.9999934434890747 1.0\n",
      "0.9996954202651978 1.0\n",
      "0.9999567270278931 1.0\n",
      "3.3191483908012742e-06 0.0\n",
      "3.820285201072693e-06 0.0\n",
      "0.9999020099639893 1.0\n",
      "0.9998501539230347 1.0\n",
      "0.9999220371246338 1.0\n",
      "3.5517280139174545e-06 0.0\n",
      "3.394957502678153e-06 0.0\n",
      "0.9999943971633911 1.0\n",
      "0.9999971389770508 1.0\n",
      "4.813342911802465e-06 0.0\n",
      "6.227564153959975e-05 0.0\n",
      "0.999920129776001 1.0\n",
      "0.9999799728393555 1.0\n",
      "0.9999661445617676 1.0\n",
      "3.132459823973477e-06 0.0\n",
      "1.3455973203235772e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "2.025252251769416e-05 0.0\n",
      "0.9999971389770508 1.0\n",
      "0.9999924898147583 1.0\n",
      "3.039363491552649e-06 0.0\n",
      "0.9999942779541016 1.0\n",
      "0.9999970197677612 1.0\n",
      "TRAIN[steps=10500] loss=0.000756 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10500] loss=0.030572 acc=0.701 P=0.689 R=0.706 F1=0.697491 \n",
      "model sim and label tuples:\n",
      "0.0003163989167660475 0.0\n",
      "0.9999966621398926 1.0\n",
      "6.3702086663397495e-06 0.0\n",
      "0.9999946355819702 1.0\n",
      "0.9999864101409912 1.0\n",
      "0.0001384727074764669 0.0\n",
      "5.007501022191718e-05 0.0\n",
      "0.9996035695075989 1.0\n",
      "4.539656947599724e-05 0.0\n",
      "0.9999048709869385 1.0\n",
      "0.999996542930603 1.0\n",
      "4.9957752707996406e-06 0.0\n",
      "0.0005843995022587478 0.0\n",
      "4.605768026522128e-06 0.0\n",
      "0.9311375617980957 1.0\n",
      "0.9977217316627502 1.0\n",
      "4.403434559208108e-06 0.0\n",
      "0.9999294281005859 1.0\n",
      "0.9991693496704102 1.0\n",
      "0.9999929666519165 1.0\n",
      "1.599565257492941e-05 0.0\n",
      "1.6074529412435368e-05 0.0\n",
      "0.9999922513961792 1.0\n",
      "1.97040026250761e-05 0.0\n",
      "0.0010543906828388572 0.0\n",
      "1.7844324247562326e-05 0.0\n",
      "3.588214212868479e-06 0.0\n",
      "0.00028479640604928136 0.0\n",
      "0.9999029636383057 1.0\n",
      "0.9998503923416138 1.0\n",
      "4.764628465636633e-05 0.0\n",
      "9.538457561575342e-06 0.0\n",
      "0.9967005848884583 1.0\n",
      "0.9999905824661255 1.0\n",
      "3.7792108287248993e-06 0.0\n",
      "0.9999648332595825 1.0\n",
      "3.731144943230902e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9952753782272339 1.0\n",
      "0.9994838237762451 1.0\n",
      "0.00012974893616046757 0.0\n",
      "3.4852973840315826e-06 0.0\n",
      "0.9998254179954529 1.0\n",
      "7.7497934398707e-06 0.0\n",
      "0.999995231628418 1.0\n",
      "0.00023236425477080047 0.0\n",
      "2.366995431657415e-05 0.0\n",
      "0.9997181296348572 1.0\n",
      "4.062454536324367e-06 0.0\n",
      "4.350904873717809e-06 0.0\n",
      "0.9999933242797852 1.0\n",
      "0.9999098777770996 1.0\n",
      "0.9999960660934448 1.0\n",
      "5.189487637835555e-05 0.0\n",
      "0.9873140454292297 1.0\n",
      "0.0036462347488850355 0.0\n",
      "1.1422301213315222e-05 0.0\n",
      "3.72560884898121e-06 0.0\n",
      "9.659658644523006e-06 0.0\n",
      "2.058015888906084e-05 0.0\n",
      "0.9999932050704956 1.0\n",
      "0.9894826412200928 1.0\n",
      "0.9999902248382568 1.0\n",
      "0.9997135996818542 1.0\n",
      "TRAIN[steps=10600] loss=0.001795 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10600] loss=0.029923 acc=0.701 P=0.694 R=0.691 F1=0.692892 \n",
      "EPOCH:  17\n",
      "model sim and label tuples:\n",
      "9.732260514283553e-05 0.0\n",
      "0.0038404110819101334 0.0\n",
      "0.9999934434890747 1.0\n",
      "0.0012900873553007841 0.0\n",
      "2.3226826669997536e-05 0.0\n",
      "0.9999947547912598 1.0\n",
      "3.924948032363318e-06 0.0\n",
      "0.002903464250266552 1.0\n",
      "3.3452447496529203e-06 0.0\n",
      "0.9998899698257446 1.0\n",
      "0.00010109387949341908 0.0\n",
      "0.9988589286804199 1.0\n",
      "0.9997617602348328 1.0\n",
      "0.9993669390678406 1.0\n",
      "0.9995492100715637 1.0\n",
      "4.184157205600059e-06 0.0\n",
      "0.9999969005584717 1.0\n",
      "3.5904906781070167e-06 0.0\n",
      "0.9999606609344482 1.0\n",
      "0.9999901056289673 1.0\n",
      "0.003343912772834301 0.0\n",
      "0.9998558759689331 1.0\n",
      "0.9998623132705688 1.0\n",
      "0.9994687438011169 1.0\n",
      "3.648214942586492e-06 0.0\n",
      "3.3935170904442202e-06 0.0\n",
      "3.2963384910544846e-06 0.0\n",
      "0.0002967465261463076 0.0\n",
      "1.0528608981985599e-05 0.0\n",
      "0.9999681711196899 1.0\n",
      "1.6250813132501207e-05 0.0\n",
      "0.9978177547454834 1.0\n",
      "3.402061793167377e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "8.493313544022385e-06 0.0\n",
      "1.5698065908509307e-05 0.0\n",
      "1.5711140804341994e-05 0.0\n",
      "0.9954676628112793 1.0\n",
      "0.9999164342880249 1.0\n",
      "3.839654709736351e-06 0.0\n",
      "6.845197640359402e-05 0.0\n",
      "0.9999802112579346 1.0\n",
      "0.0008899082895368338 0.0\n",
      "0.9979484677314758 1.0\n",
      "0.999947190284729 1.0\n",
      "3.851372821372934e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "1.2055122169840615e-05 0.0\n",
      "0.9999434947967529 1.0\n",
      "0.9993910789489746 1.0\n",
      "0.00016152870375663042 0.0\n",
      "0.9939067959785461 1.0\n",
      "5.279075139696943e-06 0.0\n",
      "5.742233952332754e-06 0.0\n",
      "0.9998019337654114 1.0\n",
      "6.732232577633113e-05 0.0\n",
      "9.248502465197816e-06 0.0\n",
      "0.9999657869338989 1.0\n",
      "3.638642738224007e-06 0.0\n",
      "0.9999909400939941 1.0\n",
      "0.9928147792816162 1.0\n",
      "6.8916451709810644e-06 0.0\n",
      "0.9999866485595703 1.0\n",
      "0.9998849630355835 1.0\n",
      "TRAIN[steps=10700] loss=0.091860 acc=0.984 P=1.000 R=0.969 F1=0.984127\n",
      "DEV[steps=10700] loss=0.030137 acc=0.702 P=0.696 R=0.692 F1=0.693731 \n",
      "model sim and label tuples:\n",
      "0.9990884065628052 1.0\n",
      "0.9999970197677612 1.0\n",
      "0.999991774559021 1.0\n",
      "0.0008364931563846767 0.0\n",
      "0.9995831847190857 1.0\n",
      "0.09434434771537781 1.0\n",
      "3.799044861807488e-05 0.0\n",
      "5.537119704968063e-06 0.0\n",
      "0.9999935626983643 1.0\n",
      "0.9999597072601318 1.0\n",
      "6.2679328038939275e-06 0.0\n",
      "0.9999634027481079 1.0\n",
      "0.9996179342269897 1.0\n",
      "4.34112780567375e-06 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.922747790813446 1.0\n",
      "0.9998342990875244 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.999925971031189 1.0\n",
      "1.2609699297172483e-05 0.0\n",
      "7.402654773613904e-06 0.0\n",
      "4.5045194383419584e-06 0.0\n",
      "0.0001229204935953021 0.0\n",
      "0.9999009370803833 1.0\n",
      "0.9999969005584717 1.0\n",
      "3.573031790438108e-05 0.0\n",
      "0.9962863922119141 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999959468841553 1.0\n",
      "5.595309175987495e-06 0.0\n",
      "0.999980092048645 1.0\n",
      "6.3196171140589286e-06 0.0\n",
      "1.900285860756412e-05 0.0\n",
      "0.00018921573064289987 0.0\n",
      "0.0003915174165740609 0.0\n",
      "5.844932275067549e-06 0.0\n",
      "0.9973527193069458 1.0\n",
      "1.1460939276730642e-05 0.0\n",
      "0.9983561635017395 1.0\n",
      "3.284641570644453e-05 0.0\n",
      "6.804265922255581e-06 0.0\n",
      "3.1112322176340967e-06 0.0\n",
      "0.0009606548701412976 0.0\n",
      "4.759151488542557e-05 0.0\n",
      "6.273625785979675e-06 0.0\n",
      "0.004016066435724497 0.0\n",
      "5.140234497957863e-05 0.0\n",
      "0.9998729228973389 1.0\n",
      "0.9748679399490356 1.0\n",
      "0.9844191670417786 1.0\n",
      "2.162482815037947e-05 0.0\n",
      "1.969663571799174e-05 0.0\n",
      "3.85469002139871e-06 0.0\n",
      "2.008627780014649e-05 0.0\n",
      "6.323143679765053e-06 0.0\n",
      "0.00016622539260424674 0.0\n",
      "3.1915167255647248e-06 0.0\n",
      "0.9999916553497314 1.0\n",
      "0.9999821186065674 1.0\n",
      "0.9649390578269958 1.0\n",
      "6.826663593528792e-06 0.0\n",
      "6.662995019723894e-06 0.0\n",
      "4.984867246093927e-06 0.0\n",
      "0.9999120235443115 1.0\n",
      "TRAIN[steps=10800] loss=0.039619 acc=0.984 P=1.000 R=0.966 F1=0.982456\n",
      "DEV[steps=10800] loss=0.030603 acc=0.701 P=0.692 R=0.700 F1=0.696281 \n",
      "model sim and label tuples:\n",
      "0.9999905824661255 1.0\n",
      "2.533721453801263e-05 0.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999960660934448 1.0\n",
      "9.439450877835043e-06 0.0\n",
      "0.9999909400939941 1.0\n",
      "0.9996700286865234 1.0\n",
      "5.771941687271465e-06 0.0\n",
      "0.9999829530715942 1.0\n",
      "0.999996542930603 1.0\n",
      "5.0933136662933975e-05 0.0\n",
      "0.9963412880897522 1.0\n",
      "5.6383611081400886e-05 0.0\n",
      "0.9999169111251831 1.0\n",
      "3.183549097229843e-06 0.0\n",
      "0.044033460319042206 0.0\n",
      "0.9999701976776123 1.0\n",
      "5.192425305722281e-06 0.0\n",
      "4.898484803561587e-06 0.0\n",
      "7.462749181286199e-06 0.0\n",
      "1.282774428545963e-05 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999698400497437 1.0\n",
      "0.9999771118164062 1.0\n",
      "5.577455522143282e-05 0.0\n",
      "0.9999771118164062 1.0\n",
      "0.0014304746873676777 0.0\n",
      "0.9998650550842285 1.0\n",
      "0.9999966621398926 1.0\n",
      "3.4467102523194626e-06 0.0\n",
      "0.9997351765632629 1.0\n",
      "7.301272034965223e-06 0.0\n",
      "0.9999310970306396 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999815225601196 1.0\n",
      "1.5309264199458994e-05 0.0\n",
      "6.81183701090049e-06 0.0\n",
      "0.9998753070831299 1.0\n",
      "0.999972939491272 1.0\n",
      "0.9999929666519165 1.0\n",
      "7.816946890670806e-06 0.0\n",
      "0.9997939467430115 1.0\n",
      "0.999993085861206 1.0\n",
      "0.9999940395355225 1.0\n",
      "0.0008946862653829157 0.0\n",
      "3.858633135678247e-06 0.0\n",
      "0.00036734851892106235 0.0\n",
      "0.9998672008514404 1.0\n",
      "5.544221039599506e-06 0.0\n",
      "4.433398316905368e-06 0.0\n",
      "0.9976804256439209 1.0\n",
      "2.9012440791120753e-05 0.0\n",
      "4.843743681703927e-06 0.0\n",
      "0.00010176337673328817 0.0\n",
      "0.01614527404308319 0.0\n",
      "0.9999939203262329 1.0\n",
      "0.9999477863311768 1.0\n",
      "0.9999771118164062 1.0\n",
      "1.305488603975391e-05 0.0\n",
      "1.7278060113312677e-05 0.0\n",
      "0.9775256514549255 1.0\n",
      "3.3382016226823907e-06 0.0\n",
      "0.999910831451416 1.0\n",
      "3.513339606797672e-06 0.0\n",
      "TRAIN[steps=10900] loss=0.001483 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=10900] loss=0.030537 acc=0.700 P=0.690 R=0.699 F1=0.694447 \n",
      "model sim and label tuples:\n",
      "9.276344644604251e-05 0.0\n",
      "0.9899964332580566 1.0\n",
      "4.138535132369725e-06 0.0\n",
      "3.89425167668378e-06 0.0\n",
      "0.9999916553497314 1.0\n",
      "0.0005939623224548995 0.0\n",
      "0.9999840259552002 1.0\n",
      "0.9974682331085205 1.0\n",
      "2.040879371634219e-05 0.0\n",
      "0.9999933242797852 1.0\n",
      "0.9999877214431763 1.0\n",
      "2.5345794711029157e-05 0.0\n",
      "3.339325758133782e-06 0.0\n",
      "0.00026856426848098636 0.0\n",
      "0.9994825124740601 1.0\n",
      "0.9998258948326111 1.0\n",
      "3.3148558031825814e-06 0.0\n",
      "4.780018116434803e-06 0.0\n",
      "5.2553359637386166e-06 0.0\n",
      "3.7496160075534135e-05 0.0\n",
      "2.3539032554253936e-05 0.0\n",
      "5.965273430774687e-06 0.0\n",
      "0.0007492417935281992 0.0\n",
      "0.014292463660240173 0.0\n",
      "0.9988445043563843 1.0\n",
      "0.0001306662306888029 0.0\n",
      "0.00960103701800108 0.0\n",
      "0.999994158744812 1.0\n",
      "4.440761131263571e-06 0.0\n",
      "4.024343070341274e-05 0.0\n",
      "0.9999955892562866 1.0\n",
      "0.9999960660934448 1.0\n",
      "0.9999946355819702 1.0\n",
      "4.823898507311242e-06 0.0\n",
      "0.9999947547912598 1.0\n",
      "0.9992272853851318 1.0\n",
      "0.9999954700469971 1.0\n",
      "0.9999951124191284 1.0\n",
      "0.9999946355819702 1.0\n",
      "0.9999724626541138 1.0\n",
      "0.00030984508339315653 0.0\n",
      "0.9995918869972229 1.0\n",
      "0.9999909400939941 1.0\n",
      "0.0003217572520952672 0.0\n",
      "0.9999945163726807 1.0\n",
      "0.0023397807963192463 0.0\n",
      "0.0018294546753168106 0.0\n",
      "0.9939998388290405 1.0\n",
      "5.629413863061927e-06 0.0\n",
      "0.030507447198033333 0.0\n",
      "6.20967875875067e-06 0.0\n",
      "0.999914288520813 1.0\n",
      "0.9999502897262573 1.0\n",
      "0.0015844467561692 0.0\n",
      "0.0003241912927478552 0.0\n",
      "0.9999949932098389 1.0\n",
      "0.9999865293502808 1.0\n",
      "3.198695139872143e-06 0.0\n",
      "0.0021606518421322107 0.0\n",
      "5.617015176540008e-06 0.0\n",
      "1.3463699360727333e-05 0.0\n",
      "0.9998332262039185 1.0\n",
      "7.003804967098404e-06 0.0\n",
      "0.9999970197677612 1.0\n",
      "TRAIN[steps=11000] loss=0.001376 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11000] loss=0.030583 acc=0.701 P=0.689 R=0.706 F1=0.697844 \n",
      "model sim and label tuples:\n",
      "0.9999923706054688 1.0\n",
      "0.9999796152114868 1.0\n",
      "0.9999940395355225 1.0\n",
      "0.9999758005142212 1.0\n",
      "0.9998927116394043 1.0\n",
      "1.1438641195127275e-05 0.0\n",
      "0.9999780654907227 1.0\n",
      "0.021420156583189964 0.0\n",
      "0.0007835747092030942 0.0\n",
      "0.0017439830116927624 0.0\n",
      "0.9999103546142578 1.0\n",
      "5.482796041178517e-05 0.0\n",
      "0.9999611377716064 1.0\n",
      "0.9999716281890869 1.0\n",
      "0.999976634979248 1.0\n",
      "0.9855906963348389 1.0\n",
      "0.9998751878738403 1.0\n",
      "0.9999895095825195 1.0\n",
      "0.9998875856399536 1.0\n",
      "0.9997499585151672 1.0\n",
      "3.532294613250997e-06 0.0\n",
      "0.9987590312957764 1.0\n",
      "0.00016104149108286947 0.0\n",
      "0.9999793767929077 1.0\n",
      "4.227773843012983e-06 0.0\n",
      "4.272378191672033e-06 0.0\n",
      "6.742840923834592e-06 0.0\n",
      "0.00011113070650026202 0.0\n",
      "0.000675361545290798 0.0\n",
      "0.9992910623550415 1.0\n",
      "0.9960987567901611 1.0\n",
      "0.9997580647468567 1.0\n",
      "1.2585815966303926e-05 0.0\n",
      "0.9865967631340027 1.0\n",
      "0.006430402398109436 0.0\n",
      "0.9999967813491821 1.0\n",
      "0.9999563694000244 1.0\n",
      "0.999923825263977 1.0\n",
      "3.4637496355571784e-06 0.0\n",
      "0.9999929666519165 1.0\n",
      "3.89801198252826e-06 0.0\n",
      "0.9999672174453735 1.0\n",
      "3.6553021800500574e-06 0.0\n",
      "0.9984577894210815 1.0\n",
      "3.2991690659400774e-06 0.0\n",
      "2.5617324354243465e-05 0.0\n",
      "0.9999350309371948 1.0\n",
      "0.9999792575836182 1.0\n",
      "0.9998897314071655 1.0\n",
      "0.9999431371688843 1.0\n",
      "0.0001546255953144282 0.0\n",
      "3.1075287552084774e-06 0.0\n",
      "0.9993383288383484 1.0\n",
      "0.000861901615280658 0.0\n",
      "0.00979494210332632 0.0\n",
      "0.9829185009002686 1.0\n",
      "4.531448666966753e-06 0.0\n",
      "3.231521304769558e-06 0.0\n",
      "0.0013621962862089276 0.0\n",
      "0.9429509043693542 1.0\n",
      "0.00044719749712385237 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.999994158744812 1.0\n",
      "TRAIN[steps=11100] loss=0.002469 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11100] loss=0.030450 acc=0.702 P=0.695 R=0.696 F1=0.695536 \n",
      "model sim and label tuples:\n",
      "0.00022882914345245808 0.0\n",
      "0.9999359846115112 1.0\n",
      "0.0028161497320979834 0.0\n",
      "0.9999922513961792 1.0\n",
      "0.24000662565231323 0.0\n",
      "4.426216491992818e-06 0.0\n",
      "0.999994158744812 1.0\n",
      "0.9976243376731873 1.0\n",
      "1.1021114914910868e-05 0.0\n",
      "4.868613814323908e-06 0.0\n",
      "0.999994158744812 1.0\n",
      "6.110253889346495e-05 0.0\n",
      "8.521173185727093e-06 0.0\n",
      "8.729637556825764e-06 0.0\n",
      "0.0003202219377271831 0.0\n",
      "0.9998080134391785 1.0\n",
      "0.9999229907989502 1.0\n",
      "0.0006840298301540315 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.9828438758850098 1.0\n",
      "0.999894380569458 1.0\n",
      "0.024058647453784943 0.0\n",
      "0.9999523162841797 1.0\n",
      "0.9995892643928528 1.0\n",
      "0.9999886751174927 1.0\n",
      "2.315362507943064e-05 0.0\n",
      "0.9999209642410278 1.0\n",
      "0.00019870680989697576 0.0\n",
      "5.173892350285314e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "9.492558092460968e-06 0.0\n",
      "0.999995231628418 1.0\n",
      "6.06659068580484e-06 0.0\n",
      "0.9999887943267822 1.0\n",
      "0.02013273350894451 0.0\n",
      "0.9993152618408203 1.0\n",
      "0.9998181462287903 1.0\n",
      "4.587935472954996e-05 0.0\n",
      "3.860701326630078e-06 0.0\n",
      "3.799777914537117e-06 0.0\n",
      "0.9999886751174927 1.0\n",
      "0.9999959468841553 1.0\n",
      "6.1105920394766144e-06 0.0\n",
      "3.573247113308753e-06 0.0\n",
      "4.460442141862586e-06 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.00017664622282609344 0.0\n",
      "6.911997388669988e-06 0.0\n",
      "3.300654498161748e-06 0.0\n",
      "0.9999885559082031 1.0\n",
      "6.724026752635837e-05 0.0\n",
      "0.0003786637098528445 0.0\n",
      "7.877654752519447e-06 0.0\n",
      "0.0003136804443784058 0.0\n",
      "0.9997279047966003 1.0\n",
      "0.9994145631790161 1.0\n",
      "0.9999382495880127 1.0\n",
      "0.9999856948852539 1.0\n",
      "6.772563665435882e-06 0.0\n",
      "0.9999561309814453 1.0\n",
      "0.999996542930603 1.0\n",
      "0.999984622001648 1.0\n",
      "0.9999963045120239 1.0\n",
      "4.561773948807968e-06 0.0\n",
      "TRAIN[steps=11200] loss=0.005425 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11200] loss=0.030578 acc=0.700 P=0.689 R=0.704 F1=0.696543 \n",
      "EPOCH:  18\n",
      "model sim and label tuples:\n",
      "0.9999896287918091 1.0\n",
      "0.999679446220398 1.0\n",
      "0.9999101161956787 1.0\n",
      "0.9999923706054688 1.0\n",
      "0.975601851940155 1.0\n",
      "0.9999786615371704 1.0\n",
      "0.9965729713439941 1.0\n",
      "0.9999480247497559 1.0\n",
      "0.00046268737060017884 0.0\n",
      "6.588751421077177e-05 0.0\n",
      "0.9999505281448364 1.0\n",
      "3.2128107250173343e-06 0.0\n",
      "6.693133036606014e-05 0.0\n",
      "9.018247510539368e-05 0.0\n",
      "0.0006191183347254992 0.0\n",
      "0.0007842922350391746 0.0\n",
      "0.9837054014205933 1.0\n",
      "0.999994158744812 1.0\n",
      "0.9893536567687988 1.0\n",
      "0.9999847412109375 1.0\n",
      "4.143380920140771e-06 0.0\n",
      "0.00010335096158087254 0.0\n",
      "0.001106341602280736 0.0\n",
      "2.5849898520391434e-05 0.0\n",
      "0.9999915361404419 1.0\n",
      "0.9823236465454102 1.0\n",
      "0.9997265934944153 1.0\n",
      "0.9999922513961792 1.0\n",
      "3.8798057175881695e-06 0.0\n",
      "0.9998620748519897 1.0\n",
      "6.526293873321265e-05 0.0\n",
      "0.9985143542289734 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999932050704956 1.0\n",
      "0.9999375343322754 1.0\n",
      "2.1218997062533163e-05 0.0\n",
      "2.1961363017908297e-05 0.0\n",
      "0.9999884366989136 1.0\n",
      "0.9999586343765259 1.0\n",
      "3.919969458365813e-06 0.0\n",
      "0.03610202297568321 0.0\n",
      "1.2925306691613514e-05 0.0\n",
      "4.091064965905389e-06 0.0\n",
      "0.9999927282333374 1.0\n",
      "0.9997454285621643 1.0\n",
      "3.921551069652196e-06 0.0\n",
      "3.364585836607148e-06 0.0\n",
      "0.9999911785125732 1.0\n",
      "0.9911310076713562 1.0\n",
      "8.570610589231364e-06 0.0\n",
      "0.001419240259565413 0.0\n",
      "0.9999967813491821 1.0\n",
      "0.8332266807556152 1.0\n",
      "4.21262893723906e-06 0.0\n",
      "4.720802735391771e-06 0.0\n",
      "5.800570306746522e-06 0.0\n",
      "4.0279603126691654e-05 0.0\n",
      "7.73330066294875e-06 0.0\n",
      "5.078590675111627e-06 0.0\n",
      "0.9999946355819702 1.0\n",
      "0.9999920129776001 1.0\n",
      "0.999995231628418 1.0\n",
      "1.677796899457462e-05 0.0\n",
      "0.9984777569770813 1.0\n",
      "TRAIN[steps=11300] loss=0.004854 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11300] loss=0.030304 acc=0.701 P=0.696 R=0.688 F1=0.692086 \n",
      "model sim and label tuples:\n",
      "0.0013515789760276675 0.0\n",
      "3.5057287277595606e-06 0.0\n",
      "0.9999961853027344 1.0\n",
      "3.778821792366216e-06 0.0\n",
      "0.9999885559082031 1.0\n",
      "4.565516974253114e-06 0.0\n",
      "7.2752704909362365e-06 0.0\n",
      "5.669476195180323e-06 0.0\n",
      "0.9999443292617798 1.0\n",
      "0.999992847442627 1.0\n",
      "1.2859737580583896e-05 0.0\n",
      "0.9999948740005493 1.0\n",
      "0.999711811542511 1.0\n",
      "1.2700140359811485e-05 0.0\n",
      "3.789261654674192e-06 0.0\n",
      "6.710605248372303e-06 0.0\n",
      "1.3139821021468379e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "2.0848518033744767e-05 0.0\n",
      "3.6873314002150437e-06 0.0\n",
      "0.9997910857200623 1.0\n",
      "9.476801642449573e-05 0.0\n",
      "1.3186581782065332e-05 0.0\n",
      "0.9999933242797852 1.0\n",
      "5.121494905324653e-05 0.0\n",
      "1.3122351447236724e-05 0.0\n",
      "0.9994801878929138 1.0\n",
      "0.9998891353607178 1.0\n",
      "1.734953730192501e-05 0.0\n",
      "0.0009392853826284409 0.0\n",
      "0.000209339676075615 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999943971633911 1.0\n",
      "0.00045927916653454304 0.0\n",
      "0.9999569654464722 1.0\n",
      "0.9998160004615784 1.0\n",
      "0.9999154806137085 1.0\n",
      "0.9999805688858032 1.0\n",
      "0.0030312410090118647 0.0\n",
      "0.9946349859237671 1.0\n",
      "0.9999034404754639 1.0\n",
      "0.9999935626983643 1.0\n",
      "0.000493489729706198 0.0\n",
      "1.35719237732701e-05 0.0\n",
      "0.9999833106994629 1.0\n",
      "0.9999775886535645 1.0\n",
      "0.00014040312089491636 0.0\n",
      "3.8504287658724934e-05 0.0\n",
      "0.9988480806350708 1.0\n",
      "0.004019289277493954 0.0\n",
      "0.9999969005584717 1.0\n",
      "0.00017409055726602674 0.0\n",
      "0.9980652928352356 1.0\n",
      "5.978537956252694e-06 0.0\n",
      "0.9999631643295288 1.0\n",
      "0.9999960660934448 1.0\n",
      "0.04451974481344223 0.0\n",
      "0.00016819393204059452 0.0\n",
      "0.998747706413269 1.0\n",
      "0.9999703168869019 1.0\n",
      "5.387560577219119e-06 0.0\n",
      "5.807554043713026e-05 0.0\n",
      "0.9998917579650879 1.0\n",
      "0.9996460676193237 1.0\n",
      "TRAIN[steps=11400] loss=0.001077 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11400] loss=0.030408 acc=0.700 P=0.687 R=0.709 F1=0.697651 \n",
      "model sim and label tuples:\n",
      "0.9998283386230469 1.0\n",
      "0.9999375343322754 1.0\n",
      "3.3034505122486735e-06 0.0\n",
      "3.490639755909797e-06 0.0\n",
      "0.9999622106552124 1.0\n",
      "0.9992191791534424 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.999976634979248 1.0\n",
      "0.99998939037323 1.0\n",
      "5.005541879654629e-06 0.0\n",
      "0.9865504503250122 1.0\n",
      "0.9998300075531006 1.0\n",
      "0.9979625940322876 1.0\n",
      "0.9999163150787354 1.0\n",
      "3.0953719942772295e-06 0.0\n",
      "0.00018195304437540472 0.0\n",
      "0.9999963045120239 1.0\n",
      "0.9997060894966125 1.0\n",
      "1.2428788977558725e-05 0.0\n",
      "3.756663136300631e-05 0.0\n",
      "1.4182988707034383e-05 0.0\n",
      "0.9999425411224365 1.0\n",
      "0.9999476671218872 1.0\n",
      "0.9999927282333374 1.0\n",
      "3.093073473792174e-06 0.0\n",
      "1.172119391412707e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9985507130622864 1.0\n",
      "0.9999895095825195 1.0\n",
      "2.6835023163584992e-05 0.0\n",
      "6.4924802245514e-06 0.0\n",
      "0.9999836683273315 1.0\n",
      "0.9999970197677612 1.0\n",
      "0.9999657869338989 1.0\n",
      "0.9999970197677612 1.0\n",
      "3.2042978546087397e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9999881982803345 1.0\n",
      "3.3373837595718214e-06 0.0\n",
      "0.9995928406715393 1.0\n",
      "0.9999958276748657 1.0\n",
      "4.4939292820345145e-06 0.0\n",
      "4.684941359300865e-06 0.0\n",
      "0.9998070597648621 1.0\n",
      "0.9998599290847778 1.0\n",
      "4.313951194490073e-06 0.0\n",
      "0.9999814033508301 1.0\n",
      "0.0010376301361247897 0.0\n",
      "0.0015491682570427656 0.0\n",
      "9.088211299967952e-06 0.0\n",
      "0.9999947547912598 1.0\n",
      "7.445875235134736e-05 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999898672103882 1.0\n",
      "0.9997952580451965 1.0\n",
      "0.0026017455384135246 0.0\n",
      "1.016900841932511e-05 0.0\n",
      "0.993285596370697 1.0\n",
      "3.389362609595992e-05 0.0\n",
      "5.806585249956697e-05 0.0\n",
      "0.0006845218595117331 0.0\n",
      "0.999230146408081 1.0\n",
      "0.9999110698699951 1.0\n",
      "0.9999407529830933 1.0\n",
      "TRAIN[steps=11500] loss=0.000530 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11500] loss=0.030848 acc=0.701 P=0.691 R=0.702 F1=0.696483 \n",
      "model sim and label tuples:\n",
      "0.9999963045120239 1.0\n",
      "6.955501248739893e-06 0.0\n",
      "1.0765310435090214e-05 0.0\n",
      "0.001817056443542242 0.0\n",
      "0.9998176693916321 1.0\n",
      "5.115598105476238e-05 0.0\n",
      "0.0007258654222823679 0.0\n",
      "3.699663238876383e-06 0.0\n",
      "0.010876123793423176 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.00010665755689842626 0.0\n",
      "6.056659458408831e-06 0.0\n",
      "0.9995185136795044 1.0\n",
      "0.0006614524754695594 0.0\n",
      "5.384457836044021e-06 0.0\n",
      "1.5002270629338454e-05 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.9999918937683105 1.0\n",
      "0.002609858289361 0.0\n",
      "0.9995520710945129 1.0\n",
      "4.242430804879405e-06 0.0\n",
      "0.9999719858169556 1.0\n",
      "0.9999908208847046 1.0\n",
      "4.513808107731165e-06 0.0\n",
      "1.608587808732409e-05 0.0\n",
      "1.2510283340816386e-05 0.0\n",
      "0.9999940395355225 1.0\n",
      "0.9998530149459839 1.0\n",
      "3.522401584632462e-06 0.0\n",
      "0.9999616146087646 1.0\n",
      "0.9879482984542847 1.0\n",
      "0.99981290102005 1.0\n",
      "4.291489403840387e-06 0.0\n",
      "0.0003544108185451478 0.0\n",
      "1.9203747797291726e-05 0.0\n",
      "0.9999902248382568 1.0\n",
      "4.291784352972172e-05 0.0\n",
      "0.005573034752160311 0.0\n",
      "0.9863130450248718 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999947547912598 1.0\n",
      "1.5107407307368703e-05 0.0\n",
      "0.0069719054736196995 0.0\n",
      "0.9999924898147583 1.0\n",
      "4.789879312738776e-06 0.0\n",
      "0.9999605417251587 1.0\n",
      "0.9999685287475586 1.0\n",
      "0.9998273253440857 1.0\n",
      "0.9999754428863525 1.0\n",
      "4.09535914513981e-06 0.0\n",
      "0.999977707862854 1.0\n",
      "0.9999929666519165 1.0\n",
      "0.9962596893310547 1.0\n",
      "3.5800926525553223e-06 0.0\n",
      "0.0005247491644695401 0.0\n",
      "0.9999966621398926 1.0\n",
      "3.283296337031061e-06 0.0\n",
      "0.00011862022074637935 0.0\n",
      "3.3776898362702923e-06 0.0\n",
      "0.9999963045120239 1.0\n",
      "3.3469937079644296e-06 0.0\n",
      "0.0013750531943514943 0.0\n",
      "7.960250513860956e-06 0.0\n",
      "4.358912519819569e-06 0.0\n",
      "TRAIN[steps=11600] loss=0.000994 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11600] loss=0.030511 acc=0.703 P=0.696 R=0.695 F1=0.695518 \n",
      "model sim and label tuples:\n",
      "9.022122867463622e-06 0.0\n",
      "5.082821189716924e-06 0.0\n",
      "0.9999386072158813 1.0\n",
      "0.0014496599324047565 0.0\n",
      "0.9998016953468323 1.0\n",
      "2.0741608750540763e-05 0.0\n",
      "0.9998103976249695 1.0\n",
      "3.5126627153658774e-06 0.0\n",
      "0.0014646734343841672 0.0\n",
      "0.999886155128479 1.0\n",
      "0.9999943971633911 1.0\n",
      "6.951408431632444e-05 0.0\n",
      "0.9999833106994629 1.0\n",
      "1.91075487236958e-05 0.0\n",
      "9.737258551467676e-06 0.0\n",
      "0.9999394416809082 1.0\n",
      "3.94298604078358e-06 0.0\n",
      "0.9999886751174927 1.0\n",
      "0.998975396156311 1.0\n",
      "8.289071047329344e-06 0.0\n",
      "4.779942537425086e-05 0.0\n",
      "0.9976915121078491 1.0\n",
      "0.999854326248169 1.0\n",
      "9.365699952468276e-05 0.0\n",
      "1.8417493265587837e-05 0.0\n",
      "0.9999685287475586 1.0\n",
      "0.9999878406524658 1.0\n",
      "0.9999939203262329 1.0\n",
      "0.00016949184646364301 0.0\n",
      "0.9999854564666748 1.0\n",
      "1.0271414794260636e-05 0.0\n",
      "0.00040942596388049424 0.0\n",
      "0.999796450138092 1.0\n",
      "0.00024394634237978607 0.0\n",
      "0.9999932050704956 1.0\n",
      "0.999996542930603 1.0\n",
      "0.9999892711639404 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.00017425465921405703 0.0\n",
      "0.999901533126831 1.0\n",
      "0.999993085861206 1.0\n",
      "0.9999899864196777 1.0\n",
      "0.9998362064361572 1.0\n",
      "0.9993763566017151 1.0\n",
      "0.9999963045120239 1.0\n",
      "5.2073195547563955e-05 0.0\n",
      "0.997490644454956 1.0\n",
      "4.615705165633699e-06 0.0\n",
      "0.9999940395355225 1.0\n",
      "4.86470071336953e-06 0.0\n",
      "0.9999911785125732 1.0\n",
      "0.999996542930603 1.0\n",
      "0.00023152894573286176 0.0\n",
      "0.9999731779098511 1.0\n",
      "0.9999967813491821 1.0\n",
      "0.999900221824646 1.0\n",
      "0.9999722242355347 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9991992115974426 1.0\n",
      "3.8135531212901697e-05 0.0\n",
      "3.4519771361374296e-06 0.0\n",
      "0.9993081092834473 1.0\n",
      "1.763096770446282e-05 0.0\n",
      "0.9999951124191284 1.0\n",
      "TRAIN[steps=11700] loss=0.000221 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11700] loss=0.030042 acc=0.703 P=0.698 R=0.689 F1=0.693625 \n",
      "model sim and label tuples:\n",
      "0.0025645126588642597 0.0\n",
      "0.0045217410661280155 0.0\n",
      "8.825998520478606e-05 0.0\n",
      "4.8761644393380266e-06 0.0\n",
      "0.9999221563339233 1.0\n",
      "0.9999654293060303 1.0\n",
      "8.422213431913406e-05 0.0\n",
      "4.404295395943336e-06 0.0\n",
      "0.9999511241912842 1.0\n",
      "0.000441633106674999 0.0\n",
      "0.9935152530670166 1.0\n",
      "4.766669371747412e-05 0.0\n",
      "0.0030353900510817766 0.0\n",
      "0.9999929666519165 1.0\n",
      "0.9999942779541016 1.0\n",
      "4.337121936259791e-06 0.0\n",
      "6.530944574478781e-06 0.0\n",
      "0.9999717473983765 1.0\n",
      "0.9998251795768738 1.0\n",
      "0.9891800880432129 1.0\n",
      "0.0020043260883539915 0.0\n",
      "0.9999920129776001 1.0\n",
      "0.9999301433563232 1.0\n",
      "0.9989199638366699 1.0\n",
      "0.0002962749858852476 0.0\n",
      "4.751547021442093e-05 0.0\n",
      "2.6357061869930476e-05 0.0\n",
      "5.304540081851883e-06 0.0\n",
      "0.9999963045120239 1.0\n",
      "0.000997742055915296 0.0\n",
      "0.00019630689348559827 0.0\n",
      "4.878457275481196e-06 0.0\n",
      "0.00012117584265070036 0.0\n",
      "4.150107542955084e-06 0.0\n",
      "1.0407655281596817e-05 0.0\n",
      "0.9941604733467102 1.0\n",
      "3.461698952378356e-06 0.0\n",
      "0.00014279616880230606 0.0\n",
      "3.887179900630144e-06 0.0\n",
      "3.6685169106931426e-06 0.0\n",
      "1.691710713203065e-05 0.0\n",
      "3.6217584238329437e-06 0.0\n",
      "0.999638557434082 1.0\n",
      "0.9995179176330566 1.0\n",
      "0.9999949932098389 1.0\n",
      "8.653818440507166e-06 0.0\n",
      "0.003531790804117918 0.0\n",
      "3.8776455767219886e-05 0.0\n",
      "5.9578487707767636e-06 0.0\n",
      "0.9999719858169556 1.0\n",
      "0.9998619556427002 1.0\n",
      "3.980628662247909e-06 0.0\n",
      "0.9999599456787109 1.0\n",
      "9.002389560919255e-06 0.0\n",
      "3.507599831209518e-05 0.0\n",
      "7.950349754537456e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.00041990139288827777 0.0\n",
      "0.999958872795105 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.009186387062072754 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999947547912598 1.0\n",
      "0.999995231628418 1.0\n",
      "TRAIN[steps=11800] loss=0.000842 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11800] loss=0.029962 acc=0.702 P=0.695 R=0.693 F1=0.694171 \n",
      "EPOCH:  19\n",
      "model sim and label tuples:\n",
      "0.9999910593032837 1.0\n",
      "3.222673694835976e-05 0.0\n",
      "9.858497651293874e-05 0.0\n",
      "0.9999963045120239 1.0\n",
      "3.6187095702189254e-06 0.0\n",
      "6.021298304403899e-06 0.0\n",
      "0.9931720495223999 1.0\n",
      "0.9972198009490967 1.0\n",
      "0.9655120372772217 1.0\n",
      "0.00010142903920495883 0.0\n",
      "0.0018573463894426823 0.0\n",
      "0.9999924898147583 1.0\n",
      "0.9999696016311646 1.0\n",
      "4.965054358763155e-06 0.0\n",
      "0.9999961853027344 1.0\n",
      "0.0016156804049387574 0.0\n",
      "5.9436501942400355e-06 0.0\n",
      "5.521985258383211e-06 0.0\n",
      "0.9993171691894531 1.0\n",
      "9.857469194685109e-06 0.0\n",
      "0.9999709129333496 1.0\n",
      "8.024167073017452e-06 0.0\n",
      "0.9999923706054688 1.0\n",
      "0.00013149027654435486 0.0\n",
      "0.9999805688858032 1.0\n",
      "7.805622590240091e-05 0.0\n",
      "0.990380048751831 1.0\n",
      "1.3143190699338447e-05 0.0\n",
      "0.9788699150085449 1.0\n",
      "0.9999076128005981 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.9999759197235107 1.0\n",
      "3.7987958876328776e-06 0.0\n",
      "4.024042027594987e-06 0.0\n",
      "1.3022668099438306e-05 0.0\n",
      "0.9998950958251953 1.0\n",
      "6.337596914818278e-06 0.0\n",
      "3.792070856434293e-06 0.0\n",
      "9.201021020999178e-05 0.0\n",
      "0.9995040893554688 1.0\n",
      "0.9999954700469971 1.0\n",
      "0.9956466555595398 1.0\n",
      "4.307545896153897e-05 0.0\n",
      "7.284344064828474e-06 0.0\n",
      "0.999922513961792 1.0\n",
      "0.9999958276748657 1.0\n",
      "4.726740371552296e-06 0.0\n",
      "0.9999943971633911 1.0\n",
      "5.29314002051251e-06 0.0\n",
      "5.252940354694147e-06 0.0\n",
      "0.9998075366020203 1.0\n",
      "4.338417056715116e-06 0.0\n",
      "0.000465808785520494 0.0\n",
      "0.9954987168312073 1.0\n",
      "8.297250315081328e-05 0.0\n",
      "0.9999551773071289 1.0\n",
      "0.9999880790710449 1.0\n",
      "9.483872418059036e-06 0.0\n",
      "0.00011481266119517386 0.0\n",
      "5.434731974673923e-06 0.0\n",
      "0.9999897480010986 1.0\n",
      "3.922130872524576e-06 0.0\n",
      "0.9999076128005981 1.0\n",
      "TRAIN[steps=11900] loss=0.001429 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=11900] loss=0.029867 acc=0.699 P=0.687 R=0.705 F1=0.695925 \n",
      "model sim and label tuples:\n",
      "4.204978722555097e-06 0.0\n",
      "4.8156711272895336e-05 0.0\n",
      "0.9999889135360718 1.0\n",
      "0.9997475743293762 1.0\n",
      "5.280026471154997e-06 0.0\n",
      "0.03759419173002243 1.0\n",
      "0.999295711517334 1.0\n",
      "4.117218395549571e-06 0.0\n",
      "3.955154625145951e-06 0.0\n",
      "0.9999371767044067 1.0\n",
      "0.9999877214431763 1.0\n",
      "0.9998980760574341 1.0\n",
      "0.999996542930603 1.0\n",
      "0.9980745315551758 1.0\n",
      "0.999984860420227 1.0\n",
      "5.70333713767468e-06 0.0\n",
      "0.06746213883161545 0.0\n",
      "2.556765866756905e-05 0.0\n",
      "0.9995529055595398 1.0\n",
      "0.9999185800552368 1.0\n",
      "1.3560136721935123e-05 0.0\n",
      "0.00026431772857904434 0.0\n",
      "1.5618386896676384e-05 0.0\n",
      "0.0008494288777001202 0.0\n",
      "0.0005515607772395015 0.0\n",
      "0.9999947547912598 1.0\n",
      "1.456140944355866e-05 0.0\n",
      "4.721770892501809e-06 0.0\n",
      "6.905648479005322e-05 0.0\n",
      "0.21670600771903992 0.0\n",
      "0.0006019736174494028 0.0\n",
      "5.233030969975516e-05 0.0\n",
      "0.9999955892562866 1.0\n",
      "0.9999246597290039 1.0\n",
      "0.9999866485595703 1.0\n",
      "0.999996542930603 1.0\n",
      "0.00023176154354587197 0.0\n",
      "3.668905492304475e-06 0.0\n",
      "0.9999768733978271 1.0\n",
      "0.9999874830245972 1.0\n",
      "0.9990377426147461 1.0\n",
      "0.9999924898147583 1.0\n",
      "0.9999955892562866 1.0\n",
      "2.6816296667675488e-05 0.0\n",
      "0.9995111227035522 1.0\n",
      "4.0199838622356765e-06 0.0\n",
      "0.9999384880065918 1.0\n",
      "0.9999949932098389 1.0\n",
      "0.999992847442627 1.0\n",
      "0.004653915297240019 0.0\n",
      "0.9999902248382568 1.0\n",
      "2.5573048333171755e-05 0.0\n",
      "3.66826157005562e-06 0.0\n",
      "7.0728260652686e-06 0.0\n",
      "0.999403715133667 1.0\n",
      "8.773328772804234e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.9928596019744873 1.0\n",
      "0.9997828602790833 1.0\n",
      "0.9999912977218628 1.0\n",
      "0.9999618530273438 1.0\n",
      "3.7114311908226227e-06 0.0\n",
      "0.9999573230743408 1.0\n",
      "0.9999372959136963 1.0\n",
      "TRAIN[steps=12000] loss=0.056499 acc=0.984 P=1.000 R=0.971 F1=0.985507\n",
      "DEV[steps=12000] loss=0.029850 acc=0.699 P=0.689 R=0.700 F1=0.694512 \n",
      "model sim and label tuples:\n",
      "4.0745485421211924e-06 0.0\n",
      "5.555181814997923e-06 0.0\n",
      "0.037451207637786865 0.0\n",
      "0.0002458335948176682 0.0\n",
      "0.00018300169904250652 0.0\n",
      "0.9999805688858032 1.0\n",
      "0.999983549118042 1.0\n",
      "6.729652795911534e-06 0.0\n",
      "0.0021521111484616995 0.0\n",
      "0.9999645948410034 1.0\n",
      "0.00020271366520319134 0.0\n",
      "4.732093293569051e-05 0.0\n",
      "0.0009643551311455667 0.0\n",
      "0.999594509601593 1.0\n",
      "9.077815047930926e-05 0.0\n",
      "0.9998714923858643 1.0\n",
      "0.996377170085907 1.0\n",
      "0.9999948740005493 1.0\n",
      "0.9997777342796326 1.0\n",
      "3.430095603107475e-05 0.0\n",
      "0.9988133907318115 1.0\n",
      "0.9999889135360718 1.0\n",
      "0.9999747276306152 1.0\n",
      "0.9999451637268066 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.9998836517333984 1.0\n",
      "4.3880671000806615e-05 0.0\n",
      "4.300498403608799e-06 0.0\n",
      "0.9999958276748657 1.0\n",
      "2.1565661882050335e-05 0.0\n",
      "0.9994938373565674 1.0\n",
      "2.8629034204641357e-05 0.0\n",
      "7.976466804393567e-06 0.0\n",
      "0.000812761951237917 0.0\n",
      "5.090995273349108e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "0.0014492050977423787 0.0\n",
      "0.9999946355819702 1.0\n",
      "3.859387106786016e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.9996932744979858 1.0\n",
      "0.02165057882666588 0.0\n",
      "0.00019077223259955645 0.0\n",
      "5.417339707491919e-06 0.0\n",
      "0.9999889135360718 1.0\n",
      "0.00022152747260406613 0.0\n",
      "0.9998063445091248 1.0\n",
      "0.9998713731765747 1.0\n",
      "0.9999936819076538 1.0\n",
      "0.9999922513961792 1.0\n",
      "4.345468369137961e-06 0.0\n",
      "0.9955343008041382 1.0\n",
      "3.7987440009601414e-05 0.0\n",
      "4.7089188228710555e-06 0.0\n",
      "0.9999948740005493 1.0\n",
      "0.0005757372127845883 0.0\n",
      "8.611585144535638e-06 0.0\n",
      "7.363689746853197e-06 0.0\n",
      "0.00042134226532652974 0.0\n",
      "0.9999325275421143 1.0\n",
      "0.9998082518577576 1.0\n",
      "3.854722763207974e-06 0.0\n",
      "2.7982054234598763e-05 0.0\n",
      "5.683670678990893e-06 0.0\n",
      "TRAIN[steps=12100] loss=0.001245 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12100] loss=0.029865 acc=0.699 P=0.689 R=0.700 F1=0.694467 \n",
      "model sim and label tuples:\n",
      "5.780215360573493e-06 0.0\n",
      "0.9996681213378906 1.0\n",
      "0.9999898672103882 1.0\n",
      "3.598335388232954e-05 0.0\n",
      "3.93694790545851e-06 0.0\n",
      "0.9998020529747009 1.0\n",
      "0.00045221822801977396 0.0\n",
      "0.006425758358091116 0.0\n",
      "6.977976681810105e-06 0.0\n",
      "0.9999923706054688 1.0\n",
      "0.9979000091552734 1.0\n",
      "7.580392320960527e-06 0.0\n",
      "0.999984622001648 1.0\n",
      "6.171947006805567e-06 0.0\n",
      "0.9997720122337341 1.0\n",
      "0.9999940395355225 1.0\n",
      "0.00035661127185449004 0.0\n",
      "0.0001051609797286801 0.0\n",
      "3.7763177260785596e-06 0.0\n",
      "0.0005359317292459309 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.99989914894104 1.0\n",
      "0.9998012185096741 1.0\n",
      "4.6985893277451396e-05 0.0\n",
      "8.710194379091263e-06 0.0\n",
      "0.9999821186065674 1.0\n",
      "0.9999904632568359 1.0\n",
      "0.9981733560562134 1.0\n",
      "0.001075900043360889 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.9999889135360718 1.0\n",
      "0.0002649631642270833 0.0\n",
      "0.9999645948410034 1.0\n",
      "0.9999911785125732 1.0\n",
      "9.022312042361591e-06 0.0\n",
      "9.678054993855767e-06 0.0\n",
      "8.417830395046622e-05 0.0\n",
      "4.1875341594277415e-06 0.0\n",
      "0.9994109869003296 1.0\n",
      "6.66535288473824e-06 0.0\n",
      "0.99998939037323 1.0\n",
      "0.0008316081948578358 0.0\n",
      "1.2080130545655265e-05 0.0\n",
      "0.6506528258323669 1.0\n",
      "4.126904514123453e-06 0.0\n",
      "5.21927449881332e-06 0.0\n",
      "1.206037723022746e-05 0.0\n",
      "0.9999896287918091 1.0\n",
      "3.051989187952131e-05 0.0\n",
      "0.01973484642803669 0.0\n",
      "0.9943389296531677 1.0\n",
      "0.021344132721424103 0.0\n",
      "0.9742943644523621 1.0\n",
      "0.9981229901313782 1.0\n",
      "0.9998182654380798 1.0\n",
      "5.2064033297938295e-06 0.0\n",
      "0.9999818801879883 1.0\n",
      "5.2054569096071646e-05 0.0\n",
      "5.580717697739601e-06 0.0\n",
      "0.9990620017051697 1.0\n",
      "0.9929372668266296 1.0\n",
      "4.523853476712247e-06 0.0\n",
      "5.71989630770986e-06 0.0\n",
      "0.9999897480010986 1.0\n",
      "TRAIN[steps=12200] loss=0.008270 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12200] loss=0.029893 acc=0.699 P=0.688 R=0.703 F1=0.695590 \n",
      "model sim and label tuples:\n",
      "0.00017888718866743147 0.0\n",
      "0.9997612833976746 1.0\n",
      "0.9999947547912598 1.0\n",
      "3.849532276944956e-06 0.0\n",
      "9.261008926841896e-06 0.0\n",
      "0.9999938011169434 1.0\n",
      "1.5125612662814092e-05 0.0\n",
      "0.00029290892416611314 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.9995349645614624 1.0\n",
      "6.333614692266565e-06 0.0\n",
      "3.762396318052197e-06 0.0\n",
      "0.9989068508148193 1.0\n",
      "0.9999185800552368 1.0\n",
      "0.9999622106552124 1.0\n",
      "0.9964454770088196 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999945163726807 1.0\n",
      "0.9998801946640015 1.0\n",
      "0.0023388268891721964 0.0\n",
      "0.00011024020932381973 0.0\n",
      "0.0034958156757056713 0.0\n",
      "0.9999737739562988 1.0\n",
      "0.999914288520813 1.0\n",
      "0.9999746084213257 1.0\n",
      "3.852885129163042e-06 0.0\n",
      "9.94528363662539e-06 0.0\n",
      "0.00046607860713265836 0.0\n",
      "3.2282256142934784e-05 0.0\n",
      "2.1427204046631232e-05 0.0\n",
      "0.9997225403785706 1.0\n",
      "3.7262661862769164e-06 0.0\n",
      "0.0004083620442543179 0.0\n",
      "0.0008852994651533663 0.0\n",
      "0.9998857975006104 1.0\n",
      "8.75909154274268e-06 0.0\n",
      "8.476329639961477e-06 0.0\n",
      "4.932188676320948e-05 0.0\n",
      "0.9999963045120239 1.0\n",
      "0.9977040886878967 1.0\n",
      "0.9999639987945557 1.0\n",
      "4.1464877540420275e-06 0.0\n",
      "6.372044026647927e-06 0.0\n",
      "0.00944663118571043 0.0\n",
      "7.784200533933472e-06 0.0\n",
      "0.00010430847760289907 0.0\n",
      "0.9995369911193848 1.0\n",
      "0.9999823570251465 1.0\n",
      "1.2068937394360546e-05 0.0\n",
      "0.9998834133148193 1.0\n",
      "1.1290940165054053e-05 0.0\n",
      "2.8778547857655212e-05 0.0\n",
      "6.473483517765999e-05 0.0\n",
      "4.624384928320069e-06 0.0\n",
      "8.029962373257149e-06 0.0\n",
      "1.0010026016971096e-05 0.0\n",
      "4.0515938053431455e-06 0.0\n",
      "0.999995231628418 1.0\n",
      "0.9999470710754395 1.0\n",
      "3.679753717733547e-05 0.0\n",
      "4.087797151441919e-06 0.0\n",
      "6.899677828187123e-05 0.0\n",
      "0.9999127388000488 1.0\n",
      "5.69823805562919e-06 0.0\n",
      "TRAIN[steps=12300] loss=0.000429 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12300] loss=0.029924 acc=0.699 P=0.689 R=0.703 F1=0.695546 \n",
      "model sim and label tuples:\n",
      "0.9993212223052979 1.0\n",
      "9.513182158116251e-05 0.0\n",
      "0.9988898634910583 1.0\n",
      "0.9999955892562866 1.0\n",
      "0.9999604225158691 1.0\n",
      "0.0269762072712183 0.0\n",
      "0.9999393224716187 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.042055707424879074 0.0\n",
      "0.994212806224823 1.0\n",
      "0.9941920638084412 1.0\n",
      "0.9999399185180664 1.0\n",
      "0.9996253252029419 1.0\n",
      "4.731724402518012e-06 0.0\n",
      "0.00012428686022758484 0.0\n",
      "0.0011976161040365696 0.0\n",
      "0.00014238039148040116 0.0\n",
      "0.002736348658800125 0.0\n",
      "0.9978876709938049 1.0\n",
      "4.474105480767321e-06 0.0\n",
      "0.00043011634261347353 0.0\n",
      "0.9997122883796692 1.0\n",
      "1.211600101669319e-05 0.0\n",
      "3.7664601677533938e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "7.979101064847782e-05 0.0\n",
      "0.9999889135360718 1.0\n",
      "0.999647855758667 1.0\n",
      "4.601228283718228e-06 0.0\n",
      "0.9999954700469971 1.0\n",
      "4.644370619644178e-06 0.0\n",
      "0.9999324083328247 1.0\n",
      "0.9998799562454224 1.0\n",
      "0.9999891519546509 1.0\n",
      "0.9999567270278931 1.0\n",
      "0.9998743534088135 1.0\n",
      "0.999996542930603 1.0\n",
      "0.00021480211580637842 0.0\n",
      "6.096871857153019e-06 0.0\n",
      "0.9999958276748657 1.0\n",
      "7.571643800474703e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.00010816016583703458 0.0\n",
      "0.0006176481838338077 0.0\n",
      "0.0005483044078573585 0.0\n",
      "0.9998922348022461 1.0\n",
      "5.045040961704217e-05 0.0\n",
      "0.999995231628418 1.0\n",
      "0.9999933242797852 1.0\n",
      "4.129983153688954e-06 0.0\n",
      "8.027297553780954e-06 0.0\n",
      "0.9999148845672607 1.0\n",
      "8.986477951111738e-06 0.0\n",
      "0.9999922513961792 1.0\n",
      "4.660217018681578e-05 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.0001178332167910412 0.0\n",
      "1.9807413991657086e-05 0.0\n",
      "1.3533446690416895e-05 0.0\n",
      "0.9999865293502808 1.0\n",
      "7.146438292693347e-06 1.0\n",
      "2.885147841880098e-05 0.0\n",
      "1.4371905308507849e-05 0.0\n",
      "3.7149009131098865e-06 0.0\n",
      "TRAIN[steps=12400] loss=0.186614 acc=0.984 P=1.000 R=0.969 F1=0.984127\n",
      "DEV[steps=12400] loss=0.029969 acc=0.699 P=0.689 R=0.701 F1=0.694927 \n",
      "EPOCH:  20\n",
      "model sim and label tuples:\n",
      "3.751747271962813e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9954297542572021 1.0\n",
      "0.9999129772186279 1.0\n",
      "9.268581197829917e-05 0.0\n",
      "2.563607085903641e-05 0.0\n",
      "0.9995892643928528 1.0\n",
      "0.9999890327453613 1.0\n",
      "0.9999682903289795 1.0\n",
      "0.9999043941497803 1.0\n",
      "0.9999834299087524 1.0\n",
      "0.9999904632568359 1.0\n",
      "4.280510438547935e-06 0.0\n",
      "0.999988317489624 1.0\n",
      "0.999995231628418 1.0\n",
      "0.013590246438980103 0.0\n",
      "0.9983750581741333 1.0\n",
      "0.0005496306112036109 0.0\n",
      "0.0008186182822100818 0.0\n",
      "9.091632819036022e-05 0.0\n",
      "2.82772980426671e-05 0.0\n",
      "0.9996389150619507 1.0\n",
      "7.836136319383513e-06 0.0\n",
      "5.909872925258242e-06 0.0\n",
      "0.9999672174453735 1.0\n",
      "0.999955415725708 1.0\n",
      "1.0578962246654555e-05 0.0\n",
      "0.0006908205104991794 0.0\n",
      "0.9999833106994629 1.0\n",
      "0.9999831914901733 1.0\n",
      "0.001131474389694631 0.0\n",
      "0.9999549388885498 1.0\n",
      "0.00044543255353346467 0.0\n",
      "0.002255051862448454 0.0\n",
      "0.9996097683906555 1.0\n",
      "7.265265594469383e-05 0.0\n",
      "0.9996200799942017 1.0\n",
      "0.9999957084655762 1.0\n",
      "6.5512367655173875e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.9999746084213257 1.0\n",
      "0.0018578864401206374 0.0\n",
      "0.00025435228599235415 0.0\n",
      "0.9995400905609131 1.0\n",
      "0.0001737510901875794 0.0\n",
      "0.9993427395820618 1.0\n",
      "0.999841570854187 1.0\n",
      "3.507919473122456e-06 0.0\n",
      "0.9997181296348572 1.0\n",
      "9.269482688978314e-06 0.0\n",
      "5.0691825890680775e-05 0.0\n",
      "0.0031708329916000366 0.0\n",
      "0.999893069267273 1.0\n",
      "0.9999948740005493 1.0\n",
      "0.00012427702313289046 0.0\n",
      "0.00010560564987827092 0.0\n",
      "0.9999853372573853 1.0\n",
      "0.9992378950119019 1.0\n",
      "4.024164809379727e-06 0.0\n",
      "0.000391494482755661 0.0\n",
      "0.9994007349014282 1.0\n",
      "0.9999960660934448 1.0\n",
      "3.0574512493330985e-05 0.0\n",
      "0.9999924898147583 1.0\n",
      "TRAIN[steps=12500] loss=0.000584 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12500] loss=0.029986 acc=0.700 P=0.690 R=0.699 F1=0.694634 \n",
      "model sim and label tuples:\n",
      "9.164916991721839e-05 0.0\n",
      "0.9999774694442749 1.0\n",
      "0.0020430691074579954 0.0\n",
      "0.999660849571228 1.0\n",
      "0.9999631643295288 1.0\n",
      "4.7795001592021435e-05 0.0\n",
      "0.0030854048673063517 0.0\n",
      "3.6449146136874333e-06 0.0\n",
      "3.5675882372743217e-06 0.0\n",
      "9.44269140745746e-06 0.0\n",
      "0.0021609633695334196 0.0\n",
      "0.9730480313301086 1.0\n",
      "0.9984252452850342 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999252557754517 1.0\n",
      "0.9994742274284363 1.0\n",
      "6.480652245954843e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "2.445553764118813e-05 0.0\n",
      "0.9996203184127808 1.0\n",
      "1.7524638678878546e-05 0.0\n",
      "0.9999468326568604 1.0\n",
      "0.9999759197235107 1.0\n",
      "7.972649655130226e-06 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.9997859597206116 1.0\n",
      "0.973987340927124 1.0\n",
      "0.9997575879096985 1.0\n",
      "0.999996542930603 1.0\n",
      "0.0035272175446152687 0.0\n",
      "0.9999207258224487 1.0\n",
      "0.9999886751174927 1.0\n",
      "7.755848855595104e-06 0.0\n",
      "0.00033134460682049394 0.0\n",
      "5.1462480769259855e-05 0.0\n",
      "0.999992847442627 1.0\n",
      "0.9999949932098389 1.0\n",
      "0.999992847442627 1.0\n",
      "4.414505383465439e-06 0.0\n",
      "5.030455668020295e-06 0.0\n",
      "0.9995608925819397 1.0\n",
      "5.485198016685899e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9999765157699585 1.0\n",
      "0.999994158744812 1.0\n",
      "1.781957871571649e-05 0.0\n",
      "3.90448576581548e-06 0.0\n",
      "0.9998972415924072 1.0\n",
      "0.9999938011169434 1.0\n",
      "0.02670516073703766 0.0\n",
      "4.145796083321329e-06 0.0\n",
      "0.00024937777197919786 0.0\n",
      "0.9999765157699585 1.0\n",
      "0.9977680444717407 1.0\n",
      "9.817409591050819e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "4.435427854332374e-06 0.0\n",
      "0.9995868802070618 1.0\n",
      "0.9606373310089111 1.0\n",
      "0.9997805953025818 1.0\n",
      "0.9993923902511597 1.0\n",
      "3.844934781227494e-06 0.0\n",
      "3.5825894428853644e-06 0.0\n",
      "9.701331691758242e-06 0.0\n",
      "TRAIN[steps=12600] loss=0.002193 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12600] loss=0.030017 acc=0.700 P=0.691 R=0.696 F1=0.693627 \n",
      "model sim and label tuples:\n",
      "0.999984860420227 1.0\n",
      "0.9998283386230469 1.0\n",
      "0.999962568283081 1.0\n",
      "0.005904470570385456 0.0\n",
      "0.9984737038612366 1.0\n",
      "3.815603122347966e-06 0.0\n",
      "6.076939098420553e-05 0.0\n",
      "0.0002126386243617162 0.0\n",
      "0.9872263073921204 1.0\n",
      "0.9999488592147827 1.0\n",
      "0.9999775886535645 1.0\n",
      "0.9999924898147583 1.0\n",
      "0.00036759692011401057 0.0\n",
      "4.761777745443396e-06 0.0\n",
      "0.9999845027923584 1.0\n",
      "6.353191110974876e-06 0.0\n",
      "0.999843955039978 1.0\n",
      "0.9999864101409912 1.0\n",
      "0.9999819993972778 1.0\n",
      "0.9999831914901733 1.0\n",
      "0.966681718826294 1.0\n",
      "0.9975787997245789 1.0\n",
      "1.725178117339965e-05 0.0\n",
      "0.9997923970222473 1.0\n",
      "6.307370767899556e-06 0.0\n",
      "3.758537104658899e-06 0.0\n",
      "0.00165455078240484 0.0\n",
      "0.9999855756759644 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.00017294516146648675 0.0\n",
      "0.002759881317615509 0.0\n",
      "3.827739419648424e-06 0.0\n",
      "0.9999655485153198 1.0\n",
      "0.004966326989233494 0.0\n",
      "0.9998853206634521 1.0\n",
      "0.9921782612800598 1.0\n",
      "0.999993085861206 1.0\n",
      "0.00146114535164088 0.0\n",
      "9.267697896575555e-05 0.0\n",
      "0.9984519481658936 1.0\n",
      "1.0823027878359426e-05 0.0\n",
      "0.9975850582122803 1.0\n",
      "0.9786645174026489 1.0\n",
      "4.27012992076925e-06 0.0\n",
      "0.9976392984390259 1.0\n",
      "0.9999771118164062 1.0\n",
      "1.6847989172674716e-05 0.0\n",
      "0.9681356549263 1.0\n",
      "0.0010326117044314742 0.0\n",
      "0.9944242835044861 1.0\n",
      "4.499869646679144e-06 0.0\n",
      "0.580586850643158 0.0\n",
      "0.0065482547506690025 0.0\n",
      "3.5937450775236357e-06 0.0\n",
      "0.00015943095786496997 0.0\n",
      "0.9998608827590942 1.0\n",
      "0.9999816417694092 1.0\n",
      "9.270959708373994e-06 0.0\n",
      "4.363245807326166e-06 0.0\n",
      "0.9143821001052856 1.0\n",
      "0.9999886751174927 1.0\n",
      "3.7662118756998098e-06 0.0\n",
      "0.005378748290240765 0.0\n",
      "4.205551158520393e-05 0.0\n",
      "TRAIN[steps=12700] loss=0.017421 acc=0.984 P=0.971 R=1.000 F1=0.985075\n",
      "DEV[steps=12700] loss=0.030044 acc=0.699 P=0.691 R=0.696 F1=0.693486 \n",
      "model sim and label tuples:\n",
      "0.00018098200962413102 0.0\n",
      "3.259404547861777e-05 0.0\n",
      "3.79927428184601e-06 0.0\n",
      "1.8216682292404585e-05 0.0\n",
      "0.9997474551200867 1.0\n",
      "0.0039592040702700615 0.0\n",
      "1.4488285160041414e-05 0.0\n",
      "8.101088315015659e-05 0.0\n",
      "0.01900825835764408 0.0\n",
      "1.4944665963412262e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999802112579346 1.0\n",
      "4.711016572400695e-06 0.0\n",
      "4.69536280434113e-06 0.0\n",
      "3.5929397199652158e-06 0.0\n",
      "0.9999418258666992 1.0\n",
      "0.9942740797996521 1.0\n",
      "0.9871886372566223 1.0\n",
      "1.1538946637301706e-05 0.0\n",
      "5.798319307359634e-06 0.0\n",
      "1.1380146133888047e-05 0.0\n",
      "3.7629629332514014e-06 0.0\n",
      "0.005146726034581661 0.0\n",
      "1.1170174730068538e-05 0.0\n",
      "0.0010777022689580917 0.0\n",
      "7.226114121294813e-06 0.0\n",
      "0.9990112781524658 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.999914288520813 1.0\n",
      "0.9999945163726807 1.0\n",
      "0.9999656677246094 1.0\n",
      "0.0005577219417318702 0.0\n",
      "0.9999815225601196 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9997150301933289 1.0\n",
      "2.100423807860352e-05 0.0\n",
      "0.007017461117357016 0.0\n",
      "4.310554231778951e-06 0.0\n",
      "0.9999940395355225 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.999995231628418 1.0\n",
      "7.209504019556334e-06 0.0\n",
      "1.4623851711803582e-05 0.0\n",
      "0.9998235106468201 1.0\n",
      "0.999988317489624 1.0\n",
      "2.1148673113202676e-05 0.0\n",
      "0.9999830722808838 1.0\n",
      "0.9996612071990967 1.0\n",
      "0.00010546767589403316 0.0\n",
      "1.7118378309533e-05 0.0\n",
      "0.002525466727092862 0.0\n",
      "0.9999707937240601 1.0\n",
      "0.9999711513519287 1.0\n",
      "5.238513040239923e-05 0.0\n",
      "3.197077239747159e-05 0.0\n",
      "7.88984980317764e-05 0.0\n",
      "0.9999463558197021 1.0\n",
      "0.9999725818634033 1.0\n",
      "3.5406555980443954e-06 0.0\n",
      "2.1238634872133844e-05 0.0\n",
      "0.9991340041160583 1.0\n",
      "0.005837358999997377 0.0\n",
      "0.9999794960021973 1.0\n",
      "TRAIN[steps=12800] loss=0.001065 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12800] loss=0.030097 acc=0.699 P=0.689 R=0.699 F1=0.693882 \n",
      "model sim and label tuples:\n",
      "0.999988317489624 1.0\n",
      "0.9840547442436218 1.0\n",
      "0.999351441860199 1.0\n",
      "8.171475201379508e-05 0.0\n",
      "0.9999868869781494 1.0\n",
      "1.4235239177651238e-05 0.0\n",
      "0.9993440508842468 1.0\n",
      "0.9999933242797852 1.0\n",
      "0.9998903274536133 1.0\n",
      "0.002892027609050274 0.0\n",
      "0.997273862361908 1.0\n",
      "0.9999836683273315 1.0\n",
      "0.9999898672103882 1.0\n",
      "0.9999843835830688 1.0\n",
      "3.548704171407735e-06 0.0\n",
      "0.9998261332511902 1.0\n",
      "0.001991893397644162 0.0\n",
      "0.9999276399612427 1.0\n",
      "0.00025640608510002494 0.0\n",
      "0.002269890857860446 0.0\n",
      "0.9978812336921692 1.0\n",
      "0.9996600151062012 1.0\n",
      "0.0001735752448439598 0.0\n",
      "1.1727252967830282e-05 0.0\n",
      "0.9999939203262329 1.0\n",
      "9.129360478254966e-06 0.0\n",
      "0.9999924898147583 1.0\n",
      "0.9999682903289795 1.0\n",
      "0.9930752515792847 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9978055357933044 1.0\n",
      "0.9996182918548584 1.0\n",
      "0.9999808073043823 1.0\n",
      "0.9999842643737793 1.0\n",
      "0.9999669790267944 1.0\n",
      "0.9997088313102722 1.0\n",
      "0.00013112805027049035 0.0\n",
      "0.0006640326464548707 0.0\n",
      "0.00019107625121250749 0.0\n",
      "0.9999938011169434 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.8214594125747681 1.0\n",
      "0.0006326826987788081 0.0\n",
      "0.0012623901711776853 0.0\n",
      "0.000167164922459051 0.0\n",
      "5.9356234487495385e-06 0.0\n",
      "0.019596196711063385 0.0\n",
      "0.9991428852081299 1.0\n",
      "0.9998301267623901 1.0\n",
      "0.9999716281890869 1.0\n",
      "0.999931812286377 1.0\n",
      "0.011356133967638016 0.0\n",
      "6.782012405892601e-06 0.0\n",
      "3.681459929794073e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999964237213135 1.0\n",
      "4.386647560750134e-06 0.0\n",
      "9.873152521322481e-06 0.0\n",
      "4.9264021072303876e-05 0.0\n",
      "3.398267836018931e-06 0.0\n",
      "1.2184864317532629e-05 0.0\n",
      "5.3553289944829885e-06 0.0\n",
      "0.006017791572958231 0.0\n",
      "0.9999958276748657 1.0\n",
      "TRAIN[steps=12900] loss=0.004357 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=12900] loss=0.030188 acc=0.699 P=0.689 R=0.700 F1=0.694255 \n",
      "model sim and label tuples:\n",
      "0.9999957084655762 1.0\n",
      "0.999975323677063 1.0\n",
      "0.0009325369028374553 0.0\n",
      "4.661085768020712e-06 0.0\n",
      "1.0716828910517506e-05 0.0\n",
      "7.306204679480288e-06 0.0\n",
      "0.9996380805969238 1.0\n",
      "0.9998745918273926 1.0\n",
      "3.0181570764398202e-05 0.0\n",
      "1.3811387361784e-05 0.0\n",
      "0.9999878406524658 1.0\n",
      "0.983453094959259 1.0\n",
      "1.0250131708744448e-05 0.0\n",
      "0.00016855790454428643 0.0\n",
      "3.898148861480877e-05 0.0\n",
      "2.124355705745984e-05 0.0\n",
      "3.4424499517626828e-06 0.0\n",
      "0.00020759014296345413 0.0\n",
      "5.549071374844061e-06 0.0\n",
      "3.829201523330994e-05 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.9999897480010986 1.0\n",
      "0.9999961853027344 1.0\n",
      "1.2509448424680158e-05 0.0\n",
      "6.525441131088883e-05 0.0\n",
      "0.9999837875366211 1.0\n",
      "0.009593287482857704 0.0\n",
      "0.0006598371546715498 0.0\n",
      "0.9564822912216187 1.0\n",
      "0.99998939037323 1.0\n",
      "0.9998687505722046 1.0\n",
      "0.9997816681861877 1.0\n",
      "0.9998918771743774 1.0\n",
      "1.905863064166624e-05 0.0\n",
      "5.413819962996058e-05 0.0\n",
      "5.43845453648828e-06 0.0\n",
      "0.9995521903038025 1.0\n",
      "0.9980000853538513 1.0\n",
      "4.321445430832682e-06 0.0\n",
      "0.9998459815979004 1.0\n",
      "0.00026024325052276254 0.0\n",
      "3.4623394640220795e-06 0.0\n",
      "5.87560953135835e-06 0.0\n",
      "0.9999293088912964 1.0\n",
      "0.9847993850708008 1.0\n",
      "0.999982476234436 1.0\n",
      "0.9999924898147583 1.0\n",
      "3.6906949389958754e-06 0.0\n",
      "3.587981382224825e-06 0.0\n",
      "2.232701262983028e-05 0.0\n",
      "4.653206815419253e-06 0.0\n",
      "4.206033281661803e-06 0.0\n",
      "9.318709999206476e-06 0.0\n",
      "0.9999798536300659 1.0\n",
      "0.999956488609314 1.0\n",
      "0.9999927282333374 1.0\n",
      "0.999823272228241 1.0\n",
      "0.9999860525131226 1.0\n",
      "0.9999853372573853 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.18684476613998413 0.0\n",
      "0.9999816417694092 1.0\n",
      "2.2700800400343724e-05 0.0\n",
      "0.9999878406524658 1.0\n",
      "TRAIN[steps=13000] loss=0.004682 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13000] loss=0.030213 acc=0.699 P=0.689 R=0.699 F1=0.694201 \n",
      "model sim and label tuples:\n",
      "0.0028542368672788143 0.0\n",
      "0.00040992762660607696 0.0\n",
      "0.9999377727508545 1.0\n",
      "0.0082665104418993 0.0\n",
      "0.9999239444732666 1.0\n",
      "0.0001780015736585483 0.0\n",
      "4.374557647679467e-06 0.0\n",
      "5.897194114368176e-06 0.0\n",
      "0.9999897480010986 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999942779541016 1.0\n",
      "1.1001341590599623e-05 0.0\n",
      "1.6524898455827497e-05 0.0\n",
      "5.083846554043703e-05 0.0\n",
      "0.99996018409729 1.0\n",
      "4.8506815801374614e-05 0.0\n",
      "4.335075118433451e-06 0.0\n",
      "4.396166787046241e-06 0.0\n",
      "0.9999879598617554 1.0\n",
      "3.5209736779506784e-06 0.0\n",
      "4.093430106877349e-05 0.0\n",
      "0.9999758005142212 1.0\n",
      "5.505242825165624e-06 0.0\n",
      "0.9999231100082397 1.0\n",
      "1.5741452443762682e-05 0.0\n",
      "1.4993759577919263e-05 0.0\n",
      "0.9999948740005493 1.0\n",
      "0.9996922016143799 1.0\n",
      "4.280812390788924e-06 0.0\n",
      "0.9999861717224121 1.0\n",
      "3.6987562452850398e-06 0.0\n",
      "1.5081654055393301e-05 0.0\n",
      "0.997258186340332 1.0\n",
      "0.0010164178675040603 0.0\n",
      "4.6873901737853885e-06 0.0\n",
      "3.417035486563691e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999929666519165 1.0\n",
      "8.803958917269483e-05 0.0\n",
      "1.0592923899821471e-05 0.0\n",
      "0.9998786449432373 1.0\n",
      "7.172629011620302e-06 0.0\n",
      "0.999995231628418 1.0\n",
      "0.9999967813491821 1.0\n",
      "4.899190571450163e-06 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.9999957084655762 1.0\n",
      "0.9999926090240479 1.0\n",
      "6.857566040707752e-05 0.0\n",
      "0.9999659061431885 1.0\n",
      "0.9999943971633911 1.0\n",
      "0.9999896287918091 1.0\n",
      "0.09305109828710556 0.0\n",
      "0.9999843835830688 1.0\n",
      "0.9999856948852539 1.0\n",
      "3.563881591617246e-06 0.0\n",
      "6.774654320906848e-05 0.0\n",
      "4.50984516646713e-05 0.0\n",
      "8.695429642102681e-06 0.0\n",
      "0.00040777833783067763 0.0\n",
      "0.9999825954437256 1.0\n",
      "0.00013682521239388734 0.0\n",
      "8.052220800891519e-05 0.0\n",
      "0.00029285141499713063 0.0\n",
      "TRAIN[steps=13100] loss=0.001805 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13100] loss=0.030255 acc=0.699 P=0.690 R=0.698 F1=0.693890 \n",
      "EPOCH:  21\n",
      "model sim and label tuples:\n",
      "0.0026075926143676043 0.0\n",
      "0.13910700380802155 0.0\n",
      "0.00016229630273301154 0.0\n",
      "5.1756264838331845e-06 0.0\n",
      "3.695484565469087e-06 0.0\n",
      "0.9999951124191284 1.0\n",
      "0.00010767702769953758 0.0\n",
      "4.244097908667754e-06 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.9997991919517517 1.0\n",
      "0.9966232776641846 1.0\n",
      "0.9796122908592224 1.0\n",
      "3.6348656067275442e-06 0.0\n",
      "6.145849965832895e-06 0.0\n",
      "3.562494839570718e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.05838482081890106 0.0\n",
      "0.9999392032623291 1.0\n",
      "0.9999924898147583 1.0\n",
      "3.5019365896005183e-06 0.0\n",
      "0.9953393936157227 1.0\n",
      "8.515202353009954e-06 0.0\n",
      "0.9999924898147583 1.0\n",
      "4.448657819011714e-06 0.0\n",
      "0.9999877214431763 1.0\n",
      "0.9997460246086121 1.0\n",
      "0.0027338394429534674 0.0\n",
      "0.9999920129776001 1.0\n",
      "0.00033891122438944876 0.0\n",
      "0.9999449253082275 1.0\n",
      "1.3895622942072805e-05 0.0\n",
      "0.9998927116394043 1.0\n",
      "1.107870411942713e-05 0.0\n",
      "4.340225586929591e-06 0.0\n",
      "0.9998182654380798 1.0\n",
      "0.9996359348297119 1.0\n",
      "0.999981164932251 1.0\n",
      "1.2969670933671296e-05 0.0\n",
      "0.9701140522956848 1.0\n",
      "3.281502586105489e-06 0.0\n",
      "3.7155923564569093e-06 0.0\n",
      "0.999728262424469 1.0\n",
      "0.9999963045120239 1.0\n",
      "0.9998055100440979 1.0\n",
      "5.90608624406741e-06 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999401569366455 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9998074173927307 1.0\n",
      "0.9998807907104492 1.0\n",
      "0.9999939203262329 1.0\n",
      "4.119861841900274e-06 0.0\n",
      "0.0003029112995136529 0.0\n",
      "0.9924160242080688 1.0\n",
      "2.96052767225774e-05 0.0\n",
      "0.9999940395355225 1.0\n",
      "0.9999964237213135 1.0\n",
      "0.0003791535273194313 0.0\n",
      "0.9998505115509033 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999164342880249 1.0\n",
      "3.6851906770607457e-06 0.0\n",
      "4.575203638523817e-05 0.0\n",
      "0.9951500296592712 1.0\n",
      "TRAIN[steps=13200] loss=0.004541 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13200] loss=0.030335 acc=0.699 P=0.689 R=0.701 F1=0.694857 \n",
      "model sim and label tuples:\n",
      "0.9936989545822144 1.0\n",
      "0.9999688863754272 1.0\n",
      "5.946128112555016e-06 0.0\n",
      "6.160179964354029e-06 0.0\n",
      "0.03533756732940674 0.0\n",
      "0.0005073683569207788 0.0\n",
      "0.9997996687889099 1.0\n",
      "0.9999915361404419 1.0\n",
      "0.006553828250616789 0.0\n",
      "0.9998644590377808 1.0\n",
      "3.119635221082717e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "3.3949704629776534e-06 0.0\n",
      "0.9999256134033203 1.0\n",
      "0.00015203896327875555 0.0\n",
      "1.5020235878182575e-05 0.0\n",
      "3.686623676912859e-05 0.0\n",
      "3.94304242945509e-06 0.0\n",
      "0.007691627833992243 0.0\n",
      "0.9999938011169434 1.0\n",
      "3.7451657135534333e-06 0.0\n",
      "0.9999697208404541 1.0\n",
      "3.36114476340299e-06 0.0\n",
      "0.9997521042823792 1.0\n",
      "0.0004561091191135347 0.0\n",
      "0.9999861717224121 1.0\n",
      "5.488840088219149e-06 0.0\n",
      "6.4778837440826464e-06 0.0\n",
      "0.0035713270772248507 0.0\n",
      "9.144948307948653e-06 0.0\n",
      "5.9558265093073715e-06 0.0\n",
      "0.0065050432458519936 0.0\n",
      "0.9999691247940063 1.0\n",
      "5.801597944810055e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.0004360406310297549 0.0\n",
      "0.999971866607666 1.0\n",
      "0.999964714050293 1.0\n",
      "0.9999963045120239 1.0\n",
      "0.9999881982803345 1.0\n",
      "3.4000931918853894e-06 0.0\n",
      "0.9999700784683228 1.0\n",
      "0.9999494552612305 1.0\n",
      "0.9998088479042053 1.0\n",
      "6.810173545090947e-06 0.0\n",
      "3.3085707400459796e-06 0.0\n",
      "0.000977278221398592 0.0\n",
      "0.9999427795410156 1.0\n",
      "7.159118467825465e-06 0.0\n",
      "2.34894087043358e-05 0.0\n",
      "0.00011241789616178721 0.0\n",
      "0.9999963045120239 1.0\n",
      "0.9998563528060913 1.0\n",
      "0.999416708946228 1.0\n",
      "4.0933009586296976e-06 0.0\n",
      "4.010521479358431e-06 0.0\n",
      "0.9998273849487305 1.0\n",
      "0.0006619289051741362 0.0\n",
      "0.999933123588562 1.0\n",
      "0.011671759188175201 0.0\n",
      "9.310936547990423e-06 0.0\n",
      "3.748771996470168e-05 0.0\n",
      "1.6332691302523017e-05 0.0\n",
      "0.9999322891235352 1.0\n",
      "TRAIN[steps=13300] loss=0.001317 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13300] loss=0.030382 acc=0.699 P=0.690 R=0.698 F1=0.694048 \n",
      "model sim and label tuples:\n",
      "1.1897305739694275e-05 0.0\n",
      "0.9999809265136719 1.0\n",
      "3.342581749166129e-06 0.0\n",
      "7.010059562162496e-06 0.0\n",
      "4.429839464137331e-06 0.0\n",
      "5.6018689065240324e-05 0.0\n",
      "9.074335684999824e-05 0.0\n",
      "5.599156065727584e-05 0.0\n",
      "1.3850864888809156e-05 0.0\n",
      "0.00016060093184933066 0.0\n",
      "1.5291434465325437e-05 0.0\n",
      "0.00016059189510997385 0.0\n",
      "0.9990139007568359 1.0\n",
      "0.006040068343281746 0.0\n",
      "6.540112735820003e-06 0.0\n",
      "0.9999954700469971 1.0\n",
      "0.0003163684450555593 0.0\n",
      "0.9999717473983765 1.0\n",
      "0.9999115467071533 1.0\n",
      "0.9976872205734253 1.0\n",
      "4.4590920879272744e-05 0.0\n",
      "0.019995031878352165 0.0\n",
      "0.9995192289352417 1.0\n",
      "0.9999946355819702 1.0\n",
      "0.999427318572998 1.0\n",
      "0.9998212456703186 1.0\n",
      "0.9999189376831055 1.0\n",
      "5.1522933972592e-06 0.0\n",
      "0.9999929666519165 1.0\n",
      "0.998883068561554 1.0\n",
      "0.9999940395355225 1.0\n",
      "0.9998179078102112 1.0\n",
      "0.9999948740005493 1.0\n",
      "3.5047828532697167e-06 0.0\n",
      "0.0005279044853523374 0.0\n",
      "6.389527698047459e-05 0.0\n",
      "0.9999785423278809 1.0\n",
      "0.9988766312599182 1.0\n",
      "0.9999685287475586 1.0\n",
      "0.999993085861206 1.0\n",
      "0.9999810457229614 1.0\n",
      "6.588440555788111e-06 0.0\n",
      "1.3762869457423221e-05 0.0\n",
      "0.0010713653173297644 0.0\n",
      "0.9999887943267822 1.0\n",
      "0.9991187453269958 1.0\n",
      "0.9992629885673523 1.0\n",
      "0.9993335604667664 1.0\n",
      "1.262923251488246e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9932674169540405 1.0\n",
      "0.0019236644729971886 0.0\n",
      "0.9996212720870972 1.0\n",
      "4.336897109169513e-05 0.0\n",
      "0.0002418248332105577 0.0\n",
      "3.5470127386361128e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "6.096447577874642e-06 0.0\n",
      "3.7930611142655835e-05 0.0\n",
      "1.8185455701313913e-05 0.0\n",
      "0.9995273351669312 1.0\n",
      "0.9998075366020203 1.0\n",
      "0.9999812841415405 1.0\n",
      "8.707980305189267e-05 0.0\n",
      "TRAIN[steps=13400] loss=0.000761 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13400] loss=0.030466 acc=0.699 P=0.688 R=0.702 F1=0.694814 \n",
      "model sim and label tuples:\n",
      "1.0869523975998163e-05 0.0\n",
      "0.993935763835907 1.0\n",
      "0.6837921142578125 1.0\n",
      "0.9999871253967285 1.0\n",
      "0.9999966621398926 1.0\n",
      "3.5746888897847384e-06 0.0\n",
      "0.9999829530715942 1.0\n",
      "0.9999264478683472 1.0\n",
      "0.9966208934783936 1.0\n",
      "0.0019084985833615065 0.0\n",
      "5.053294444223866e-06 0.0\n",
      "0.9999922513961792 1.0\n",
      "0.9985697269439697 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.9998947381973267 1.0\n",
      "0.0006416162941604853 0.0\n",
      "0.9998890161514282 1.0\n",
      "0.0017846785485744476 0.0\n",
      "1.0992259376507718e-05 0.0\n",
      "0.9989497065544128 1.0\n",
      "0.9999967813491821 1.0\n",
      "4.493171945796348e-05 0.0\n",
      "0.999987006187439 1.0\n",
      "0.9999850988388062 1.0\n",
      "0.9999561309814453 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999650716781616 1.0\n",
      "0.999809205532074 1.0\n",
      "0.9999924898147583 1.0\n",
      "0.9999781847000122 1.0\n",
      "6.439989647333277e-06 0.0\n",
      "0.015483659692108631 0.0\n",
      "0.9999879598617554 1.0\n",
      "0.9905486702919006 1.0\n",
      "3.751826170628192e-06 0.0\n",
      "4.553129201667616e-06 0.0\n",
      "0.0004050552670378238 0.0\n",
      "0.00048486454761587083 0.0\n",
      "0.9998948574066162 1.0\n",
      "0.9999961853027344 1.0\n",
      "6.107393346610479e-06 0.0\n",
      "4.024871032015653e-06 0.0\n",
      "7.592923338961555e-06 0.0\n",
      "0.999993085861206 1.0\n",
      "0.0002214620035374537 0.0\n",
      "0.9999963045120239 1.0\n",
      "4.60454321000725e-05 0.0\n",
      "0.9999889135360718 1.0\n",
      "3.8488128666358534e-06 0.0\n",
      "0.008190939202904701 0.0\n",
      "0.9981282353401184 1.0\n",
      "3.4671764296945184e-05 0.0\n",
      "0.9999949932098389 1.0\n",
      "0.0011770720593631268 0.0\n",
      "3.1768011012900388e-06 0.0\n",
      "9.701896487968042e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "7.816232027835213e-06 0.0\n",
      "3.3948504096770193e-06 0.0\n",
      "0.9999790191650391 1.0\n",
      "4.576563696900848e-06 0.0\n",
      "2.446069265715778e-05 0.0\n",
      "4.903425633528968e-06 0.0\n",
      "0.9999969005584717 1.0\n",
      "TRAIN[steps=13500] loss=0.006796 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13500] loss=0.030477 acc=0.699 P=0.689 R=0.700 F1=0.694264 \n",
      "model sim and label tuples:\n",
      "0.9999675750732422 1.0\n",
      "1.1174873179697897e-05 0.0\n",
      "0.999991774559021 1.0\n",
      "3.240056912545697e-06 0.0\n",
      "0.9999682903289795 1.0\n",
      "0.0012241994263604283 0.0\n",
      "3.9955070860742126e-06 0.0\n",
      "4.1145412978949025e-06 0.0\n",
      "0.9997598528862 1.0\n",
      "0.9999662637710571 1.0\n",
      "3.4130132462451e-06 0.0\n",
      "0.9977664947509766 1.0\n",
      "0.00029502547113224864 0.0\n",
      "0.00179485441185534 0.0\n",
      "0.9990991353988647 1.0\n",
      "1.557963332743384e-05 0.0\n",
      "0.0002455251233186573 0.0\n",
      "3.884426405420527e-06 0.0\n",
      "3.942049715988105e-06 0.0\n",
      "1.946366137417499e-05 0.0\n",
      "0.9999939203262329 1.0\n",
      "2.993202542711515e-05 0.0\n",
      "0.9999926090240479 1.0\n",
      "0.0020813767332583666 0.0\n",
      "0.9979817867279053 1.0\n",
      "4.670776888815453e-06 0.0\n",
      "2.3084881831891835e-05 0.0\n",
      "0.7966291904449463 0.0\n",
      "2.2429640011978336e-05 0.0\n",
      "0.9998559951782227 1.0\n",
      "4.661316779674962e-06 0.0\n",
      "2.437220791762229e-05 0.0\n",
      "0.011439074762165546 0.0\n",
      "0.00020178189151920378 0.0\n",
      "3.646291816039593e-06 0.0\n",
      "0.0006888735806569457 0.0\n",
      "0.9999308586120605 1.0\n",
      "0.999996542930603 1.0\n",
      "3.519957681419328e-05 0.0\n",
      "0.0020030979067087173 0.0\n",
      "0.9910320043563843 1.0\n",
      "0.9999892711639404 1.0\n",
      "0.9999834299087524 1.0\n",
      "1.4522148376272526e-05 0.0\n",
      "0.9999672174453735 1.0\n",
      "0.9997063279151917 1.0\n",
      "8.293197424791288e-06 0.0\n",
      "0.9999967813491821 1.0\n",
      "0.0004940510261803865 0.0\n",
      "0.9995806813240051 1.0\n",
      "8.158122363965958e-05 0.0\n",
      "0.9979123473167419 1.0\n",
      "0.999909520149231 1.0\n",
      "0.9999033212661743 1.0\n",
      "3.230476977478247e-06 0.0\n",
      "0.9991694688796997 1.0\n",
      "1.0721776561695151e-05 0.0\n",
      "0.9999881982803345 1.0\n",
      "0.0005356085021048784 0.0\n",
      "0.9999139308929443 1.0\n",
      "0.015014050528407097 0.0\n",
      "5.818669160362333e-05 0.0\n",
      "0.9998852014541626 1.0\n",
      "0.9999886751174927 1.0\n",
      "TRAIN[steps=13600] loss=0.025753 acc=0.984 P=0.966 R=1.000 F1=0.982456\n",
      "DEV[steps=13600] loss=0.030498 acc=0.699 P=0.691 R=0.696 F1=0.693540 \n",
      "model sim and label tuples:\n",
      "0.9999840259552002 1.0\n",
      "0.9999557733535767 1.0\n",
      "0.9648351669311523 1.0\n",
      "3.420357643335592e-06 0.0\n",
      "0.9998676776885986 1.0\n",
      "0.999697208404541 1.0\n",
      "0.9999943971633911 1.0\n",
      "4.621673269866733e-06 0.0\n",
      "2.124260390701238e-05 0.0\n",
      "3.4860775485867634e-05 0.0\n",
      "0.003436558647081256 0.0\n",
      "4.0758777686278336e-06 0.0\n",
      "0.9999818801879883 1.0\n",
      "3.318034487165278e-06 0.0\n",
      "1.0008995559473988e-05 0.0\n",
      "1.2861405593866948e-05 0.0\n",
      "0.9998277425765991 1.0\n",
      "3.9958958950592205e-06 0.0\n",
      "0.0006098632584325969 0.0\n",
      "0.9999357461929321 1.0\n",
      "0.0017361894715577364 0.0\n",
      "0.9996951818466187 1.0\n",
      "3.778370955842547e-05 0.0\n",
      "0.999935507774353 1.0\n",
      "0.9999713897705078 1.0\n",
      "0.0005734480218961835 0.0\n",
      "0.9998133778572083 1.0\n",
      "0.00011985397577518597 0.0\n",
      "8.433794573647901e-05 0.0\n",
      "0.01921982318162918 0.0\n",
      "0.00012361575500108302 0.0\n",
      "3.2268158065562602e-06 0.0\n",
      "1.5601191989844665e-05 0.0\n",
      "4.404942228575237e-05 0.0\n",
      "0.9999740123748779 1.0\n",
      "3.542475951689994e-06 0.0\n",
      "0.0025764082092791796 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.9999841451644897 1.0\n",
      "2.9161619750084355e-05 0.0\n",
      "3.6094384086027276e-06 0.0\n",
      "0.9999901056289673 1.0\n",
      "0.9999948740005493 1.0\n",
      "0.9999876022338867 1.0\n",
      "0.9999814033508301 1.0\n",
      "0.9107264280319214 1.0\n",
      "0.009910020045936108 0.0\n",
      "0.999931812286377 1.0\n",
      "0.0009113603155128658 0.0\n",
      "0.9999876022338867 1.0\n",
      "0.9999768733978271 1.0\n",
      "0.0011495787184685469 0.0\n",
      "6.700910489598755e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9991410970687866 1.0\n",
      "0.00014353088045027107 0.0\n",
      "0.9999955892562866 1.0\n",
      "1.437763603462372e-05 0.0\n",
      "0.9999896287918091 1.0\n",
      "3.1247052447724855e-06 0.0\n",
      "0.9965739250183105 1.0\n",
      "0.9999924898147583 1.0\n",
      "0.9999868869781494 1.0\n",
      "0.9999703168869019 1.0\n",
      "TRAIN[steps=13700] loss=0.002755 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13700] loss=0.030565 acc=0.699 P=0.690 R=0.696 F1=0.693140 \n",
      "EPOCH:  22\n",
      "model sim and label tuples:\n",
      "0.9999964237213135 1.0\n",
      "0.9907976388931274 1.0\n",
      "5.027889528719243e-06 0.0\n",
      "1.7788202967494726e-05 0.0\n",
      "1.1733036444638856e-05 0.0\n",
      "0.9999823570251465 1.0\n",
      "3.7463672924786806e-05 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.0013199036475270987 0.0\n",
      "0.0001910494756884873 0.0\n",
      "2.5821072995313443e-05 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.001835571601986885 0.0\n",
      "0.9999953508377075 1.0\n",
      "0.0009127746452577412 0.0\n",
      "3.565356792023522e-06 0.0\n",
      "0.9967429041862488 1.0\n",
      "0.9997513890266418 1.0\n",
      "0.004707579500973225 0.0\n",
      "0.9998809099197388 1.0\n",
      "1.2199386219435837e-05 0.0\n",
      "0.984286904335022 1.0\n",
      "4.288924174034037e-05 0.0\n",
      "0.9999617338180542 1.0\n",
      "0.9999904632568359 1.0\n",
      "0.9999886751174927 1.0\n",
      "0.00014107923198025674 0.0\n",
      "7.8379045589827e-05 0.0\n",
      "0.9999555349349976 1.0\n",
      "0.008054227568209171 0.0\n",
      "4.2146421037614346e-06 0.0\n",
      "3.9112796912377235e-06 0.0\n",
      "0.014567471109330654 0.0\n",
      "0.9999691247940063 1.0\n",
      "0.9999970197677612 1.0\n",
      "9.690328442957252e-05 0.0\n",
      "0.9999698400497437 1.0\n",
      "0.9999938011169434 1.0\n",
      "0.9828270673751831 1.0\n",
      "0.0003480010782368481 0.0\n",
      "0.028031334280967712 0.0\n",
      "5.366362074710196e-06 0.0\n",
      "0.00012141620391048491 0.0\n",
      "3.283894557171152e-06 0.0\n",
      "0.00041689706267789006 0.0\n",
      "3.1900315207167296e-06 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999637603759766 1.0\n",
      "5.717087788070785e-06 0.0\n",
      "5.438174412120134e-06 0.0\n",
      "3.3713115499267587e-06 0.0\n",
      "0.9999969005584717 1.0\n",
      "1.647995122766588e-05 0.0\n",
      "3.7336967579904012e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "8.260094546130858e-06 0.0\n",
      "2.629788468766492e-05 0.0\n",
      "0.9938473105430603 1.0\n",
      "0.9999933242797852 1.0\n",
      "5.7662587096274365e-06 0.0\n",
      "0.0013277687830850482 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.0005221628816798329 0.0\n",
      "0.0016384482150897384 0.0\n",
      "TRAIN[steps=13800] loss=0.001837 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13800] loss=0.030603 acc=0.699 P=0.691 R=0.695 F1=0.693085 \n",
      "model sim and label tuples:\n",
      "7.416887001454597e-06 0.0\n",
      "0.9964313507080078 1.0\n",
      "0.9999198913574219 1.0\n",
      "3.2531820579606574e-06 0.0\n",
      "3.758913635465433e-06 0.0\n",
      "0.9999477863311768 1.0\n",
      "0.9999454021453857 1.0\n",
      "0.9999943971633911 1.0\n",
      "8.584841270931065e-05 0.0\n",
      "0.9999852180480957 1.0\n",
      "0.9934996366500854 1.0\n",
      "0.00013890130503568798 0.0\n",
      "0.9999669790267944 1.0\n",
      "3.2255206861009356e-06 0.0\n",
      "3.273800530223525e-06 0.0\n",
      "0.0004975140909664333 0.0\n",
      "0.9999611377716064 1.0\n",
      "1.767067442415282e-05 0.0\n",
      "0.9998878240585327 1.0\n",
      "0.0017316408921033144 0.0\n",
      "0.99998939037323 1.0\n",
      "4.487458681978751e-06 0.0\n",
      "3.646699042292312e-05 0.0\n",
      "0.9967594742774963 1.0\n",
      "0.9999858140945435 1.0\n",
      "8.453741429548245e-06 0.0\n",
      "0.9998487234115601 1.0\n",
      "0.9999969005584717 1.0\n",
      "1.0491003195056692e-05 0.0\n",
      "4.633236130757723e-06 0.0\n",
      "0.9998321533203125 1.0\n",
      "6.494146418845048e-06 0.0\n",
      "0.9999769926071167 1.0\n",
      "0.9972495436668396 1.0\n",
      "3.1055526505952002e-06 0.0\n",
      "0.9999139308929443 1.0\n",
      "6.090950409998186e-05 0.0\n",
      "0.00015577426529489458 0.0\n",
      "0.9999643564224243 1.0\n",
      "0.9999524354934692 1.0\n",
      "0.0015930937370285392 0.0\n",
      "0.0001612767664482817 0.0\n",
      "0.9999797344207764 1.0\n",
      "0.9999574422836304 1.0\n",
      "0.9920393228530884 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.9940938353538513 1.0\n",
      "1.731241718516685e-05 0.0\n",
      "0.9999405145645142 1.0\n",
      "0.999990701675415 1.0\n",
      "3.1843537726672366e-06 0.0\n",
      "0.010075418278574944 0.0\n",
      "0.00013351469533517957 0.0\n",
      "0.9999077320098877 1.0\n",
      "0.9999687671661377 1.0\n",
      "5.590464570559561e-05 0.0\n",
      "0.9998024106025696 1.0\n",
      "0.9999512434005737 1.0\n",
      "0.9999828338623047 1.0\n",
      "3.426273906370625e-05 0.0\n",
      "0.0012064534239470959 0.0\n",
      "0.9999797344207764 1.0\n",
      "0.9999967813491821 1.0\n",
      "0.004669127985835075 0.0\n",
      "TRAIN[steps=13900] loss=0.000817 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=13900] loss=0.030708 acc=0.699 P=0.690 R=0.696 F1=0.693195 \n",
      "model sim and label tuples:\n",
      "3.020613121407223e-06 0.0\n",
      "0.0007734696846455336 0.0\n",
      "0.9998728036880493 1.0\n",
      "2.9159618861740455e-05 0.0\n",
      "0.9999715089797974 1.0\n",
      "0.9999938011169434 1.0\n",
      "0.997722327709198 1.0\n",
      "0.999987006187439 1.0\n",
      "0.9998922348022461 1.0\n",
      "0.0001290750369662419 0.0\n",
      "0.9999521970748901 1.0\n",
      "3.929667400370818e-06 0.0\n",
      "5.702177440980449e-05 0.0\n",
      "0.9995747208595276 1.0\n",
      "2.9918121526861796e-06 0.0\n",
      "0.9999538660049438 1.0\n",
      "1.7427508282708004e-05 0.0\n",
      "1.4510173059534281e-05 0.0\n",
      "0.9999837875366211 1.0\n",
      "0.9995606541633606 1.0\n",
      "3.365853444847744e-06 0.0\n",
      "0.007161317393183708 0.0\n",
      "0.9999295473098755 1.0\n",
      "0.9998914003372192 1.0\n",
      "3.127018544546445e-06 0.0\n",
      "3.712623993123998e-06 0.0\n",
      "3.899067905877018e-06 0.0\n",
      "0.00048126536421477795 0.0\n",
      "0.11652357876300812 0.0\n",
      "4.3620057113002986e-06 0.0\n",
      "8.488884304824751e-06 0.0\n",
      "0.9999170303344727 1.0\n",
      "0.9999954700469971 1.0\n",
      "8.267684279417153e-06 0.0\n",
      "0.9998573064804077 1.0\n",
      "0.01812046952545643 0.0\n",
      "0.9999662637710571 1.0\n",
      "9.637019684305415e-05 0.0\n",
      "5.050057097832905e-06 0.0\n",
      "0.00022165845439303666 0.0\n",
      "3.50012160197366e-05 0.0\n",
      "0.9942448735237122 1.0\n",
      "0.9999860525131226 1.0\n",
      "0.998434841632843 1.0\n",
      "0.9999909400939941 1.0\n",
      "0.9837643504142761 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999290704727173 1.0\n",
      "3.136354644084349e-05 0.0\n",
      "0.9987836480140686 1.0\n",
      "0.002096466487273574 0.0\n",
      "0.9974848031997681 1.0\n",
      "3.6750334402313456e-06 0.0\n",
      "3.496696763249929e-06 0.0\n",
      "0.999992847442627 1.0\n",
      "0.7873740196228027 1.0\n",
      "0.00012025132309645414 0.0\n",
      "0.9997461438179016 1.0\n",
      "4.954000814905157e-06 0.0\n",
      "0.999988317489624 1.0\n",
      "0.9998061060905457 1.0\n",
      "1.8541826648288406e-05 0.0\n",
      "3.944712716474896e-06 0.0\n",
      "0.999457061290741 1.0\n",
      "TRAIN[steps=14000] loss=0.006643 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14000] loss=0.030755 acc=0.699 P=0.690 R=0.697 F1=0.693532 \n",
      "model sim and label tuples:\n",
      "3.6046808418177534e-06 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.00010616764484439045 0.0\n",
      "0.9976994395256042 1.0\n",
      "1.210741902468726e-05 0.0\n",
      "0.9999393224716187 1.0\n",
      "6.237733759917319e-05 0.0\n",
      "0.00011003793042618781 0.0\n",
      "0.9979307651519775 1.0\n",
      "0.999724805355072 1.0\n",
      "5.636193236568943e-06 0.0\n",
      "3.645770220828126e-06 0.0\n",
      "1.1240605999773834e-05 0.0\n",
      "0.9999316930770874 1.0\n",
      "3.926277258869959e-06 0.0\n",
      "0.9999961853027344 1.0\n",
      "6.983808816585224e-06 0.0\n",
      "0.999988317489624 1.0\n",
      "8.987609362520743e-06 0.0\n",
      "0.9996009469032288 1.0\n",
      "0.9995710253715515 1.0\n",
      "0.0014908639714121819 0.0\n",
      "0.9999879598617554 1.0\n",
      "0.00011656456626951694 0.0\n",
      "0.9999687671661377 1.0\n",
      "0.9999958276748657 1.0\n",
      "0.00014843582175672054 0.0\n",
      "1.1094500223407522e-05 0.0\n",
      "0.9999971389770508 1.0\n",
      "3.4514205253799446e-06 0.0\n",
      "3.3212984362762654e-06 0.0\n",
      "0.9999969005584717 1.0\n",
      "3.383986722838017e-06 0.0\n",
      "0.9999959468841553 1.0\n",
      "0.999987006187439 1.0\n",
      "0.9998440742492676 1.0\n",
      "0.0011963716242462397 0.0\n",
      "5.928286100242985e-06 0.0\n",
      "0.0038198158144950867 0.0\n",
      "0.0002313614822924137 0.0\n",
      "4.356852514320053e-05 0.0\n",
      "4.0735194488661364e-05 0.0\n",
      "0.9999791383743286 1.0\n",
      "0.0019394728587940335 0.0\n",
      "0.16980236768722534 0.0\n",
      "0.9976021647453308 1.0\n",
      "0.0025659974198788404 0.0\n",
      "0.9999771118164062 1.0\n",
      "0.9998059868812561 1.0\n",
      "0.999942421913147 1.0\n",
      "0.9999918937683105 1.0\n",
      "0.9989809393882751 1.0\n",
      "0.9999626874923706 1.0\n",
      "3.1771221529197646e-06 0.0\n",
      "0.9999790191650391 1.0\n",
      "0.9999971389770508 1.0\n",
      "3.3958283438551007e-06 0.0\n",
      "0.9999960660934448 1.0\n",
      "0.013630018569529057 0.0\n",
      "0.998239278793335 1.0\n",
      "4.3333540816092864e-05 0.0\n",
      "6.89880907884799e-05 0.0\n",
      "0.9999971389770508 1.0\n",
      "3.222061877750093e-06 0.0\n",
      "TRAIN[steps=14100] loss=0.003489 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14100] loss=0.030832 acc=0.700 P=0.691 R=0.698 F1=0.694410 \n",
      "model sim and label tuples:\n",
      "0.9932377338409424 1.0\n",
      "0.00196466245688498 0.0\n",
      "0.0015504381153732538 0.0\n",
      "0.9999951124191284 1.0\n",
      "0.9999872446060181 1.0\n",
      "4.034805442643119e-06 0.0\n",
      "0.9996527433395386 1.0\n",
      "0.9999971389770508 1.0\n",
      "1.8964372429763898e-05 0.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999970197677612 1.0\n",
      "4.703739250544459e-06 0.0\n",
      "3.0479061479127267e-06 0.0\n",
      "0.999990701675415 1.0\n",
      "1.0826786819961853e-05 0.0\n",
      "0.0002014646743191406 0.0\n",
      "0.9988999366760254 1.0\n",
      "1.0293753803125583e-05 0.0\n",
      "0.9989902377128601 1.0\n",
      "3.3893995805556187e-06 0.0\n",
      "0.9968794584274292 1.0\n",
      "0.9651968479156494 1.0\n",
      "8.263722702395171e-05 0.0\n",
      "2.1383171770139597e-05 0.0\n",
      "4.946424269292038e-06 0.0\n",
      "0.0023596754763275385 0.0\n",
      "8.569472265662625e-05 0.0\n",
      "0.9992108345031738 1.0\n",
      "0.9999959468841553 1.0\n",
      "0.000431594904512167 0.0\n",
      "6.546147233166266e-06 0.0\n",
      "7.0759297159384005e-06 0.0\n",
      "2.0505107386270538e-05 0.0\n",
      "8.71920638019219e-05 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999372959136963 1.0\n",
      "4.035761594423093e-05 0.0\n",
      "0.9999970197677612 1.0\n",
      "3.4781985505105695e-06 0.0\n",
      "1.254818471352337e-05 0.0\n",
      "3.368633406353183e-05 0.0\n",
      "1.7074047718779184e-05 0.0\n",
      "0.9985530972480774 1.0\n",
      "2.0307430531829596e-05 0.0\n",
      "0.9980683922767639 1.0\n",
      "3.846842446364462e-06 0.0\n",
      "0.9943280816078186 1.0\n",
      "2.9234352041385137e-06 0.0\n",
      "0.9998366832733154 1.0\n",
      "0.9999892711639404 1.0\n",
      "0.999996542930603 1.0\n",
      "0.9960963129997253 1.0\n",
      "0.9940503239631653 1.0\n",
      "1.0869492143683601e-05 0.0\n",
      "0.004656206350773573 0.0\n",
      "2.221599243057426e-05 0.0\n",
      "0.9999513626098633 1.0\n",
      "5.0310745791648515e-06 0.0\n",
      "0.9999868869781494 1.0\n",
      "0.00010873762948904186 0.0\n",
      "3.6256499242881546e-06 0.0\n",
      "0.9999809265136719 1.0\n",
      "0.9996703863143921 1.0\n",
      "0.0011298066237941384 0.0\n",
      "TRAIN[steps=14200] loss=0.001269 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14200] loss=0.030877 acc=0.699 P=0.691 R=0.696 F1=0.693352 \n",
      "model sim and label tuples:\n",
      "0.9999960660934448 1.0\n",
      "0.9938209652900696 1.0\n",
      "5.387355031416519e-06 0.0\n",
      "3.381021997483913e-06 0.0\n",
      "0.001007838873192668 0.0\n",
      "0.9391050934791565 1.0\n",
      "2.9534159693866968e-05 0.0\n",
      "0.9998555183410645 1.0\n",
      "0.9999779462814331 1.0\n",
      "0.9999967813491821 1.0\n",
      "0.9995800852775574 1.0\n",
      "0.9972621202468872 1.0\n",
      "0.014764375984668732 0.0\n",
      "0.9999321699142456 1.0\n",
      "0.9999947547912598 1.0\n",
      "1.499983682151651e-05 0.0\n",
      "0.9998291730880737 1.0\n",
      "0.9998681545257568 1.0\n",
      "0.9999732971191406 1.0\n",
      "0.995816171169281 1.0\n",
      "0.9999655485153198 1.0\n",
      "0.9986132383346558 1.0\n",
      "0.9999955892562866 1.0\n",
      "5.7729168474907055e-05 0.0\n",
      "3.1804354421183234e-06 0.0\n",
      "5.518947091331938e-06 0.0\n",
      "5.83380096941255e-06 0.0\n",
      "5.452060122479452e-06 0.0\n",
      "0.9994558691978455 1.0\n",
      "2.8866245429526316e-06 0.0\n",
      "0.9999958276748657 1.0\n",
      "4.210921815683832e-06 0.0\n",
      "0.999994158744812 1.0\n",
      "0.9999948740005493 1.0\n",
      "9.232173397322185e-06 0.0\n",
      "8.627812349004671e-06 0.0\n",
      "0.011640939861536026 0.0\n",
      "0.9999955892562866 1.0\n",
      "0.999866247177124 1.0\n",
      "0.9999853372573853 1.0\n",
      "3.3554467790963827e-06 0.0\n",
      "0.9949055910110474 1.0\n",
      "4.5221062464406714e-05 0.0\n",
      "0.009410309605300426 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.011201419867575169 0.0\n",
      "3.3407909540983383e-06 0.0\n",
      "0.9999475479125977 1.0\n",
      "1.918816997203976e-05 0.0\n",
      "0.0008214664994738996 0.0\n",
      "3.4107983083231375e-05 0.0\n",
      "6.823292551416671e-06 0.0\n",
      "2.823340219038073e-05 0.0\n",
      "0.9999934434890747 1.0\n",
      "3.314473360660486e-06 0.0\n",
      "0.00020067000878043473 0.0\n",
      "0.00027537127607502043 0.0\n",
      "0.9999934434890747 1.0\n",
      "0.9972301125526428 1.0\n",
      "0.9999955892562866 1.0\n",
      "3.735606014743098e-06 0.0\n",
      "0.0009736017673276365 0.0\n",
      "1.1936936061829329e-05 0.0\n",
      "0.0003140853368677199 0.0\n",
      "TRAIN[steps=14300] loss=0.002160 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14300] loss=0.030990 acc=0.700 P=0.690 R=0.701 F1=0.695458 \n",
      "EPOCH:  23\n",
      "model sim and label tuples:\n",
      "0.9999971389770508 1.0\n",
      "9.840371058089659e-05 0.0\n",
      "3.0971762043918716e-06 0.0\n",
      "0.9999629259109497 1.0\n",
      "0.9992802739143372 1.0\n",
      "2.053164098470006e-05 0.0\n",
      "0.9984656572341919 1.0\n",
      "0.3896101713180542 0.0\n",
      "3.5459472655929858e-06 0.0\n",
      "0.9999947547912598 1.0\n",
      "3.547503183654044e-06 0.0\n",
      "1.9951059584855102e-05 0.0\n",
      "6.6873694777314086e-06 0.0\n",
      "3.816279786406085e-06 0.0\n",
      "0.0015983134508132935 0.0\n",
      "9.236506230081432e-06 0.0\n",
      "0.0006151182460598648 0.0\n",
      "0.9999920129776001 1.0\n",
      "3.182137788826367e-06 0.0\n",
      "3.782073690672405e-06 0.0\n",
      "0.00011983419972239062 0.0\n",
      "0.009992226958274841 0.0\n",
      "0.00019530358258634806 0.0\n",
      "0.9995717406272888 1.0\n",
      "0.999714195728302 1.0\n",
      "0.0004318348364904523 0.0\n",
      "1.894640263344627e-05 0.0\n",
      "9.460422006668523e-06 0.0\n",
      "2.876051439670846e-06 0.0\n",
      "9.730822057463229e-05 0.0\n",
      "4.731615263153799e-05 0.0\n",
      "0.995108425617218 1.0\n",
      "1.048214289767202e-05 0.0\n",
      "0.999862551689148 1.0\n",
      "2.955005811600131e-06 0.0\n",
      "0.9999910593032837 1.0\n",
      "0.9999971389770508 1.0\n",
      "0.9999967813491821 1.0\n",
      "0.9999850988388062 1.0\n",
      "5.139533641340677e-06 0.0\n",
      "0.9752876162528992 1.0\n",
      "6.406037573469803e-05 0.0\n",
      "0.999779999256134 1.0\n",
      "1.6217822121689096e-05 0.0\n",
      "3.1498013868258568e-06 0.0\n",
      "0.9999794960021973 1.0\n",
      "5.438324933493277e-06 0.0\n",
      "0.9999233484268188 1.0\n",
      "6.015787676005857e-06 0.0\n",
      "3.5512637168721994e-06 0.0\n",
      "2.3200462237582542e-05 0.0\n",
      "3.2333831768482924e-05 0.0\n",
      "0.9993725419044495 1.0\n",
      "0.9984488487243652 1.0\n",
      "2.9579746296803933e-06 0.0\n",
      "7.935825851745903e-05 0.0\n",
      "0.999468982219696 1.0\n",
      "8.594002792960964e-06 0.0\n",
      "1.1439961781434249e-05 0.0\n",
      "4.4428452383726835e-06 0.0\n",
      "0.9998095631599426 1.0\n",
      "4.815564807358896e-06 0.0\n",
      "2.2993021048023365e-05 0.0\n",
      "3.312425633339444e-06 0.0\n",
      "TRAIN[steps=14400] loss=0.008495 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14400] loss=0.031071 acc=0.700 P=0.690 R=0.700 F1=0.694953 \n",
      "model sim and label tuples:\n",
      "0.9999886751174927 1.0\n",
      "0.8973874449729919 1.0\n",
      "4.1591929402784444e-06 0.0\n",
      "0.9999793767929077 1.0\n",
      "0.9963845014572144 1.0\n",
      "8.639585757919122e-06 0.0\n",
      "3.2020220714912284e-06 0.0\n",
      "2.9377224564086646e-06 0.0\n",
      "0.0007856094744056463 0.0\n",
      "3.519134452290018e-06 0.0\n",
      "6.679126090602949e-05 0.0\n",
      "1.5363775673904456e-05 0.0\n",
      "3.3533833629917353e-06 0.0\n",
      "0.9999746084213257 1.0\n",
      "0.9999973773956299 1.0\n",
      "0.9999949932098389 1.0\n",
      "4.936912955599837e-05 0.0\n",
      "3.261276788180112e-06 0.0\n",
      "0.999879002571106 1.0\n",
      "3.0170431273290887e-06 0.0\n",
      "2.0852176021435298e-05 0.0\n",
      "0.9999685287475586 1.0\n",
      "3.965200448874384e-06 0.0\n",
      "3.6054063912160927e-06 0.0\n",
      "0.9932422041893005 1.0\n",
      "0.9999570846557617 1.0\n",
      "0.013285333290696144 0.0\n",
      "2.3254087864188477e-05 0.0\n",
      "1.6994739780784585e-05 0.0\n",
      "0.0023644068278372288 0.0\n",
      "0.9999958276748657 1.0\n",
      "0.9999827146530151 1.0\n",
      "0.999946117401123 1.0\n",
      "0.9999825954437256 1.0\n",
      "3.1963932087819558e-06 0.0\n",
      "7.2128191277442966e-06 0.0\n",
      "3.396920192244579e-06 0.0\n",
      "0.0016074329614639282 0.0\n",
      "0.9999703168869019 1.0\n",
      "7.656169145775493e-06 0.0\n",
      "6.50895208309521e-06 0.0\n",
      "0.003985910676419735 0.0\n",
      "6.122900686023058e-06 0.0\n",
      "0.00012691564916167408 0.0\n",
      "0.999445378780365 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999029636383057 1.0\n",
      "0.002184191718697548 0.0\n",
      "0.6282656788825989 1.0\n",
      "5.4194219956116285e-06 0.0\n",
      "0.9987500905990601 1.0\n",
      "8.255030479631387e-06 0.0\n",
      "0.9999790191650391 1.0\n",
      "4.367996098153526e-06 0.0\n",
      "4.199616341793444e-06 0.0\n",
      "3.29551789945981e-06 0.0\n",
      "0.999988317489624 1.0\n",
      "3.1162810046225786e-05 0.0\n",
      "2.9311206617421703e-06 0.0\n",
      "2.7008885808754712e-05 0.0\n",
      "0.9999376535415649 1.0\n",
      "0.007846031337976456 0.0\n",
      "0.00020570041669998318 0.0\n",
      "3.2257787552225636e-06 0.0\n",
      "TRAIN[steps=14500] loss=0.009668 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14500] loss=0.031167 acc=0.700 P=0.689 R=0.706 F1=0.697105 \n",
      "model sim and label tuples:\n",
      "3.137027533739456e-06 0.0\n",
      "0.9999967813491821 1.0\n",
      "0.005977340508252382 0.0\n",
      "0.9994896650314331 1.0\n",
      "0.9996942281723022 1.0\n",
      "0.0003860362048726529 0.0\n",
      "0.000729941762983799 0.0\n",
      "0.9999732971191406 1.0\n",
      "0.0004536766791716218 0.0\n",
      "1.4042559996596538e-05 0.0\n",
      "2.0368583136587404e-05 0.0\n",
      "0.0021239432971924543 0.0\n",
      "0.9999574422836304 1.0\n",
      "0.0001312471431447193 0.0\n",
      "0.9378587007522583 1.0\n",
      "7.917964467196725e-06 0.0\n",
      "0.999994158744812 1.0\n",
      "0.9999945163726807 1.0\n",
      "0.9998610019683838 1.0\n",
      "4.100509249838069e-05 0.0\n",
      "0.999984860420227 1.0\n",
      "0.9999656677246094 1.0\n",
      "0.9999971389770508 1.0\n",
      "0.999995231628418 1.0\n",
      "0.999935507774353 1.0\n",
      "0.9990780353546143 1.0\n",
      "0.9999853372573853 1.0\n",
      "0.00467501487582922 0.0\n",
      "3.363424411872984e-06 0.0\n",
      "0.9999822378158569 1.0\n",
      "0.9932068586349487 1.0\n",
      "0.9999264478683472 1.0\n",
      "0.9999045133590698 1.0\n",
      "2.7312167730997317e-05 0.0\n",
      "0.9997475743293762 1.0\n",
      "0.013849827460944653 0.0\n",
      "9.860284626483917e-05 0.0\n",
      "6.698092420265311e-06 0.0\n",
      "0.9999951124191284 1.0\n",
      "0.9999973773956299 1.0\n",
      "0.9999734163284302 1.0\n",
      "0.9985937476158142 1.0\n",
      "0.9939329028129578 1.0\n",
      "0.9998295307159424 1.0\n",
      "3.298816409369465e-06 0.0\n",
      "0.9999668598175049 1.0\n",
      "0.9986036419868469 1.0\n",
      "0.9992565512657166 1.0\n",
      "3.2260006719297962e-06 0.0\n",
      "4.6330942495842464e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "3.155898866680218e-06 0.0\n",
      "0.9999736547470093 1.0\n",
      "3.35125696437899e-05 0.0\n",
      "0.9999933242797852 1.0\n",
      "2.9763336897303816e-06 0.0\n",
      "0.9999966621398926 1.0\n",
      "0.9999912977218628 1.0\n",
      "7.786382411723025e-06 0.0\n",
      "0.9999953508377075 1.0\n",
      "5.3394647693494335e-05 0.0\n",
      "3.865884991682833e-06 0.0\n",
      "0.9999971389770508 1.0\n",
      "0.999996542930603 1.0\n",
      "TRAIN[steps=14600] loss=0.001754 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14600] loss=0.031215 acc=0.700 P=0.690 R=0.703 F1=0.696235 \n",
      "model sim and label tuples:\n",
      "1.2435036296665203e-05 0.0\n",
      "1.0391052455815952e-05 0.0\n",
      "0.9980820417404175 1.0\n",
      "6.035483966115862e-05 0.0\n",
      "2.706831992327352e-06 0.0\n",
      "0.9999904632568359 1.0\n",
      "3.327464810354286e-06 0.0\n",
      "3.1155195756582543e-06 0.0\n",
      "3.3695564525260124e-06 0.0\n",
      "0.9999943971633911 1.0\n",
      "0.0028503919020295143 0.0\n",
      "0.9998779296875 1.0\n",
      "0.9999973773956299 1.0\n",
      "0.9999915361404419 1.0\n",
      "4.165599875705084e-06 0.0\n",
      "2.895701527450001e-06 0.0\n",
      "0.001304643228650093 0.0\n",
      "0.9999957084655762 1.0\n",
      "4.576319042826071e-06 0.0\n",
      "0.9985848665237427 1.0\n",
      "0.0003698042419273406 0.0\n",
      "0.9999971389770508 1.0\n",
      "0.9999717473983765 1.0\n",
      "0.9999860525131226 1.0\n",
      "0.011743460781872272 0.0\n",
      "0.9581657648086548 1.0\n",
      "0.9999970197677612 1.0\n",
      "0.9983173608779907 1.0\n",
      "0.9999762773513794 1.0\n",
      "0.9999517202377319 1.0\n",
      "0.0005102735012769699 0.0\n",
      "3.0543189950549277e-06 0.0\n",
      "4.485473255044781e-06 0.0\n",
      "0.9990342855453491 1.0\n",
      "3.6601161355065415e-06 0.0\n",
      "0.9999973773956299 1.0\n",
      "0.0001616838708287105 0.0\n",
      "0.9999946355819702 1.0\n",
      "0.9999911785125732 1.0\n",
      "0.9997456669807434 1.0\n",
      "0.9999954700469971 1.0\n",
      "0.9999710321426392 1.0\n",
      "0.9999380111694336 1.0\n",
      "3.524682824718184e-06 0.0\n",
      "0.9999308586120605 1.0\n",
      "0.011678531765937805 0.0\n",
      "3.1757165288581746e-06 0.0\n",
      "0.9997478127479553 1.0\n",
      "2.0395311366883107e-05 0.0\n",
      "0.0004609539464581758 0.0\n",
      "8.342337241629139e-05 0.0\n",
      "3.161746008117916e-06 0.0\n",
      "4.461041953618405e-06 0.0\n",
      "0.9998170733451843 1.0\n",
      "0.9999961853027344 1.0\n",
      "0.9999969005584717 1.0\n",
      "0.9999938011169434 1.0\n",
      "2.779091801130562e-06 0.0\n",
      "3.702287358464673e-05 0.0\n",
      "8.126901775540318e-06 0.0\n",
      "0.9999971389770508 1.0\n",
      "3.752408929358353e-06 0.0\n",
      "4.0694376366445795e-06 0.0\n",
      "0.9999972581863403 1.0\n",
      "TRAIN[steps=14700] loss=0.001241 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14700] loss=0.031249 acc=0.700 P=0.691 R=0.700 F1=0.695369 \n",
      "model sim and label tuples:\n",
      "0.00010283700248692185 0.0\n",
      "0.0006225309916771948 0.0\n",
      "2.1883737645111978e-05 0.0\n",
      "1.338492802460678e-05 0.0\n",
      "6.790594397898531e-06 0.0\n",
      "6.988146196817979e-06 0.0\n",
      "0.9997028708457947 1.0\n",
      "0.9998390674591064 1.0\n",
      "3.496483259368688e-06 0.0\n",
      "5.1792532758554444e-05 0.0\n",
      "0.9999926090240479 1.0\n",
      "6.565183866769075e-06 0.0\n",
      "6.982543254707707e-06 0.0\n",
      "8.857625289238058e-06 0.0\n",
      "0.0001261691068066284 0.0\n",
      "0.9998860359191895 1.0\n",
      "0.9999890327453613 1.0\n",
      "2.802242988764192e-06 0.0\n",
      "0.9999963045120239 1.0\n",
      "3.2789091619633837e-06 0.0\n",
      "3.099253490290721e-06 0.0\n",
      "4.206125231576152e-05 0.0\n",
      "2.496577872079797e-05 0.0\n",
      "0.9999768733978271 1.0\n",
      "0.9999791383743286 1.0\n",
      "0.9998763799667358 1.0\n",
      "0.9837678074836731 1.0\n",
      "1.0255812412651721e-05 0.0\n",
      "3.0854871511110105e-06 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9999322891235352 1.0\n",
      "0.9912967681884766 1.0\n",
      "0.9999889135360718 1.0\n",
      "5.3635372751159593e-05 0.0\n",
      "0.9999899864196777 1.0\n",
      "0.9999911785125732 1.0\n",
      "0.9985588192939758 1.0\n",
      "0.9996005892753601 1.0\n",
      "0.9999973773956299 1.0\n",
      "3.4522172427386977e-06 0.0\n",
      "2.997217734446167e-06 0.0\n",
      "0.00012124184286221862 0.0\n",
      "0.9972062706947327 1.0\n",
      "0.9995409250259399 1.0\n",
      "5.915238943998702e-05 0.0\n",
      "0.9999880790710449 1.0\n",
      "0.11493057757616043 0.0\n",
      "0.9999927282333374 1.0\n",
      "5.790196155430749e-06 0.0\n",
      "0.9999973773956299 1.0\n",
      "0.9999972581863403 1.0\n",
      "3.1918848435452674e-06 0.0\n",
      "0.999734103679657 1.0\n",
      "3.6349349556985544e-06 0.0\n",
      "0.0003465699846856296 0.0\n",
      "5.768266782979481e-05 0.0\n",
      "0.9999924898147583 1.0\n",
      "8.14467784948647e-06 0.0\n",
      "0.9999970197677612 1.0\n",
      "6.489559746114537e-05 0.0\n",
      "2.441135984554421e-05 0.0\n",
      "0.00044867623364552855 0.0\n",
      "0.0006198401097208261 0.0\n",
      "0.9831642508506775 1.0\n",
      "TRAIN[steps=14800] loss=0.002708 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14800] loss=0.031292 acc=0.700 P=0.691 R=0.697 F1=0.693886 \n",
      "model sim and label tuples:\n",
      "0.9998663663864136 1.0\n",
      "0.9999638795852661 1.0\n",
      "0.9999887943267822 1.0\n",
      "0.9999966621398926 1.0\n",
      "0.0011075114598497748 0.0\n",
      "0.9999833106994629 1.0\n",
      "2.3871889425208792e-05 0.0\n",
      "4.05920900448109e-06 0.0\n",
      "8.464599886792712e-06 0.0\n",
      "0.9994747042655945 1.0\n",
      "3.0075029826548416e-06 0.0\n",
      "1.8272063243784942e-05 0.0\n",
      "0.9999791383743286 1.0\n",
      "2.8445399493648438e-06 0.0\n",
      "3.1067879717738833e-06 0.0\n",
      "2.1589223251794465e-05 0.0\n",
      "1.1175490726600401e-05 0.0\n",
      "0.9999912977218628 1.0\n",
      "0.9255819916725159 1.0\n",
      "7.20292518963106e-05 0.0\n",
      "0.9999973773956299 1.0\n",
      "0.9997288584709167 1.0\n",
      "0.9999972581863403 1.0\n",
      "1.4165280845190864e-05 0.0\n",
      "0.9999690055847168 1.0\n",
      "0.9999972581863403 1.0\n",
      "0.9999961853027344 1.0\n",
      "9.563471394358203e-05 0.0\n",
      "0.9995049238204956 1.0\n",
      "0.0004234788357280195 0.0\n",
      "2.7215421596338274e-06 0.0\n",
      "0.00013045196828898042 0.0\n",
      "0.9999960660934448 1.0\n",
      "6.694657258776715e-06 0.0\n",
      "3.055002889595926e-05 0.0\n",
      "0.9999579191207886 1.0\n",
      "0.00024109738296829164 0.0\n",
      "0.9999954700469971 1.0\n",
      "0.9999973773956299 1.0\n",
      "0.0001445042435079813 0.0\n",
      "2.3771955966367386e-05 0.0\n",
      "3.6111152894591214e-06 0.0\n",
      "0.9999697208404541 1.0\n",
      "1.5166114280873444e-05 0.0\n",
      "0.9980792999267578 1.0\n",
      "3.7552122194028925e-06 0.0\n",
      "0.9999957084655762 1.0\n",
      "0.9998520612716675 1.0\n",
      "0.9999699592590332 1.0\n",
      "3.958875367970904e-06 0.0\n",
      "0.9996403455734253 1.0\n",
      "1.7171680156025104e-05 0.0\n",
      "0.00022063979122322053 0.0\n",
      "0.0003359039837960154 0.0\n",
      "3.3947274005186046e-06 0.0\n",
      "3.4563779536256334e-06 0.0\n",
      "5.965019590803422e-05 0.0\n",
      "2.896723572121118e-06 0.0\n",
      "5.711649009754183e-06 0.0\n",
      "0.9999926090240479 1.0\n",
      "0.9999921321868896 1.0\n",
      "0.9960293769836426 1.0\n",
      "6.106071850808803e-06 0.0\n",
      "0.9999945163726807 1.0\n",
      "TRAIN[steps=14900] loss=0.001383 acc=1.000 P=1.000 R=1.000 F1=1.000000\n",
      "DEV[steps=14900] loss=0.031345 acc=0.699 P=0.691 R=0.697 F1=0.693603 \n",
      "EPOCH:  24\n",
      "model sim and label tuples:\n",
      "0.9999518394470215 1.0\n",
      "0.9999942779541016 1.0\n",
      "0.9999628067016602 1.0\n",
      "0.9989700317382812 1.0\n",
      "3.690538869705051e-05 0.0\n",
      "0.00016286135360132903 0.0\n",
      "0.0020264177583158016 0.0\n",
      "7.081633611960569e-06 0.0\n",
      "2.8761398425558582e-05 0.0\n",
      "0.004424108657985926 0.0\n",
      "0.9999902248382568 1.0\n",
      "4.4451762732933275e-06 0.0\n",
      "0.9999583959579468 1.0\n",
      "0.9998750686645508 1.0\n",
      "0.03744308650493622 0.0\n",
      "0.4148732125759125 1.0\n",
      "0.0016910084523260593 0.0\n",
      "8.98571488505695e-06 0.0\n",
      "4.9904294428415596e-05 0.0\n",
      "0.999996542930603 1.0\n",
      "0.9999953508377075 1.0\n",
      "3.250223016948439e-05 0.0\n",
      "0.0006277477950789034 0.0\n",
      "0.999153733253479 1.0\n",
      "0.0002739104093052447 0.0\n",
      "0.00041880106437020004 0.0\n",
      "2.967162572531379e-06 0.0\n",
      "0.9999972581863403 1.0\n",
      "0.9999963045120239 1.0\n",
      "0.9991894364356995 1.0\n",
      "0.9997077584266663 1.0\n",
      "0.000416535563999787 0.0\n",
      "0.999911904335022 1.0\n",
      "0.4343917965888977 0.0\n",
      "0.0002893393102567643 0.0\n",
      "0.9998955726623535 1.0\n",
      "3.549591156115639e-06 0.0\n",
      "8.286454431072343e-06 0.0\n",
      "0.0009926622733473778 0.0\n",
      "0.999575674533844 1.0\n",
      "0.9999909400939941 1.0\n",
      "2.859376763808541e-06 0.0\n",
      "0.9999973773956299 1.0\n",
      "2.7314699764247052e-06 0.0\n",
      "3.445205265961704e-06 0.0\n",
      "0.002542420057579875 0.0\n",
      "0.00011398969945730641 0.0\n",
      "0.9999970197677612 1.0\n",
      "0.9999899864196777 1.0\n",
      "0.999426007270813 1.0\n",
      "0.9999971389770508 1.0\n",
      "1.2144040738348849e-05 0.0\n",
      "0.9999964237213135 1.0\n",
      "0.9999953508377075 1.0\n",
      "0.9998095631599426 1.0\n",
      "1.0350650882173795e-05 0.0\n",
      "1.924675962072797e-05 0.0\n",
      "4.031824573758058e-06 0.0\n",
      "1.0765186743810773e-05 0.0\n",
      "2.5284716684836894e-05 0.0\n",
      "0.9999371767044067 1.0\n",
      "0.0005198097787797451 0.0\n",
      "3.2352956623071805e-05 0.0\n",
      "0.9958094358444214 1.0\n",
      "TRAIN[steps=15000] loss=0.023618 acc=0.984 P=1.000 R=0.966 F1=0.982456\n",
      "DEV[steps=15000] loss=0.031405 acc=0.699 P=0.691 R=0.695 F1=0.692551 \n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  25\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  26\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  27\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  28\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  29\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  30\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  31\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  32\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  33\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  34\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  35\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  36\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  37\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  38\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  39\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  40\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  41\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  42\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  43\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  44\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  45\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  46\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  47\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  48\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  49\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  50\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  51\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  52\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  53\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  54\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  55\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  56\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  57\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  58\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  59\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  60\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  61\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  62\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  63\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  64\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  65\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  66\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  67\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  68\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  69\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  70\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  71\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  72\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  73\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  74\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  75\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  76\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  77\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  78\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  79\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  80\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  81\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  82\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  83\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  84\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  85\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  86\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  87\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  88\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  89\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  90\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  91\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  92\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  93\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  94\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  95\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  96\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  97\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  98\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  99\n",
      "No improvement for a long time, early-stopping at best F1=0.7074009776115417\n",
      "EPOCH:  100\n"
     ]
    }
   ],
   "source": [
    "args = {'lr': 0.001, \n",
    "        'optimizer': 'Adam', \n",
    "        \"cuda\": True, \n",
    "        'epochs': 100,\n",
    "        'batch_size': batch_size,\n",
    "        'log_interval': 100,\n",
    "        'test_interval': 100,\n",
    "        'save_interval' : 1000,\n",
    "        'save_dir': './models/'\n",
    "        }\n",
    "\n",
    "train(train_loader, val_loader, model, args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here is the plots from the training \n",
    "![image](./plots/losses-and-acc3.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task 4: Evaluation (20 Points)\n",
    "\n",
    "1. Report some suitable evaluation metrics. If you stick to standard classification, please report the classification metrics we discussed in the evaluation notebook.\n",
    "2. Check some example and results from the training data.\n",
    "3. Check some examples and results from the validation data (not used for training).\n",
    "4. Come up with one pair of questions and see if your model can produce a reasonable prediction for them. For this, you need to apply the preprocessing pipeline and encoding on the questions' text and make inference to see if the model predicts that they are duplicates.\n",
    "5. Conclude with some comments\n",
    "6. Give us your feedback about the task (at least a sentence)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 11, 300])\n",
      "64\n",
      "question1 question2 predection label\n",
      "Can I prepare for GATE in a month? How should I prepare for GATE in a month? tensor([True]) tensor(1)\n",
      "What do we do if engine smokes? What should I do if my engine is smoking? tensor([True]) tensor(1)\n",
      "What is the oldest civilization on Earth? Which is the world's oldest civilization? tensor([False]) tensor(0)\n",
      "How much weight can I lose in 10 weeks? How much weight can I lose in 8 weeks? tensor([True]) tensor(1)\n",
      "How can I get meth out of my system ASAP? How can I flush meth out of my blood? tensor([False]) tensor(0)\n",
      "How is mental retardation viewed in Asia? How is mental retardation viewed in Europe? tensor([True]) tensor(1)\n",
      "What are the best ways to crack IT interviews? How do I crack an interview? tensor([False]) tensor(0)\n",
      "Are all billionaires avid readers? Are you an avid reader? tensor([True]) tensor(1)\n",
      "What is the best supplement to gain height? What are the best way to gain height? tensor([False]) tensor(0)\n",
      "What is nanotechnology? What is the future of nanotechnology? tensor([False]) tensor(0)\n",
      "What is the answer to this question? If this is the question, what is the answer? tensor([True]) tensor(1)\n",
      "Why is the life of an IAS officer painful? Is life being an IAS officer overhyped in India? tensor([False]) tensor(0)\n",
      "What should I know before I buy a Linux laptop? What are some tips for buying a Linux laptop? tensor([True]) tensor(1)\n",
      "How good are Planet Audio amps? Are Planet Audio amps any good? tensor([True]) tensor(1)\n",
      "What helps you pass a meth test? Does efferdent help pass a drug test? tensor([True]) tensor(1)\n",
      "Why we need Antivirus for computer? Why do I need malware protection? tensor([True]) tensor(1)\n",
      "What is the square root of 100%? What is the square root of the square root? tensor([False]) tensor(0)\n",
      "Which bike is better, Trek or Cannondale? Is the Trek 4500 mountain bike a good bike? tensor([True]) tensor(1)\n",
      "Who is the best person you've ever met and why? Who is the best person you've ever met? tensor([True]) tensor(1)\n",
      "What is giftedness? What does giftedness feel like? tensor([False]) tensor(0)\n",
      "How did Trump win the presidency? Why did Donald Trump win the election? tensor([True]) tensor(1)\n",
      "How many days before my period can I get pregnant? Which time of the month can a girl get pregnant? tensor([False]) tensor(0)\n",
      "What is the oldest religion on earth? Which is the oldest relegion in the world? tensor([True]) tensor(1)\n",
      "How can I speak English like natives? How could an English learner speak fluently? tensor([True]) tensor(1)\n",
      "Are Jatts and Pathans alike in mentality? Can you kill yourself with a 40 caliber handgun? tensor([True]) tensor(1)\n",
      "What does red velvet cake taste like? Why do some people hate to eat chicken? tensor([True]) tensor(1)\n",
      "How does one make rice pudding in a rice cooker? How do I make rice? tensor([False]) tensor(0)\n",
      "What are the asymptotes of y=cotx? What are the asymptotes of y=Tanx? tensor([True]) tensor(1)\n",
      "Is it harmful to eat bananas when you have a cold? Are bruised bananas safe to eat? tensor([True]) tensor(1)\n",
      "How do I reduce my weight? How can I lose an extreme amount of weight? tensor([True]) tensor(1)\n",
      "How should one deal with anxiety attacks? How do you deal with anxiety attacks? tensor([False]) tensor(0)\n",
      "How do I know humanity? How do I know if I have any humanity? tensor([True]) tensor(1)\n",
      "How do I fly? How can one fly in their dreams? tensor([False]) tensor(0)\n",
      "What are the best Radiohead B Side songs? What are the best Radiohead songs? tensor([False]) tensor(0)\n",
      "How can I lose my weight quickly ? How can someone lose weight quickly? tensor([True]) tensor(1)\n",
      "What are the best iOS app websites? What are the best iOS apps and why? tensor([False]) tensor(0)\n",
      "How can I find a good and cheap hotel in Goa? What are cheap but good hotels to live in goa? tensor([True]) tensor(1)\n",
      "When will someone run faster than Usain Bolt? How can I run faster than Usain Bolt? tensor([False]) tensor(0)\n",
      "What are the best SEO practices for 2016? What were the Best SEO practices for 2012? tensor([False]) tensor(0)\n",
      "What is a good comeback for \"Being gay is a sin\"? Do people ever go from gay to straight? tensor([True]) tensor(1)\n",
      "How do I talk to people comfortably? How do I become more comfortable talking to women? tensor([True]) tensor(1)\n",
      "What does a eureka moment feel like? What was your \"eureka\" moment? tensor([True]) tensor(1)\n",
      "What is the scope of pharmaceuticals in India? Scope of pharmacy in India? tensor([True]) tensor(1)\n",
      "What are the best Naruto quotes? What are the best Madara quotes in Naruto? tensor([False]) tensor(0)\n",
      "What is a pure substance in thermodynamics? What is a pure substance? tensor([True]) tensor(1)\n",
      "What is the best way to immigrate to New Zealand? Should I immigrate to Australia or New Zealand? tensor([True]) tensor(1)\n",
      "Are spiders considered insects? How are spiders related to insects? tensor([True]) tensor(1)\n",
      "How can I live a simple life? What does it mean to live a simple life? tensor([False]) tensor(0)\n",
      "How do international student make money? How can international students make money? tensor([False]) tensor(0)\n",
      "What are all the best places to visit in goa? What places should one visit in Goa? tensor([True]) tensor(1)\n",
      "How do you access the Walmart Wire from home? What is Walmart Wire? tensor([False]) tensor(0)\n",
      "What is it like to go through Course 21 at MIT? What is it like to go through Course 10 at MIT? tensor([True]) tensor(1)\n",
      "What is prostate milking? How is it done? What is the purpose of a prostate massage? tensor([False]) tensor(0)\n",
      "How do I get a book published? How can a new writer get published? tensor([True]) tensor(1)\n",
      "How do I apply for internship in IT firms? How do I apply for internship? tensor([False]) tensor(0)\n",
      "How do I calm myself down in anger? How do you calm someone down in anger? tensor([False]) tensor(0)\n",
      "Is it unethical to eat meat? What is wrong with eating meat? tensor([True]) tensor(1)\n",
      "What are the different types of computer clusters? What are types of computer mouse? tensor([False]) tensor(0)\n",
      "How do I improve my pronunciation in English? How do I enhance my English? tensor([True]) tensor(1)\n",
      "How can I learn outside of school? How do you learn in school? tensor([False]) tensor(0)\n",
      "What make the British Raj to leave the India? Why did the British leave our country India? tensor([False]) tensor(0)\n",
      "What are some of the saddest movies ever made? What is the saddest movie you've ever seen? tensor([True]) tensor(1)\n",
      "What is the best way to study ancient Greek? Why should I study ancient Greek? tensor([False]) tensor(0)\n",
      "Which is the fastest chemical reaction? What is the fastest chemical reaction? tensor([True]) tensor(1)\n"
     ]
    }
   ],
   "source": [
    "\n",
    "\n",
    "##### My evaluation metrics was the accurecy since it is very suitable for this task and and the classes were balanced  \n",
    "\n",
    "def predict_from_iter(iter):\n",
    "    #model for testing\n",
    "    model_test =  Dup_Qestions(batch_size=batch_size)\n",
    "    model_test = torch.load(\"./models/snapshot_steps15000.pt\")\n",
    "    model_test.eval()\n",
    "    questions_dataset.questions = True\n",
    "    \n",
    "    with torch.no_grad():    \n",
    "        for batch in iter:\n",
    "            q1_vec, q2_vec, label, q1, q2 = batch\n",
    "            if torch.cuda.is_available():\n",
    "                    q1_vec = q1_vec.to(\"cuda\")\n",
    "                    q2_vec = q2_vec.to(\"cuda\")\n",
    "            q1_vec = q1_vec.float()\n",
    "            q2_vec = q2_vec.float()\n",
    "            print(q1_vec.shape)\n",
    "            score = model_test(q1_vec, q2_vec).cpu()\n",
    "            print(\"question1\", \"question2\", \"predection\", \"label\")\n",
    "            for i,_ in enumerate(q1):\n",
    "                print(q1[i], q2[i], score[i] >= 0.48, label[i])\n",
    "            break\n",
    "    questions_dataset.questions = False\n",
    "# train_dataset.questions = True\n",
    "\n",
    "\n",
    "predict_from_iter(train_loader)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## As you can see above this is the output for the first bacth from the training set. As the plots shoes this model apperently is overfitted on the traning set so all the predications are the same as the true label  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 11, 300])\n",
      "64\n",
      "question1 question2 predection label\n",
      "What are your new year resolutions’2017? What is your New Year resolution? tensor([True]) tensor(1)\n",
      "What are 6 functions of the skin? What are the 4 functions of skin? tensor([True]) tensor(0)\n",
      "What is deep lesioning? How does it help? How do you treat a low density lesion? tensor([False]) tensor(0)\n",
      "How do I write in context free grammar? How can I write a context free grammar for this? tensor([False]) tensor(0)\n",
      "How should I celbrate my 13th birthday? How should I celebrate my 13th birthday? tensor([False]) tensor(0)\n",
      "Who is your favorite band or artist and why? What's your favorite band/artist and why? tensor([False]) tensor(0)\n",
      "How can I count calories in a smart restaurant? How do you count calories? tensor([False]) tensor(0)\n",
      "Who will win, Trump or Clinton? Who will be indicted first, Trump or Clinton? tensor([False]) tensor(0)\n",
      "What are some good ways to lose weight? How can I lose post marriage weight? tensor([False]) tensor(0)\n",
      "How do I get a good score in GRE? How do I prepare for gre? tensor([True]) tensor(1)\n",
      "Are you satisfied with your life? Why or why not? Are you satisfied with your life? tensor([True]) tensor(0)\n",
      "How did Jawaharlal Nehru die? Was Jawaharlal Nehru a Muslim? tensor([False]) tensor(1)\n",
      "What is the price of an iPhone in the USA? What's the cost of iPhone in USA? tensor([False]) tensor(0)\n",
      "How could I be a fashion designer? How can you be a fashion designer? tensor([True]) tensor(1)\n",
      "Why do I moan when I have a fever? How do I get a fever quickly? tensor([False]) tensor(0)\n",
      "At what time should I drink green tea to be fit? How can green tea help you reduce belly fat? tensor([False]) tensor(0)\n",
      "What are some logical fallacies? What are five logical fallacies? tensor([False]) tensor(1)\n",
      "Why are porn sites deemed unsafe? What is an Indian porn site? tensor([False]) tensor(0)\n",
      "How can I meet WW2 veterans? How can I talk to a WW2 veteran? tensor([True]) tensor(1)\n",
      "How can I track a stolen MetroPCS phone? How do I find my stolen phone? tensor([False]) tensor(1)\n",
      "Is it ok to only learn 6 foreign languages? How do I learn a foreign language for free? tensor([True]) tensor(0)\n",
      "How important is sound design in films? What is involved in sound design for a film? tensor([True]) tensor(1)\n",
      "How do I forget someone whom I really love? How can I forget someone whom I loved so deeply? tensor([True]) tensor(1)\n",
      "Why do some people hate romantic relationships? Is being too honest a sign of low intelligence? tensor([True]) tensor(1)\n",
      "How do l study efficiently? How can I improve my study efficiency? tensor([False]) tensor(0)\n",
      "How do I rollback to Windows 8.1 from Windows 10? How do I roll back from Windows 10 to Windows 8.1? tensor([False]) tensor(1)\n",
      "Is money discrete or continuous? Is time discrete or continuous? tensor([False]) tensor(0)\n",
      "Does anyone regret having kids? Do you regret having kids? tensor([True]) tensor(1)\n",
      "How come ISIS has never attacked Israel? Why are there no ISIS attacks on Israel? tensor([False]) tensor(0)\n",
      "How do I start our daily routine as professional? What is a ideal daily routine? tensor([False]) tensor(0)\n",
      "Do women really like men in uniform? Why do women like men in uniform? tensor([True]) tensor(0)\n",
      "Where do I list my apartment for rent? How do I list my home for rent? tensor([True]) tensor(0)\n",
      "What is the stock market? What is sensex, stock market actually? tensor([False]) tensor(1)\n",
      "How do I make poached eggs on toast? How do you make poached eggs on toast? tensor([True]) tensor(1)\n",
      "How can I choose to make the right decision? How do I make right decisions? tensor([False]) tensor(1)\n",
      "How can I approach this guy that I like? How can approach guy I like? tensor([False]) tensor(0)\n",
      "What is the best book to read for a teen? Which is the best book for teens? tensor([True]) tensor(1)\n",
      "Which oven is better Samsung or LG? Which are better, Samsung or LG refrigerators? tensor([True]) tensor(1)\n",
      "What do you do if you witness a crime? What would you do if you witness a crime? tensor([True]) tensor(1)\n",
      "How do I get employed with Quora? How can I get a job on Quora? tensor([False]) tensor(0)\n",
      "Have you ever scammed a scammer? Have you ever lost money in a romance scam? tensor([False]) tensor(0)\n",
      "What will be India's GDP by 2025? What will India's GDP by 2020? tensor([True]) tensor(1)\n",
      "Why does India so scared of CPEC? What reasons does India have to worry about cpec? tensor([True]) tensor(1)\n",
      "How do you get cash without earning it, legally? How do you get money without earning it? tensor([True]) tensor(1)\n",
      "What is the best method of studying? What are some good methods to studying? tensor([True]) tensor(1)\n",
      "What does this Chinese word mean? What does this Chinese word mean on the picture? tensor([True]) tensor(0)\n",
      "Are products on Myntra actually branded? Are there only branded products sell on myntra? tensor([False]) tensor(0)\n",
      "Can you get pregnant from anal? Can you get pregnant through anal sex? tensor([False]) tensor(0)\n",
      "What do you think when you look into a mirror? Why do you look in the mirror? tensor([False]) tensor(0)\n",
      "What does abstract data type mean? What is the stringstream abstract data type? tensor([False]) tensor(1)\n",
      "What is Best design school in india? What are the best designer schools in India? tensor([False]) tensor(0)\n",
      "How do our eyes perceive colors? How do our eyes and brain process color? tensor([True]) tensor(1)\n",
      "What turns you on sexually as a woman or girl? What makes a man attractive to women? tensor([False]) tensor(0)\n",
      "Do you feel sleepy during the day? I feel sleepy all the day, what should I do? tensor([False]) tensor(0)\n",
      "What is the value of pi ? How is the value of [math]\\pi[/math] calculated? tensor([False]) tensor(0)\n",
      "What I can do to last longer during sex? How can women last longer during sex? tensor([False]) tensor(0)\n",
      "What do these symbols mean: ǡ? What do these symbols mean? tensor([False]) tensor(0)\n",
      "How are bitmap images used? What are bitmap images used for? tensor([True]) tensor(1)\n",
      "What is the best free iOS game? What are the best tapping games for iOS? tensor([False]) tensor(0)\n",
      "How did Darth Vader learned Luke was his son? How did Darth Vader know that Luke was his son? tensor([False]) tensor(0)\n",
      "How do I lose weight without doing any sport? Can I lose weight without exercise? tensor([False]) tensor(1)\n",
      "Why do all of my questions get ignored on Quora? Why are my questions not answered on Quora? tensor([True]) tensor(1)\n",
      "Can the world's oil reserves be depleted? Can the world's oil reserves ever be depleted? tensor([False]) tensor(1)\n",
      "What is the correct definition of \"Acche Din\"? \"Acche din aane wale hai\", is this really true? tensor([True]) tensor(1)\n"
     ]
    }
   ],
   "source": [
    "predict_from_iter(val_loader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## As you can see above this is the output for the first bacth from the validation set. The model do not have accurcy one on the validation set this is why you would find some wrong predications "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 152,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question1</th>\n",
       "      <th>question2</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How can i get a citizenship residence in Canada?</td>\n",
       "      <td>How to immigrate to Canada?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>How to find cheap flights?</td>\n",
       "      <td>How to travel with cheap flights ?</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          question1  \\\n",
       "0  How can i get a citizenship residence in Canada?   \n",
       "1                        How to find cheap flights?   \n",
       "\n",
       "                             question2  label  \n",
       "0          How to immigrate to Canada?      1  \n",
       "1   How to travel with cheap flights ?      1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "question1 question2 predection label\n",
      "('How can i get a citizenship residence in Canada?',) ('How to immigrate to Canada?',) tensor([[False]]) tensor([1])\n",
      "1\n",
      "question1 question2 predection label\n",
      "('How to find cheap flights?',) (' How to travel with cheap flights ?',) tensor([[False]]) tensor([1])\n"
     ]
    }
   ],
   "source": [
    "### Your code goes here ####\n",
    "\n",
    "## Since i do all the preprocessing in the question dataset class, i will do tha same for test sets \n",
    "### If you wanna test some sentences just add them to the test.csv dile\n",
    "def predict(file):\n",
    "    #model for testing\n",
    "    model_test =  Dup_Qestions(batch_size=1)\n",
    "    model_test = torch.load(\"./models/snapshot_steps15000.pt\")\n",
    "    model_test.batch_size = 1\n",
    "    model_test.eval()\n",
    "    test_dataset = QuestionsDataset(pd.read_csv(file, sep=\",\"), True)\n",
    "    test_loader = torch.utils.data.DataLoader(dataset=test_dataset,\n",
    "                                           batch_size=1,\n",
    "                                           shuffle=False, sampler=None,\n",
    "                                           collate_fn=None\n",
    "                                           )\n",
    "\n",
    "    with torch.no_grad():    \n",
    "        for q1_vec, q2_vec, label, q1, q2 in test_loader:\n",
    "                if torch.cuda.is_available():\n",
    "                        q1_vec = q1_vec.to(\"cuda\")\n",
    "                        q2_vec = q2_vec.to(\"cuda\")\n",
    "                q1_vec = q1_vec.float()\n",
    "                q2_vec = q2_vec.float()\n",
    "                score = model_test(q1_vec, q2_vec).cpu()\n",
    "                print(\"question1\", \"question2\", \"predection\", \"label\")\n",
    "                print(q1, q2, score >= 0.48, label)\n",
    "\n",
    "\n",
    "predict(\"test.csv\")\n",
    "    \n",
    "\n",
    "\n",
    "\n",
    "### do not forget to add your comments and feedback ###"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## As you can see the prediction of my model is wrong for the two questions of the test.csv file. when checking the score fo it. It predicts new test set so bad "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comments\n",
    "- you will need to download the word2vec embeddings with this command `wget -c \"https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz\"` then put the .gz file in the same folder as the notebook \n",
    "- I created two models one with LSTM and one with conv1d and i have decided to go with the LSTM one because it gives better results\n",
    "- I used the Saimens achitecture where ther is two different paths for each question and then connected them by multiplying them \n",
    "- I focused on the Acc metrics because it makes sense since the classes are balanced and by the end I made it as a simple classification problem \n",
    "- I used word2vec embbedings and I did not use  embedding layer since i did not want to adjust the embeddings \n",
    "- I really had high hopes for the model to work correctly on the test set however it did very bad on it which i do not understand why \n",
    "- I really understand a lot of concepts now that i thought i already understood before. (like batching, preparing the data, the sizes of the vectors, building the train and evaluate function)\n",
    "- You can check the last run logged metrics on WandB from here https://wandb.ai/osobky/dup-questions-project/runs/2ur13rfn\n",
    "- I used (adopted) a adjust learning rate function for the traning function so it can adjust the learning rate \n",
    "\n",
    "# Feed back \n",
    "- The task itself was not hard but the process of doing the whole pipeline was a lot. I am really glad i went through this homework. I really think it made me understand not only the whole picture but also the small parts in the process\n",
    "- However, I would suggest that you divide the task to the 4 parts we did here so we can focus more and more the that spacific part by itself\n",
    "- I entered the task saying i would like to code everything by myself to undersatnd how hard it is to do so.  And it was very hard actually to understand the whole picture. \n",
    "- as a conclusion I think i learnt a lot but i still have a long way ahead of me \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Submission Notes:**\n",
    "\n",
    "1. You can surely use external files to organize your code in classes or modules (e.g. `.py` files). However, please make sure that the notebook can run without errors and all the required files are attached in your submission (e.g. .zip file). If everything is confined in one single notebook, just submit the notebook.\n",
    "2. If you use special libraries or packages, please indicate that clearly and add a `requirements.txt` file to your submission.\n",
    "3. Do not upload the dataset nor submit it in any way.\n",
    "4. Please do not copy code from someone else, the idea here is that you get a chance to write some code in PyTorch and solve the problem on your own. On the other hand, discuss with your colleagues and support them as much as needed. \n",
    "5. You can of course reuse the code in all the notebooks of the course.\n",
    "6. You can also use code and/or ideas from the internet (e.g. Kaggle notebooks), but please always do the following:\n",
    "    - make sure you understand the code so that you can use it correctly\n",
    "    - cite the source in your notebook as markdown or as a comment in the code (`# adopted from .......`) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Evaluation Criteria:**\n",
    "\n",
    "The most important idea we will use for evaluation is that you should get every component of the complete pipeline (1) doing what it is supposed to do and (2) fitting with the other components.\n",
    "\n",
    "Concrete Examples are:\n",
    "1. Your code runs and we can reproduce the results.\n",
    "2. Dataset: Your code reads the dataset, preprocesses it, splits the text into tokens, etc...\n",
    "3. Dataloading: Your dataset and dataloader produce correct inputs and labels (two questions as input, one binary label).\n",
    "4. Model: Your model takes the input (processes each question correctly) and produces a binary prediction (or score).\n",
    "5. Loss: your chosen loss is suitable for the task (e.g. binary cross entropy in case of the simple classification setup).\n",
    "6. Training: your code for training and validation works without errors and the loss on the training and validation data decreases over the course of training. You do not have to achieve a specific performance.\n",
    "7. Evaluation: you discuss the results and pick correctly some questions from train and validation and show the respective predictions (preferably showing the original question text and not the encoded integer sequence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus:**\n",
    "If you do any of the following ideas, you get a bonus, additionally you get to learn more, which is better than the bonus:\n",
    "\n",
    "1. Build a baseline model to compare and benchmark your deep learning model against. You can in this case use a classical machine learning model from `scikit-learn`. A starter example is here: https://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html\n",
    "2. Use W&B or Tensorboard to visualize your training. TensorBoard tutorial is available here: (https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html)\n",
    "3. Achieve a relatively good performance on the task.\n",
    "4. Use a learning rate scheduler to improve training. (https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)\n",
    "5. Use pre-trained embeddings correctly.\n",
    "6. Use an advanced loss suitable for a siamese-network.\n",
    "7. Implement any relevant new idea or discuss an interesting insight about the task."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "46243e451fc90de02aec34bcce17d277994e57c72ad5e9f1f2430300b26f0563"
  },
  "kernelspec": {
   "display_name": "Python 3.7.6 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}