gtc_solaiman_RoBERTa.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "91fdfe4f",
   "metadata": {},
   "source": [
    "# Benchmark of existing approaches for detecting machine-generated text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12b54398-b324-490d-b101-f210c3357e4f",
   "metadata": {},
   "source": [
    "This notebook enables to evaluate the RoBERTa-based text classifier by Solaiman et al."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d88e7c3",
   "metadata": {},
   "source": [
    "## Solaiman: fine-tuned RoBERTa\n",
    "\n",
    "\n",
    "_Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford,\n",
    "Gretchen Krueger, Jong Wook Kim, Sarah Kreps, Miles McCain, Alex Newhouse, Jason Blazakis, Kris McGuffie, and Jasmine Wang. 2019. Release strategies and the social impacts of language models._"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c6bb8ef",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Install dependencies\n",
    "To run the code in this notebook, you must install the required dependencies. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83b626d6-d1ce-45e3-94da-069d7bf4ece7",
   "metadata": {},
   "source": [
    "#### Virtual environment \n",
    "The suggested approach for installing all required dependencies is by using a corda environment, importing configuration from the `env_transf291.yml` file. \n",
    "\n",
    "Create a new environment using conda\n",
    "``` bash \n",
    "conda env create -f env_transf291.yml\n",
    "```\n",
    "\n",
    "Activate and install `ipykernel`\n",
    "```bash\n",
    "conda activate transf291\n",
    "conda install ipykernel\n",
    "```\n",
    "You should be able to use your conda environent as Jupyter Notebook kernels. \n",
    "In case you can't use transf291, you can try by adding it as kernel as follows. \n",
    "``` bash \n",
    "ipython kernel install --user --name=transf291\n",
    "```\n",
    "\n",
    "Start Jupyter Notebook: \n",
    "```bash \n",
    "jupyter notebook\n",
    "```\n",
    "\n",
    "#### Manual installation\n",
    "Dependencies from: [https://github.com/HendrikStrobelt/detecting-fake-text/blob/master/requirements.txt](detecting-fake-text/requirements.txt)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70ac2b84",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Solaiman et al. Code\n",
    "Implementation of the RoBERTa-based text classifier. \n",
    "\n",
    "Logic extracted form `server.py` by Solaiman et al.: [https://github.com/openai/gpt-2-output-dataset/tree/master/detector](https://github.com/openai/gpt-2-output-dataset/tree/master/detector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a610aa01",
   "metadata": {},
   "outputs": [],
   "source": [
    "import transformers\n",
    "assert transformers.__version__ == '2.9.1', \"Use transformers 2.9.1. It is available in the conda environment transf291\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f3fe8f6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import RobertaForSequenceClassification, RobertaTokenizer\n",
    "import json\n",
    "import torch\n",
    "from urllib.parse import urlparse, unquote\n",
    "import pandas as pd\n",
    "import os\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fc8f4e7",
   "metadata": {},
   "source": [
    "Select model to use RoBERTa base or large"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7904206f",
   "metadata": {},
   "outputs": [],
   "source": [
    "model_name = 'roberta-large'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb19546b",
   "metadata": {},
   "source": [
    "Define model, tokenizer, and basic functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a92b498e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RobertaForSequenceClassification.from_pretrained(model_name)\n",
    "tokenizer = RobertaTokenizer.from_pretrained(model_name)\n",
    "device='cuda' if torch.cuda.is_available() else 'cpu'\n",
    "\n",
    "device = 'cpu'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "530775e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(query):\n",
    "    tokens = tokenizer.encode(query)\n",
    "    all_tokens = len(tokens)\n",
    "    tokens = tokens[:tokenizer.max_len - 2]\n",
    "    used_tokens = len(tokens)\n",
    "    tokens = torch.tensor([tokenizer.bos_token_id] + tokens + [tokenizer.eos_token_id]).unsqueeze(0)\n",
    "    mask = torch.ones_like(tokens)\n",
    "\n",
    "    with torch.no_grad():\n",
    "        logits = model(tokens.to(device), attention_mask=mask.to(device))[0]\n",
    "        probs = logits.softmax(dim=-1)\n",
    "\n",
    "    fake, real = probs.detach().cpu().flatten().numpy().tolist()\n",
    "\n",
    "    # Original: \n",
    "#    return json.dumps(dict(\n",
    "#         all_tokens=all_tokens,\n",
    "#         used_tokens=used_tokens,\n",
    "#         real_probability=real,\n",
    "#         fake_probability=fake\n",
    "#     ))\n",
    "\n",
    "# Changed to return only the binary classification result. 1 if sentence is likely machine-generated.\n",
    "    return (fake > 0.5, fake, real)\n",
    "\n",
    "def initialize(checkpoint):\n",
    "#     if checkpoint.startswith('gs://'):\n",
    "#         print(f'Downloading {checkpoint}', file=sys.stderr)\n",
    "#         subprocess.check_output(['gsutil', 'cp', checkpoint, '.'])\n",
    "#         checkpoint = os.path.basename(checkpoint)\n",
    "#         assert os.path.isfile(checkpoint)\n",
    "\n",
    "    print(f'Loading checkpoint from {checkpoint}')\n",
    "    data = torch.load(checkpoint, map_location='cpu')\n",
    "    model.load_state_dict(data['model_state_dict'])\n",
    "    model.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8710860",
   "metadata": {},
   "source": [
    "Download finetuned model and load it into RoBERTa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "744b6e67",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2023-12-11 14:52:10--  https://openaipublic.azureedge.net/gpt-2/detector-models/v1/detector-large.pt\n",
      "Risoluzione di openaipublic.azureedge.net (openaipublic.azureedge.net)... 2620:1ec:bdf::43, 2620:1ec:46::43, 13.107.213.43, ...\n",
      "Connessione a openaipublic.azureedge.net (openaipublic.azureedge.net)|2620:1ec:bdf::43|:443... connesso.\n",
      "Richiesta HTTP inviata, in attesa di risposta... 200 OK\n",
      "Lunghezza: 1425743818 (1,3G) [application/octet-stream]\n",
      "Salvataggio in: «detector-large.pt.3»\n",
      "\n",
      "detector-large.pt.3 100%[===================>]   1,33G  41,5MB/s    in 31s     \n",
      "\n",
      "2023-12-11 14:52:42 (43,6 MB/s) - «detector-large.pt.3» salvato [1425743818/1425743818]\n",
      "\n",
      "Loading checkpoint from detector-large.pt\n"
     ]
    }
   ],
   "source": [
    "# Make sure that you can run wget on this machine\n",
    "!wget https://openaipublic.azureedge.net/gpt-2/detector-models/v1/detector-large.pt\n",
    "initialize('detector-large.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a8e4d59",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c437de8-7f57-4ac2-b511-1d846a202edf",
   "metadata": {},
   "source": [
    "This section is devoted to evaluating the Solaiman et al. model under different datasets. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "68ccafed",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_path = \"./data/test/\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdb76c45-5b65-4e08-8f15-872cc9b3068c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Auxiliary functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e9842c64",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.utils import shuffle\n",
    "\n",
    "def create_dataset(raw_data, is_machine_generated=True):\n",
    "    X = np.array(raw_data)\n",
    "    if is_machine_generated:\n",
    "        y = np.ones_like(X, dtype=int)\n",
    "    else:\n",
    "        y = np.zeros_like(X, dtype=int)\n",
    "    return pd.DataFrame(data={'X':X, 'y':y})\n",
    "\n",
    "def concat_and_shuffle(datasets):\n",
    "    return shuffle(pd.concat(datasets)).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "601a8330",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import confusion_matrix, accuracy_score\n",
    "\n",
    "def safe_macro_f1(y, y_pred):\n",
    "    \"\"\"\n",
    "    Macro-averaged F1, forcing `sklearn` to report as a multiclass\n",
    "    problem even when there are just two classes. `y` is the list of\n",
    "    gold labels and `y_pred` is the list of predicted labels.\n",
    "\n",
    "    \"\"\"\n",
    "    return f1_score(y, y_pred, average='macro', pos_label=None)\n",
    "\n",
    "def safe_accuracy(y, y_pred):\n",
    "    return accuracy_score(y, y_pred, normalize=True)\n",
    "\n",
    "def gtc_evaluate(\n",
    "        dataset,\n",
    "        model,\n",
    "        score_func=safe_accuracy):\n",
    "\n",
    "    # Predictions if we have labels:\n",
    "    y_ds, preds = model.predict(dataset)\n",
    "    confusion = confusion_matrix(y_ds, preds)\n",
    "    score = score_func(y_ds, preds)\n",
    "        \n",
    "    # Return the overall scores and other experimental info:\n",
    "    return {\n",
    "        'model': model,\n",
    "#        'labeled_outputs': y_ds, \n",
    "#        'predictions': preds, \n",
    "        'confusion_matrix': confusion, \n",
    "        'score': score }\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6c26b9c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Wrapper:\n",
    "    def __init__(self):\n",
    "        pass\n",
    "\n",
    "    def predict(self, dataset, truncateAt=512):\n",
    "        y_pred = []\n",
    "        y_dataset = []\n",
    "        \n",
    "        totalsize = len(dataset)\n",
    "        progressupdate = max(min(100, int(totalsize/10)), 1)\n",
    "        prog = 0\n",
    "        print(f\"progupd: {progressupdate}; m(m(100, {int(totalsize/10)}), 1)\")\n",
    "        for index, row in dataset.iterrows():\n",
    "            prog+=1\n",
    "            if not row['X']: \n",
    "                print(\"Skipping sentence because it appears to be empty\")\n",
    "                continue\n",
    "            try:\n",
    "                truncated_sentence = row['X'][:truncateAt] if len(row['X']) > truncateAt else row['X']\n",
    "                (is_generated, fake, real) = evaluate(truncated_sentence)\n",
    "                y_pred.append(1 if is_generated else 0)     \n",
    "                y_dataset.append(row['y'])\n",
    "                \n",
    "                if (index) % progressupdate == 0:\n",
    "                    print(f\"... progress: {index} ({prog / totalsize})\")\n",
    "            except:\n",
    "                print(f\"Error while processing the query: {row['X']}\")\n",
    "                break\n",
    "        return (y_dataset, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "dfa03d90",
   "metadata": {},
   "outputs": [],
   "source": [
    "solaiman_detector = Wrapper()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b4c089",
   "metadata": {},
   "source": [
    "Testing whether the `gtc_evaluate()` works as expected. By default, we use accuracy to measure the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ea42da12",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gtc_evaluate_with_checkpoints(dataset, \n",
    "        model,\n",
    "        checkpoint = 10000,\n",
    "        score_func=safe_accuracy):\n",
    "    length = len(dataset)\n",
    "    splits = min(int(length/checkpoint), length) \n",
    "    confmatrix_sum = np.zeros((2, 2))\n",
    "    for i in range(splits + 1):\n",
    "        dataset_split = dataset[i * checkpoint : (i+1)*checkpoint]\n",
    "        if dataset_split.empty:\n",
    "            break\n",
    "        partial_r = gtc_evaluate(dataset_split, solaiman_detector, score_func=score_func)\n",
    "        print (f\"----- split #{i} (size: {len(dataset_split)}) -----\\n {partial_r['confusion_matrix']}\\n----- acc: {partial_r['score']} -----\")\n",
    "        confmatrix_sum += partial_r['confusion_matrix']\n",
    "    print(f\"====================================\")\n",
    "    print(f\" Final confusion matrix:\\n {confmatrix_sum}\")\n",
    "    (tn, fp, fn, tp) = confmatrix_sum.ravel()\n",
    "    accuracy = (tp +tn) / (tp + tn + fp + fn)\n",
    "    print(f\" Final accuracy: {accuracy}\")\n",
    "    print(f\"====================================\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8656bd2f",
   "metadata": {},
   "source": [
    "The Solaiman's model warns that sentences should have less than 512 tokens. So, let's try by filtering out senteces with more tokens than that."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af94dd0a",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on WebText + GPT2-xl-1542M data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f20cc24e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./data/test/'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6c463d65",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_hwraw_filename = 'webtext.test.human.jsonl'\n",
    "ds_hw_raw = pd.read_json(os.path.join(data_path, ds_hwraw_filename), lines = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3622eb69",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_mgraw_filename = 'GPT2-xl-1542M.test.machine.jsonl'\n",
    "ds_mg_raw = pd.read_json(os.path.join(data_path, ds_mgraw_filename), lines = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e88e8512",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Is this restaurant family-friendly ? Yes No Un...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Clinton talks about her time of 'reflection' d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>House Majority Whip Steve Scalise has been dis...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Insight Course: Lesson 14\\n\\nControl of the Mi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>BY JENNIE MCNULTY\\n\\nLesbian.com\\n\\nYou know W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>4995</td>\n",
       "      <td>The week after my 30th birthday, my best frien...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>4996</td>\n",
       "      <td>Downton Abbey doesn't return for its third sea...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>4997</td>\n",
       "      <td>Amazon Studios said today that Stephen Root ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>4998</td>\n",
       "      <td>Everyone likes clean wheels but not everyone h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>4999</td>\n",
       "      <td>This post goes out to those of you who have no...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                               text\n",
       "0        0  Is this restaurant family-friendly ? Yes No Un...\n",
       "1        1  Clinton talks about her time of 'reflection' d...\n",
       "2        2  House Majority Whip Steve Scalise has been dis...\n",
       "3        3  Insight Course: Lesson 14\\n\\nControl of the Mi...\n",
       "4        4  BY JENNIE MCNULTY\\n\\nLesbian.com\\n\\nYou know W...\n",
       "...    ...                                                ...\n",
       "4995  4995  The week after my 30th birthday, my best frien...\n",
       "4996  4996  Downton Abbey doesn't return for its third sea...\n",
       "4997  4997  Amazon Studios said today that Stephen Root ha...\n",
       "4998  4998  Everyone likes clean wheels but not everyone h...\n",
       "4999  4999  This post goes out to those of you who have no...\n",
       "\n",
       "[5000 rows x 2 columns]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_hw_raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b307736f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>What is this?\\n\\nThis is a neat little plugin ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>People carry a memorial sign to honor Robert P...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Google is launching a new scholarship campaign...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>Authored by: Kristin Scavuzzo, Managing Direct...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>&gt;at some point the MPP team stop standardizing...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>4995</td>\n",
       "      <td>Mac coverage beginning in 1926 at The Michigan...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>4996</td>\n",
       "      <td>Martyr Dadabhoyao got a big experience about s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>4997</td>\n",
       "      <td>\"The nation's smugglers have changed their tac...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>4998</td>\n",
       "      <td>The level of British public support for direct...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>4999</td>\n",
       "      <td>The Minnesota State Patrol is investigating a ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id                                               text\n",
       "0        0  What is this?\\n\\nThis is a neat little plugin ...\n",
       "1        1  People carry a memorial sign to honor Robert P...\n",
       "2        2  Google is launching a new scholarship campaign...\n",
       "3        3  Authored by: Kristin Scavuzzo, Managing Direct...\n",
       "4        4  >at some point the MPP team stop standardizing...\n",
       "...    ...                                                ...\n",
       "4995  4995  Mac coverage beginning in 1926 at The Michigan...\n",
       "4996  4996  Martyr Dadabhoyao got a big experience about s...\n",
       "4997  4997  \"The nation's smugglers have changed their tac...\n",
       "4998  4998  The level of British public support for direct...\n",
       "4999  4999  The Minnesota State Patrol is investigating a ...\n",
       "\n",
       "[5000 rows x 2 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_mg_raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "047f217a",
   "metadata": {},
   "outputs": [],
   "source": [
    "mg_raw_pd = create_dataset(ds_mg_raw[\"text\"], is_machine_generated=True)\n",
    "hw_raw_pd = create_dataset(ds_hw_raw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "906f5746",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What is this?\\n\\nThis is a neat little plugin ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>People carry a memorial sign to honor Robert P...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Google is launching a new scholarship campaign...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Authored by: Kristin Scavuzzo, Managing Direct...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&gt;at some point the MPP team stop standardizing...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>Mac coverage beginning in 1926 at The Michigan...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>Martyr Dadabhoyao got a big experience about s...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>\"The nation's smugglers have changed their tac...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>The level of British public support for direct...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>The Minnesota State Patrol is investigating a ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      X  y\n",
       "0     What is this?\\n\\nThis is a neat little plugin ...  1\n",
       "1     People carry a memorial sign to honor Robert P...  1\n",
       "2     Google is launching a new scholarship campaign...  1\n",
       "3     Authored by: Kristin Scavuzzo, Managing Direct...  1\n",
       "4     >at some point the MPP team stop standardizing...  1\n",
       "...                                                 ... ..\n",
       "4995  Mac coverage beginning in 1926 at The Michigan...  1\n",
       "4996  Martyr Dadabhoyao got a big experience about s...  1\n",
       "4997  \"The nation's smugglers have changed their tac...  1\n",
       "4998  The level of British public support for direct...  1\n",
       "4999  The Minnesota State Patrol is investigating a ...  1\n",
       "\n",
       "[5000 rows x 2 columns]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mg_raw_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0fa47d18",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Is this restaurant family-friendly ? Yes No Un...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Clinton talks about her time of 'reflection' d...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>House Majority Whip Steve Scalise has been dis...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Insight Course: Lesson 14\\n\\nControl of the Mi...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>BY JENNIE MCNULTY\\n\\nLesbian.com\\n\\nYou know W...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4995</th>\n",
       "      <td>The week after my 30th birthday, my best frien...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4996</th>\n",
       "      <td>Downton Abbey doesn't return for its third sea...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4997</th>\n",
       "      <td>Amazon Studios said today that Stephen Root ha...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4998</th>\n",
       "      <td>Everyone likes clean wheels but not everyone h...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4999</th>\n",
       "      <td>This post goes out to those of you who have no...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      X  y\n",
       "0     Is this restaurant family-friendly ? Yes No Un...  0\n",
       "1     Clinton talks about her time of 'reflection' d...  0\n",
       "2     House Majority Whip Steve Scalise has been dis...  0\n",
       "3     Insight Course: Lesson 14\\n\\nControl of the Mi...  0\n",
       "4     BY JENNIE MCNULTY\\n\\nLesbian.com\\n\\nYou know W...  0\n",
       "...                                                 ... ..\n",
       "4995  The week after my 30th birthday, my best frien...  0\n",
       "4996  Downton Abbey doesn't return for its third sea...  0\n",
       "4997  Amazon Studios said today that Stephen Root ha...  0\n",
       "4998  Everyone likes clean wheels but not everyone h...  0\n",
       "4999  This post goes out to those of you who have no...  0\n",
       "\n",
       "[5000 rows x 2 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hw_raw_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "769806c8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 0 (0.001)\n",
      "... progress: 100 (0.101)\n",
      "... progress: 200 (0.201)\n",
      "... progress: 300 (0.301)\n",
      "... progress: 400 (0.401)\n",
      "... progress: 500 (0.501)\n",
      "... progress: 600 (0.601)\n",
      "... progress: 700 (0.701)\n",
      "... progress: 800 (0.801)\n",
      "... progress: 900 (0.901)\n",
      "----- split #0 (size: 1000) -----\n",
      " [[  0   0]\n",
      " [132 868]]\n",
      "----- acc: 0.868 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 1000 (0.001)\n",
      "... progress: 1100 (0.101)\n",
      "... progress: 1200 (0.201)\n",
      "... progress: 1300 (0.301)\n",
      "... progress: 1400 (0.401)\n",
      "... progress: 1500 (0.501)\n",
      "... progress: 1600 (0.601)\n",
      "... progress: 1700 (0.701)\n",
      "... progress: 1800 (0.801)\n",
      "... progress: 1900 (0.901)\n",
      "----- split #1 (size: 1000) -----\n",
      " [[  0   0]\n",
      " [124 876]]\n",
      "----- acc: 0.876 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 2000 (0.001)\n",
      "... progress: 2100 (0.101)\n",
      "... progress: 2200 (0.201)\n",
      "... progress: 2300 (0.301)\n",
      "... progress: 2400 (0.401)\n",
      "... progress: 2500 (0.501)\n",
      "... progress: 2600 (0.601)\n",
      "... progress: 2700 (0.701)\n",
      "... progress: 2800 (0.801)\n",
      "... progress: 2900 (0.901)\n",
      "----- split #2 (size: 1000) -----\n",
      " [[  0   0]\n",
      " [137 863]]\n",
      "----- acc: 0.863 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 3000 (0.001)\n",
      "... progress: 3100 (0.101)\n",
      "... progress: 3200 (0.201)\n",
      "... progress: 3300 (0.301)\n",
      "... progress: 3400 (0.401)\n",
      "... progress: 3500 (0.501)\n",
      "... progress: 3600 (0.601)\n",
      "... progress: 3700 (0.701)\n",
      "... progress: 3800 (0.801)\n",
      "... progress: 3900 (0.901)\n",
      "----- split #3 (size: 1000) -----\n",
      " [[  0   0]\n",
      " [121 879]]\n",
      "----- acc: 0.879 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 4000 (0.001)\n",
      "... progress: 4100 (0.101)\n",
      "... progress: 4200 (0.201)\n",
      "... progress: 4300 (0.301)\n",
      "... progress: 4400 (0.401)\n",
      "... progress: 4500 (0.501)\n",
      "... progress: 4600 (0.601)\n",
      "... progress: 4700 (0.701)\n",
      "... progress: 4800 (0.801)\n",
      "... progress: 4900 (0.901)\n",
      "----- split #4 (size: 1000) -----\n",
      " [[  0   0]\n",
      " [125 875]]\n",
      "----- acc: 0.875 -----\n",
      "====================================\n",
      " Final confusion matrix:\n",
      " [[   0.    0.]\n",
      " [ 639. 4361.]]\n",
      " Final accuracy: 0.8722\n",
      "====================================\n",
      "CPU times: user 2h 51min 35s, sys: 26.5 s, total: 2h 52min 2s\n",
      "Wall time: 14min 55s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gtc_evaluate_with_checkpoints(mg_raw_pd, solaiman_detector, checkpoint=1000, score_func=safe_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "09681733",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 0 (0.001)\n",
      "... progress: 100 (0.101)\n",
      "... progress: 200 (0.201)\n",
      "... progress: 300 (0.301)\n",
      "... progress: 400 (0.401)\n",
      "... progress: 500 (0.501)\n",
      "... progress: 600 (0.601)\n",
      "... progress: 700 (0.701)\n",
      "... progress: 800 (0.801)\n",
      "... progress: 900 (0.901)\n",
      "----- split #0 (size: 1000) -----\n",
      " [[955  45]\n",
      " [  0   0]]\n",
      "----- acc: 0.955 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 1000 (0.001)\n",
      "... progress: 1100 (0.101)\n",
      "... progress: 1200 (0.201)\n",
      "... progress: 1300 (0.301)\n",
      "... progress: 1400 (0.401)\n",
      "... progress: 1500 (0.501)\n",
      "... progress: 1600 (0.601)\n",
      "... progress: 1700 (0.701)\n",
      "... progress: 1800 (0.801)\n",
      "... progress: 1900 (0.901)\n",
      "----- split #1 (size: 1000) -----\n",
      " [[947  53]\n",
      " [  0   0]]\n",
      "----- acc: 0.947 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 2000 (0.001)\n",
      "... progress: 2100 (0.101)\n",
      "... progress: 2200 (0.201)\n",
      "... progress: 2300 (0.301)\n",
      "... progress: 2400 (0.401)\n",
      "... progress: 2500 (0.501)\n",
      "... progress: 2600 (0.601)\n",
      "... progress: 2700 (0.701)\n",
      "... progress: 2800 (0.801)\n",
      "... progress: 2900 (0.901)\n",
      "----- split #2 (size: 1000) -----\n",
      " [[941  59]\n",
      " [  0   0]]\n",
      "----- acc: 0.941 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 3000 (0.001)\n",
      "... progress: 3100 (0.101)\n",
      "... progress: 3200 (0.201)\n",
      "... progress: 3300 (0.301)\n",
      "... progress: 3400 (0.401)\n",
      "... progress: 3500 (0.501)\n",
      "... progress: 3600 (0.601)\n",
      "... progress: 3700 (0.701)\n",
      "... progress: 3800 (0.801)\n",
      "... progress: 3900 (0.901)\n",
      "----- split #3 (size: 1000) -----\n",
      " [[937  63]\n",
      " [  0   0]]\n",
      "----- acc: 0.937 -----\n",
      "progupd: 100; m(m(100, 100), 1)\n",
      "... progress: 4000 (0.001)\n",
      "... progress: 4100 (0.101)\n",
      "... progress: 4200 (0.201)\n",
      "... progress: 4300 (0.301)\n",
      "... progress: 4400 (0.401)\n",
      "... progress: 4500 (0.501)\n",
      "... progress: 4600 (0.601)\n",
      "... progress: 4700 (0.701)\n",
      "... progress: 4800 (0.801)\n",
      "... progress: 4900 (0.901)\n",
      "----- split #4 (size: 1000) -----\n",
      " [[956  44]\n",
      " [  0   0]]\n",
      "----- acc: 0.956 -----\n",
      "====================================\n",
      " Final confusion matrix:\n",
      " [[4736.  264.]\n",
      " [   0.    0.]]\n",
      " Final accuracy: 0.9472\n",
      "====================================\n",
      "CPU times: user 2h 44min 55s, sys: 18.1 s, total: 2h 45min 13s\n",
      "Wall time: 14min 14s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gtc_evaluate_with_checkpoints(hw_raw_pd, solaiman_detector, checkpoint=1000, score_func=safe_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dbf0713",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GPT3-175b Machine generated text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9804b03b",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_data_path = \"./data/test/\"\n",
    "gpt3_ds_mg_filename = 'GPT3-175b.test.machine.jsonl'\n",
    "with open(os.path.join(gpt3_data_path, gpt3_ds_mg_filename), 'r') as file:\n",
    "    gpt3_ds_mg = [x for x in file.readlines() if x is not None and x != \"\\n\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7d8f3a8a",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpt3_mg_pd = create_dataset(gpt3_ds_mg, is_machine_generated=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "8cbae414",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>{\"id\":0,\"text\":\"Glacier Ridge Christian School...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>{\"id\":1,\"text\":\"His father was a professor and...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>{\"id\":2,\"text\":\"Image caption Bailiffs removed...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>{\"id\":3,\"text\":\"Contents\\n\\n1. The Meaning of ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>{\"id\":4,\"text\":\"Interviews\\n\\nInterview with A...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>480</th>\n",
       "      <td>{\"id\":480,\"text\":\"Get Rangers updates directly...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>481</th>\n",
       "      <td>{\"id\":481,\"text\":\"Posted 01 September 2010 - 0...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>482</th>\n",
       "      <td>{\"id\":482,\"text\":\"Jeff Chapman\\n\\nJeffrey Scot...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>483</th>\n",
       "      <td>{\"id\":483,\"text\":\"You can look into the Issue ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>484</th>\n",
       "      <td>{\"id\":484,\"text\":\"Airport attack: What happene...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>485 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                     X  y\n",
       "0    {\"id\":0,\"text\":\"Glacier Ridge Christian School...  1\n",
       "1    {\"id\":1,\"text\":\"His father was a professor and...  1\n",
       "2    {\"id\":2,\"text\":\"Image caption Bailiffs removed...  1\n",
       "3    {\"id\":3,\"text\":\"Contents\\n\\n1. The Meaning of ...  1\n",
       "4    {\"id\":4,\"text\":\"Interviews\\n\\nInterview with A...  1\n",
       "..                                                 ... ..\n",
       "480  {\"id\":480,\"text\":\"Get Rangers updates directly...  1\n",
       "481  {\"id\":481,\"text\":\"Posted 01 September 2010 - 0...  1\n",
       "482  {\"id\":482,\"text\":\"Jeff Chapman\\n\\nJeffrey Scot...  1\n",
       "483  {\"id\":483,\"text\":\"You can look into the Issue ...  1\n",
       "484  {\"id\":484,\"text\":\"Airport attack: What happene...  1\n",
       "\n",
       "[485 rows x 2 columns]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpt3_mg_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "14670b60",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progupd: 4; m(m(100, 4), 1)\n",
      "... progress: 0 (0.002061855670103093)\n",
      "... progress: 4 (0.010309278350515464)\n",
      "... progress: 8 (0.018556701030927835)\n",
      "... progress: 12 (0.026804123711340205)\n",
      "... progress: 16 (0.03505154639175258)\n",
      "... progress: 20 (0.04329896907216495)\n",
      "... progress: 24 (0.05154639175257732)\n",
      "... progress: 28 (0.05979381443298969)\n",
      "... progress: 32 (0.06804123711340206)\n",
      "... progress: 36 (0.07628865979381444)\n",
      "... progress: 40 (0.08453608247422681)\n",
      "... progress: 44 (0.09278350515463918)\n",
      "... progress: 48 (0.10103092783505155)\n",
      "... progress: 52 (0.10927835051546392)\n",
      "... progress: 56 (0.11752577319587629)\n",
      "... progress: 60 (0.12577319587628866)\n",
      "... progress: 64 (0.13402061855670103)\n",
      "... progress: 68 (0.1422680412371134)\n",
      "... progress: 72 (0.15051546391752577)\n",
      "... progress: 76 (0.15876288659793814)\n",
      "... progress: 80 (0.1670103092783505)\n",
      "... progress: 84 (0.17525773195876287)\n",
      "... progress: 88 (0.18350515463917524)\n",
      "... progress: 92 (0.19175257731958764)\n",
      "... progress: 96 (0.2)\n",
      "... progress: 100 (0.20824742268041238)\n",
      "... progress: 104 (0.21649484536082475)\n",
      "... progress: 108 (0.22474226804123712)\n",
      "... progress: 112 (0.2329896907216495)\n",
      "... progress: 116 (0.24123711340206186)\n",
      "... progress: 120 (0.24948453608247423)\n",
      "... progress: 124 (0.25773195876288657)\n",
      "... progress: 128 (0.26597938144329897)\n",
      "... progress: 132 (0.27422680412371137)\n",
      "... progress: 136 (0.2824742268041237)\n",
      "... progress: 140 (0.2907216494845361)\n",
      "... progress: 144 (0.29896907216494845)\n",
      "... progress: 148 (0.30721649484536084)\n",
      "... progress: 152 (0.3154639175257732)\n",
      "... progress: 156 (0.3237113402061856)\n",
      "... progress: 160 (0.3319587628865979)\n",
      "... progress: 164 (0.3402061855670103)\n",
      "... progress: 168 (0.34845360824742266)\n",
      "... progress: 172 (0.35670103092783506)\n",
      "... progress: 176 (0.3649484536082474)\n",
      "... progress: 180 (0.3731958762886598)\n",
      "... progress: 184 (0.38144329896907214)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (1026 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... progress: 188 (0.38969072164948454)\n",
      "... progress: 192 (0.3979381443298969)\n",
      "... progress: 196 (0.4061855670103093)\n",
      "... progress: 200 (0.4144329896907217)\n",
      "... progress: 204 (0.422680412371134)\n",
      "... progress: 208 (0.4309278350515464)\n",
      "... progress: 212 (0.43917525773195876)\n",
      "... progress: 216 (0.44742268041237115)\n",
      "... progress: 220 (0.4556701030927835)\n",
      "... progress: 224 (0.4639175257731959)\n",
      "... progress: 228 (0.47216494845360824)\n",
      "... progress: 232 (0.48041237113402063)\n",
      "... progress: 236 (0.488659793814433)\n",
      "... progress: 240 (0.49690721649484537)\n",
      "... progress: 244 (0.5051546391752577)\n",
      "... progress: 248 (0.51340206185567)\n",
      "... progress: 252 (0.5216494845360825)\n",
      "... progress: 256 (0.5298969072164949)\n",
      "... progress: 260 (0.5381443298969072)\n",
      "... progress: 264 (0.5463917525773195)\n",
      "... progress: 268 (0.554639175257732)\n",
      "... progress: 272 (0.5628865979381443)\n",
      "... progress: 276 (0.5711340206185567)\n",
      "... progress: 280 (0.5793814432989691)\n",
      "... progress: 284 (0.5876288659793815)\n",
      "... progress: 288 (0.5958762886597938)\n",
      "... progress: 292 (0.6041237113402061)\n",
      "... progress: 296 (0.6123711340206186)\n",
      "... progress: 300 (0.6206185567010309)\n",
      "... progress: 304 (0.6288659793814433)\n",
      "... progress: 308 (0.6371134020618556)\n",
      "... progress: 312 (0.6453608247422681)\n",
      "... progress: 316 (0.6536082474226804)\n",
      "... progress: 320 (0.6618556701030928)\n",
      "... progress: 324 (0.6701030927835051)\n",
      "... progress: 328 (0.6783505154639176)\n",
      "... progress: 332 (0.6865979381443299)\n",
      "... progress: 336 (0.6948453608247422)\n",
      "... progress: 340 (0.7030927835051546)\n",
      "... progress: 344 (0.711340206185567)\n",
      "... progress: 348 (0.7195876288659794)\n",
      "... progress: 352 (0.7278350515463917)\n",
      "... progress: 356 (0.7360824742268042)\n",
      "... progress: 360 (0.7443298969072165)\n",
      "... progress: 364 (0.7525773195876289)\n",
      "... progress: 368 (0.7608247422680412)\n",
      "... progress: 372 (0.7690721649484537)\n",
      "... progress: 376 (0.777319587628866)\n",
      "... progress: 380 (0.7855670103092783)\n",
      "... progress: 384 (0.7938144329896907)\n",
      "... progress: 388 (0.8020618556701031)\n",
      "... progress: 392 (0.8103092783505155)\n",
      "... progress: 396 (0.8185567010309278)\n",
      "... progress: 400 (0.8268041237113402)\n",
      "... progress: 404 (0.8350515463917526)\n",
      "... progress: 408 (0.843298969072165)\n",
      "... progress: 412 (0.8515463917525773)\n",
      "... progress: 416 (0.8597938144329897)\n",
      "... progress: 420 (0.8680412371134021)\n",
      "... progress: 424 (0.8762886597938144)\n",
      "... progress: 428 (0.8845360824742268)\n",
      "... progress: 432 (0.8927835051546392)\n",
      "... progress: 436 (0.9010309278350516)\n",
      "... progress: 440 (0.9092783505154639)\n",
      "... progress: 444 (0.9175257731958762)\n",
      "... progress: 448 (0.9257731958762887)\n",
      "... progress: 452 (0.934020618556701)\n",
      "... progress: 456 (0.9422680412371134)\n",
      "... progress: 460 (0.9505154639175257)\n",
      "... progress: 464 (0.9587628865979382)\n",
      "... progress: 468 (0.9670103092783505)\n",
      "... progress: 472 (0.9752577319587629)\n",
      "... progress: 476 (0.9835051546391752)\n",
      "... progress: 480 (0.9917525773195877)\n",
      "... progress: 484 (1.0)\n",
      "----- split #0 (size: 485) -----\n",
      " [[  0   0]\n",
      " [339 146]]\n",
      "----- acc: 0.30103092783505153 -----\n",
      "====================================\n",
      " Final confusion matrix:\n",
      " [[  0.   0.]\n",
      " [339. 146.]]\n",
      " Final accuracy: 0.30103092783505153\n",
      "====================================\n",
      "CPU times: user 13min 51s, sys: 1.7 s, total: 13min 53s\n",
      "Wall time: 52.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gtc_evaluate_with_checkpoints(gpt3_mg_pd, solaiman_detector, checkpoint=1000, score_func=safe_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35f28ca0",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true
   },
   "source": [
    "### Experiments on GROVER dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9edc6617",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_data_path = \"./data/test/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "f9a2fd9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_hw_filename = 'Grover.human.jsonl'\n",
    "grover_ds_hw = pd.read_json(os.path.join(grover_data_path, grover_ds_filename), lines = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67ae7d65-a3cc-42be-8bc6-3920fad87b51",
   "metadata": {},
   "source": [
    "#### Mega - p0.94"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "09329aac",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_ds_filename = 'Grover-mega-p0.94.test.machine.jsonl'\n",
    "grover_ds_mg = pd.read_json(os.path.join(grover_data_path, grover_ds_filename), lines = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "2fcd3b00",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>When I first started looking at oil assets, I ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>The Kingston and St Andrew Municipal Corporati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>news, local-news,\\nTwo protests against an inv...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>Drink-driving charges brought against Danny Dr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>Called by the Mirror after today's strong new ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>24988</td>\n",
       "      <td>This content was published on April 17, 2019 6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>24990</td>\n",
       "      <td>At times like these, it’s hard to know where t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>24994</td>\n",
       "      <td>The L train is set to shut down for 10 weeks s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>24996</td>\n",
       "      <td>President Donald Trump said on Monday he had o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>24999</td>\n",
       "      <td> Bentonville junior pitcher Jon Carter has be...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text\n",
       "0         1  When I first started looking at oil assets, I ...\n",
       "1         4  The Kingston and St Andrew Municipal Corporati...\n",
       "2         5  news, local-news,\\nTwo protests against an inv...\n",
       "3         7  Drink-driving charges brought against Danny Dr...\n",
       "4        13  Called by the Mirror after today's strong new ...\n",
       "...     ...                                                ...\n",
       "9995  24988  This content was published on April 17, 2019 6...\n",
       "9996  24990  At times like these, it’s hard to know where t...\n",
       "9997  24994  The L train is set to shut down for 10 weeks s...\n",
       "9998  24996  President Donald Trump said on Monday he had o...\n",
       "9999  24999   Bentonville junior pitcher Jon Carter has be...\n",
       "\n",
       "[10000 rows x 2 columns]"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_ds_mg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "18db7aa3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>When I first started looking at oil assets, I ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4</td>\n",
       "      <td>The Kingston and St Andrew Municipal Corporati...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>5</td>\n",
       "      <td>news, local-news,\\nTwo protests against an inv...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>Drink-driving charges brought against Danny Dr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>Called by the Mirror after today's strong new ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>24988</td>\n",
       "      <td>This content was published on April 17, 2019 6...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>24990</td>\n",
       "      <td>At times like these, it’s hard to know where t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>24994</td>\n",
       "      <td>The L train is set to shut down for 10 weeks s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>24996</td>\n",
       "      <td>President Donald Trump said on Monday he had o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>24999</td>\n",
       "      <td> Bentonville junior pitcher Jon Carter has be...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                               text\n",
       "0         1  When I first started looking at oil assets, I ...\n",
       "1         4  The Kingston and St Andrew Municipal Corporati...\n",
       "2         5  news, local-news,\\nTwo protests against an inv...\n",
       "3         7  Drink-driving charges brought against Danny Dr...\n",
       "4        13  Called by the Mirror after today's strong new ...\n",
       "...     ...                                                ...\n",
       "9995  24988  This content was published on April 17, 2019 6...\n",
       "9996  24990  At times like these, it’s hard to know where t...\n",
       "9997  24994  The L train is set to shut down for 10 weeks s...\n",
       "9998  24996  President Donald Trump said on Monday he had o...\n",
       "9999  24999   Bentonville junior pitcher Jon Carter has be...\n",
       "\n",
       "[10000 rows x 2 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_ds_hw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "c21709f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "grover_mg_pd = create_dataset(grover_ds_mg[\"text\"], is_machine_generated=True)\n",
    "grover_hw_pd = create_dataset(grover_ds_hw[\"text\"], is_machine_generated=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "fee8659c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>When I first started looking at oil assets, I ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>The Kingston and St Andrew Municipal Corporati...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>news, local-news,\\nTwo protests against an inv...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Drink-driving charges brought against Danny Dr...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Called by the Mirror after today's strong new ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>This content was published on April 17, 2019 6...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>At times like these, it’s hard to know where t...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>The L train is set to shut down for 10 weeks s...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>President Donald Trump said on Monday he had o...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td> Bentonville junior pitcher Jon Carter has be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                      X  y\n",
       "0     When I first started looking at oil assets, I ...  0\n",
       "1     The Kingston and St Andrew Municipal Corporati...  0\n",
       "2     news, local-news,\\nTwo protests against an inv...  0\n",
       "3     Drink-driving charges brought against Danny Dr...  0\n",
       "4     Called by the Mirror after today's strong new ...  0\n",
       "...                                                 ... ..\n",
       "9995  This content was published on April 17, 2019 6...  0\n",
       "9996  At times like these, it’s hard to know where t...  0\n",
       "9997  The L train is set to shut down for 10 weeks s...  0\n",
       "9998  President Donald Trump said on Monday he had o...  0\n",
       "9999   Bentonville junior pitcher Jon Carter has be...  0\n",
       "\n",
       "[10000 rows x 2 columns]"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grover_hw_pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "1e714128",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 0 (0.0004)\n",
      "... progress: 25 (0.0104)\n",
      "... progress: 50 (0.0204)\n",
      "... progress: 75 (0.0304)\n",
      "... progress: 100 (0.0404)\n",
      "... progress: 125 (0.0504)\n",
      "... progress: 150 (0.0604)\n",
      "... progress: 175 (0.0704)\n",
      "... progress: 200 (0.0804)\n",
      "... progress: 225 (0.0904)\n",
      "... progress: 250 (0.1004)\n",
      "... progress: 275 (0.1104)\n",
      "... progress: 300 (0.1204)\n",
      "... progress: 325 (0.1304)\n",
      "... progress: 350 (0.1404)\n",
      "... progress: 375 (0.1504)\n",
      "... progress: 400 (0.1604)\n",
      "... progress: 425 (0.1704)\n",
      "... progress: 450 (0.1804)\n",
      "... progress: 475 (0.1904)\n",
      "... progress: 500 (0.2004)\n",
      "... progress: 525 (0.2104)\n",
      "... progress: 550 (0.2204)\n",
      "... progress: 575 (0.2304)\n",
      "... progress: 600 (0.2404)\n",
      "... progress: 625 (0.2504)\n",
      "... progress: 650 (0.2604)\n",
      "... progress: 675 (0.2704)\n",
      "... progress: 700 (0.2804)\n",
      "... progress: 725 (0.2904)\n",
      "... progress: 750 (0.3004)\n",
      "... progress: 775 (0.3104)\n",
      "... progress: 800 (0.3204)\n",
      "... progress: 825 (0.3304)\n",
      "... progress: 850 (0.3404)\n",
      "... progress: 875 (0.3504)\n",
      "... progress: 900 (0.3604)\n",
      "... progress: 925 (0.3704)\n",
      "... progress: 950 (0.3804)\n",
      "... progress: 975 (0.3904)\n",
      "... progress: 1000 (0.4004)\n",
      "... progress: 1025 (0.4104)\n",
      "... progress: 1050 (0.4204)\n",
      "... progress: 1075 (0.4304)\n",
      "... progress: 1100 (0.4404)\n",
      "... progress: 1125 (0.4504)\n",
      "... progress: 1150 (0.4604)\n",
      "... progress: 1175 (0.4704)\n",
      "... progress: 1200 (0.4804)\n",
      "... progress: 1225 (0.4904)\n",
      "... progress: 1250 (0.5004)\n",
      "... progress: 1275 (0.5104)\n",
      "... progress: 1300 (0.5204)\n",
      "... progress: 1325 (0.5304)\n",
      "... progress: 1350 (0.5404)\n",
      "... progress: 1375 (0.5504)\n",
      "... progress: 1400 (0.5604)\n",
      "... progress: 1425 (0.5704)\n",
      "... progress: 1450 (0.5804)\n",
      "... progress: 1475 (0.5904)\n",
      "... progress: 1500 (0.6004)\n",
      "... progress: 1525 (0.6104)\n",
      "... progress: 1550 (0.6204)\n",
      "... progress: 1575 (0.6304)\n",
      "... progress: 1600 (0.6404)\n",
      "... progress: 1625 (0.6504)\n",
      "... progress: 1650 (0.6604)\n",
      "... progress: 1675 (0.6704)\n",
      "... progress: 1700 (0.6804)\n",
      "... progress: 1725 (0.6904)\n",
      "... progress: 1750 (0.7004)\n",
      "... progress: 1775 (0.7104)\n",
      "... progress: 1800 (0.7204)\n",
      "... progress: 1825 (0.7304)\n",
      "... progress: 1850 (0.7404)\n",
      "... progress: 1875 (0.7504)\n",
      "... progress: 1900 (0.7604)\n",
      "... progress: 1925 (0.7704)\n",
      "... progress: 1950 (0.7804)\n",
      "... progress: 1975 (0.7904)\n",
      "... progress: 2000 (0.8004)\n",
      "... progress: 2025 (0.8104)\n",
      "... progress: 2050 (0.8204)\n",
      "... progress: 2075 (0.8304)\n",
      "... progress: 2100 (0.8404)\n",
      "... progress: 2125 (0.8504)\n",
      "... progress: 2150 (0.8604)\n",
      "... progress: 2175 (0.8704)\n",
      "... progress: 2200 (0.8804)\n",
      "... progress: 2225 (0.8904)\n",
      "... progress: 2250 (0.9004)\n",
      "... progress: 2275 (0.9104)\n",
      "... progress: 2300 (0.9204)\n",
      "... progress: 2325 (0.9304)\n",
      "... progress: 2350 (0.9404)\n",
      "... progress: 2375 (0.9504)\n",
      "... progress: 2400 (0.9604)\n",
      "... progress: 2425 (0.9704)\n",
      "... progress: 2450 (0.9804)\n",
      "... progress: 2475 (0.9904)\n",
      "----- split #0 (size: 2500) -----\n",
      " [[   0    0]\n",
      " [ 961 1539]]\n",
      "----- acc: 0.6156 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 2500 (0.0004)\n",
      "... progress: 2525 (0.0104)\n",
      "... progress: 2550 (0.0204)\n",
      "... progress: 2575 (0.0304)\n",
      "... progress: 2600 (0.0404)\n",
      "... progress: 2625 (0.0504)\n",
      "... progress: 2650 (0.0604)\n",
      "... progress: 2675 (0.0704)\n",
      "... progress: 2700 (0.0804)\n",
      "... progress: 2725 (0.0904)\n",
      "... progress: 2750 (0.1004)\n",
      "... progress: 2775 (0.1104)\n",
      "... progress: 2800 (0.1204)\n",
      "... progress: 2825 (0.1304)\n",
      "... progress: 2850 (0.1404)\n",
      "... progress: 2875 (0.1504)\n",
      "... progress: 2900 (0.1604)\n",
      "... progress: 2925 (0.1704)\n",
      "... progress: 2950 (0.1804)\n",
      "... progress: 2975 (0.1904)\n",
      "... progress: 3000 (0.2004)\n",
      "... progress: 3025 (0.2104)\n",
      "... progress: 3050 (0.2204)\n",
      "... progress: 3075 (0.2304)\n",
      "... progress: 3100 (0.2404)\n",
      "... progress: 3125 (0.2504)\n",
      "... progress: 3150 (0.2604)\n",
      "... progress: 3175 (0.2704)\n",
      "... progress: 3200 (0.2804)\n",
      "... progress: 3225 (0.2904)\n",
      "... progress: 3250 (0.3004)\n",
      "... progress: 3275 (0.3104)\n",
      "... progress: 3300 (0.3204)\n",
      "... progress: 3325 (0.3304)\n",
      "... progress: 3350 (0.3404)\n",
      "... progress: 3375 (0.3504)\n",
      "... progress: 3400 (0.3604)\n",
      "... progress: 3425 (0.3704)\n",
      "... progress: 3450 (0.3804)\n",
      "... progress: 3475 (0.3904)\n",
      "... progress: 3500 (0.4004)\n",
      "... progress: 3525 (0.4104)\n",
      "... progress: 3550 (0.4204)\n",
      "... progress: 3575 (0.4304)\n",
      "... progress: 3600 (0.4404)\n",
      "... progress: 3625 (0.4504)\n",
      "... progress: 3650 (0.4604)\n",
      "... progress: 3675 (0.4704)\n",
      "... progress: 3700 (0.4804)\n",
      "... progress: 3725 (0.4904)\n",
      "... progress: 3750 (0.5004)\n",
      "... progress: 3775 (0.5104)\n",
      "... progress: 3800 (0.5204)\n",
      "... progress: 3825 (0.5304)\n",
      "... progress: 3850 (0.5404)\n",
      "... progress: 3875 (0.5504)\n",
      "... progress: 3900 (0.5604)\n",
      "... progress: 3925 (0.5704)\n",
      "... progress: 3950 (0.5804)\n",
      "... progress: 3975 (0.5904)\n",
      "... progress: 4000 (0.6004)\n",
      "... progress: 4025 (0.6104)\n",
      "... progress: 4050 (0.6204)\n",
      "... progress: 4075 (0.6304)\n",
      "... progress: 4100 (0.6404)\n",
      "... progress: 4125 (0.6504)\n",
      "... progress: 4150 (0.6604)\n",
      "... progress: 4175 (0.6704)\n",
      "... progress: 4200 (0.6804)\n",
      "... progress: 4225 (0.6904)\n",
      "... progress: 4250 (0.7004)\n",
      "... progress: 4275 (0.7104)\n",
      "... progress: 4300 (0.7204)\n",
      "... progress: 4325 (0.7304)\n",
      "... progress: 4350 (0.7404)\n",
      "... progress: 4375 (0.7504)\n",
      "... progress: 4400 (0.7604)\n",
      "... progress: 4425 (0.7704)\n",
      "... progress: 4450 (0.7804)\n",
      "... progress: 4475 (0.7904)\n",
      "... progress: 4500 (0.8004)\n",
      "... progress: 4525 (0.8104)\n",
      "... progress: 4550 (0.8204)\n",
      "... progress: 4575 (0.8304)\n",
      "... progress: 4600 (0.8404)\n",
      "... progress: 4625 (0.8504)\n",
      "... progress: 4650 (0.8604)\n",
      "... progress: 4675 (0.8704)\n",
      "... progress: 4700 (0.8804)\n",
      "... progress: 4725 (0.8904)\n",
      "... progress: 4750 (0.9004)\n",
      "... progress: 4775 (0.9104)\n",
      "... progress: 4800 (0.9204)\n",
      "... progress: 4825 (0.9304)\n",
      "... progress: 4850 (0.9404)\n",
      "... progress: 4875 (0.9504)\n",
      "... progress: 4900 (0.9604)\n",
      "... progress: 4925 (0.9704)\n",
      "... progress: 4950 (0.9804)\n",
      "... progress: 4975 (0.9904)\n",
      "----- split #1 (size: 2500) -----\n",
      " [[   0    0]\n",
      " [ 985 1515]]\n",
      "----- acc: 0.606 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 5000 (0.0004)\n",
      "... progress: 5025 (0.0104)\n",
      "... progress: 5050 (0.0204)\n",
      "... progress: 5075 (0.0304)\n",
      "... progress: 5100 (0.0404)\n",
      "... progress: 5125 (0.0504)\n",
      "... progress: 5150 (0.0604)\n",
      "... progress: 5175 (0.0704)\n",
      "... progress: 5200 (0.0804)\n",
      "... progress: 5225 (0.0904)\n",
      "... progress: 5250 (0.1004)\n",
      "... progress: 5275 (0.1104)\n",
      "... progress: 5300 (0.1204)\n",
      "... progress: 5325 (0.1304)\n",
      "... progress: 5350 (0.1404)\n",
      "... progress: 5375 (0.1504)\n",
      "... progress: 5400 (0.1604)\n",
      "... progress: 5425 (0.1704)\n",
      "... progress: 5450 (0.1804)\n",
      "... progress: 5475 (0.1904)\n",
      "... progress: 5500 (0.2004)\n",
      "... progress: 5525 (0.2104)\n",
      "... progress: 5550 (0.2204)\n",
      "... progress: 5575 (0.2304)\n",
      "... progress: 5600 (0.2404)\n",
      "... progress: 5625 (0.2504)\n",
      "... progress: 5650 (0.2604)\n",
      "... progress: 5675 (0.2704)\n",
      "... progress: 5700 (0.2804)\n",
      "... progress: 5725 (0.2904)\n",
      "... progress: 5750 (0.3004)\n",
      "... progress: 5775 (0.3104)\n",
      "... progress: 5800 (0.3204)\n",
      "... progress: 5825 (0.3304)\n",
      "... progress: 5850 (0.3404)\n",
      "... progress: 5875 (0.3504)\n",
      "... progress: 5900 (0.3604)\n",
      "... progress: 5925 (0.3704)\n",
      "... progress: 5950 (0.3804)\n",
      "... progress: 5975 (0.3904)\n",
      "... progress: 6000 (0.4004)\n",
      "... progress: 6025 (0.4104)\n",
      "... progress: 6050 (0.4204)\n",
      "... progress: 6075 (0.4304)\n",
      "... progress: 6100 (0.4404)\n",
      "... progress: 6125 (0.4504)\n",
      "... progress: 6150 (0.4604)\n",
      "... progress: 6175 (0.4704)\n",
      "... progress: 6200 (0.4804)\n",
      "... progress: 6225 (0.4904)\n",
      "... progress: 6250 (0.5004)\n",
      "... progress: 6275 (0.5104)\n",
      "... progress: 6300 (0.5204)\n",
      "... progress: 6325 (0.5304)\n",
      "... progress: 6350 (0.5404)\n",
      "... progress: 6375 (0.5504)\n",
      "... progress: 6400 (0.5604)\n",
      "... progress: 6425 (0.5704)\n",
      "... progress: 6450 (0.5804)\n",
      "... progress: 6475 (0.5904)\n",
      "... progress: 6500 (0.6004)\n",
      "... progress: 6525 (0.6104)\n",
      "... progress: 6550 (0.6204)\n",
      "... progress: 6575 (0.6304)\n",
      "... progress: 6600 (0.6404)\n",
      "... progress: 6625 (0.6504)\n",
      "... progress: 6650 (0.6604)\n",
      "... progress: 6675 (0.6704)\n",
      "... progress: 6700 (0.6804)\n",
      "... progress: 6725 (0.6904)\n",
      "... progress: 6750 (0.7004)\n",
      "... progress: 6775 (0.7104)\n",
      "... progress: 6800 (0.7204)\n",
      "... progress: 6825 (0.7304)\n",
      "... progress: 6850 (0.7404)\n",
      "... progress: 6875 (0.7504)\n",
      "... progress: 6900 (0.7604)\n",
      "... progress: 6925 (0.7704)\n",
      "... progress: 6950 (0.7804)\n",
      "... progress: 6975 (0.7904)\n",
      "... progress: 7000 (0.8004)\n",
      "... progress: 7025 (0.8104)\n",
      "... progress: 7050 (0.8204)\n",
      "... progress: 7075 (0.8304)\n",
      "... progress: 7100 (0.8404)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... progress: 7125 (0.8504)\n",
      "... progress: 7150 (0.8604)\n",
      "... progress: 7175 (0.8704)\n",
      "... progress: 7200 (0.8804)\n",
      "... progress: 7225 (0.8904)\n",
      "... progress: 7250 (0.9004)\n",
      "... progress: 7275 (0.9104)\n",
      "... progress: 7300 (0.9204)\n",
      "... progress: 7325 (0.9304)\n",
      "... progress: 7350 (0.9404)\n",
      "... progress: 7375 (0.9504)\n",
      "... progress: 7400 (0.9604)\n",
      "... progress: 7425 (0.9704)\n",
      "... progress: 7450 (0.9804)\n",
      "... progress: 7475 (0.9904)\n",
      "----- split #2 (size: 2500) -----\n",
      " [[   0    0]\n",
      " [ 977 1523]]\n",
      "----- acc: 0.6092 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 7500 (0.0004)\n",
      "... progress: 7525 (0.0104)\n",
      "... progress: 7550 (0.0204)\n",
      "... progress: 7575 (0.0304)\n",
      "... progress: 7600 (0.0404)\n",
      "... progress: 7625 (0.0504)\n",
      "... progress: 7650 (0.0604)\n",
      "... progress: 7675 (0.0704)\n",
      "... progress: 7700 (0.0804)\n",
      "... progress: 7725 (0.0904)\n",
      "... progress: 7750 (0.1004)\n",
      "... progress: 7775 (0.1104)\n",
      "... progress: 7800 (0.1204)\n",
      "... progress: 7825 (0.1304)\n",
      "... progress: 7850 (0.1404)\n",
      "... progress: 7875 (0.1504)\n",
      "... progress: 7900 (0.1604)\n",
      "... progress: 7925 (0.1704)\n",
      "... progress: 7950 (0.1804)\n",
      "... progress: 7975 (0.1904)\n",
      "... progress: 8000 (0.2004)\n",
      "... progress: 8025 (0.2104)\n",
      "... progress: 8050 (0.2204)\n",
      "... progress: 8075 (0.2304)\n",
      "... progress: 8100 (0.2404)\n",
      "... progress: 8125 (0.2504)\n",
      "... progress: 8150 (0.2604)\n",
      "... progress: 8175 (0.2704)\n",
      "... progress: 8200 (0.2804)\n",
      "... progress: 8225 (0.2904)\n",
      "... progress: 8250 (0.3004)\n",
      "... progress: 8275 (0.3104)\n",
      "... progress: 8300 (0.3204)\n",
      "... progress: 8325 (0.3304)\n",
      "... progress: 8350 (0.3404)\n",
      "... progress: 8375 (0.3504)\n",
      "... progress: 8400 (0.3604)\n",
      "... progress: 8425 (0.3704)\n",
      "... progress: 8450 (0.3804)\n",
      "... progress: 8475 (0.3904)\n",
      "... progress: 8500 (0.4004)\n",
      "... progress: 8525 (0.4104)\n",
      "... progress: 8550 (0.4204)\n",
      "... progress: 8575 (0.4304)\n",
      "... progress: 8600 (0.4404)\n",
      "... progress: 8625 (0.4504)\n",
      "... progress: 8650 (0.4604)\n",
      "... progress: 8675 (0.4704)\n",
      "... progress: 8700 (0.4804)\n",
      "... progress: 8725 (0.4904)\n",
      "... progress: 8750 (0.5004)\n",
      "... progress: 8775 (0.5104)\n",
      "... progress: 8800 (0.5204)\n",
      "... progress: 8825 (0.5304)\n",
      "... progress: 8850 (0.5404)\n",
      "... progress: 8875 (0.5504)\n",
      "... progress: 8900 (0.5604)\n",
      "... progress: 8925 (0.5704)\n",
      "... progress: 8950 (0.5804)\n",
      "... progress: 8975 (0.5904)\n",
      "... progress: 9000 (0.6004)\n",
      "... progress: 9025 (0.6104)\n",
      "... progress: 9050 (0.6204)\n",
      "... progress: 9075 (0.6304)\n",
      "... progress: 9100 (0.6404)\n",
      "... progress: 9125 (0.6504)\n",
      "... progress: 9150 (0.6604)\n",
      "... progress: 9175 (0.6704)\n",
      "... progress: 9200 (0.6804)\n",
      "... progress: 9225 (0.6904)\n",
      "... progress: 9250 (0.7004)\n",
      "... progress: 9275 (0.7104)\n",
      "... progress: 9300 (0.7204)\n",
      "... progress: 9325 (0.7304)\n",
      "... progress: 9350 (0.7404)\n",
      "... progress: 9375 (0.7504)\n",
      "... progress: 9400 (0.7604)\n",
      "... progress: 9425 (0.7704)\n",
      "... progress: 9450 (0.7804)\n",
      "... progress: 9475 (0.7904)\n",
      "... progress: 9500 (0.8004)\n",
      "... progress: 9525 (0.8104)\n",
      "... progress: 9550 (0.8204)\n",
      "... progress: 9575 (0.8304)\n",
      "... progress: 9600 (0.8404)\n",
      "... progress: 9625 (0.8504)\n",
      "... progress: 9650 (0.8604)\n",
      "... progress: 9675 (0.8704)\n",
      "... progress: 9700 (0.8804)\n",
      "... progress: 9725 (0.8904)\n",
      "... progress: 9750 (0.9004)\n",
      "... progress: 9775 (0.9104)\n",
      "... progress: 9800 (0.9204)\n",
      "... progress: 9825 (0.9304)\n",
      "... progress: 9850 (0.9404)\n",
      "... progress: 9875 (0.9504)\n",
      "... progress: 9900 (0.9604)\n",
      "... progress: 9925 (0.9704)\n",
      "... progress: 9950 (0.9804)\n",
      "... progress: 9975 (0.9904)\n",
      "----- split #3 (size: 2500) -----\n",
      " [[   0    0]\n",
      " [ 985 1515]]\n",
      "----- acc: 0.606 -----\n",
      "====================================\n",
      " Final confusion matrix:\n",
      " [[   0.    0.]\n",
      " [3908. 6092.]]\n",
      " Final accuracy: 0.6092\n",
      "====================================\n",
      "CPU times: user 3h 48min 23s, sys: 34.4 s, total: 3h 48min 57s\n",
      "Wall time: 14min 18s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gtc_evaluate_with_checkpoints(grover_mg_pd, solaiman_detector, checkpoint=2500, score_func=safe_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "09482509",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 0 (0.0004)\n",
      "... progress: 25 (0.0104)\n",
      "... progress: 50 (0.0204)\n",
      "... progress: 75 (0.0304)\n",
      "... progress: 100 (0.0404)\n",
      "... progress: 125 (0.0504)\n",
      "... progress: 150 (0.0604)\n",
      "... progress: 175 (0.0704)\n",
      "... progress: 200 (0.0804)\n",
      "... progress: 225 (0.0904)\n",
      "... progress: 250 (0.1004)\n",
      "... progress: 275 (0.1104)\n",
      "... progress: 300 (0.1204)\n",
      "... progress: 325 (0.1304)\n",
      "... progress: 350 (0.1404)\n",
      "... progress: 375 (0.1504)\n",
      "... progress: 400 (0.1604)\n",
      "... progress: 425 (0.1704)\n",
      "... progress: 450 (0.1804)\n",
      "... progress: 475 (0.1904)\n",
      "... progress: 500 (0.2004)\n",
      "... progress: 525 (0.2104)\n",
      "... progress: 550 (0.2204)\n",
      "... progress: 575 (0.2304)\n",
      "... progress: 600 (0.2404)\n",
      "... progress: 625 (0.2504)\n",
      "... progress: 650 (0.2604)\n",
      "... progress: 675 (0.2704)\n",
      "... progress: 700 (0.2804)\n",
      "... progress: 725 (0.2904)\n",
      "... progress: 750 (0.3004)\n",
      "... progress: 775 (0.3104)\n",
      "... progress: 800 (0.3204)\n",
      "... progress: 825 (0.3304)\n",
      "... progress: 850 (0.3404)\n",
      "... progress: 875 (0.3504)\n",
      "... progress: 900 (0.3604)\n",
      "... progress: 925 (0.3704)\n",
      "... progress: 950 (0.3804)\n",
      "... progress: 975 (0.3904)\n",
      "... progress: 1000 (0.4004)\n",
      "... progress: 1025 (0.4104)\n",
      "... progress: 1050 (0.4204)\n",
      "... progress: 1075 (0.4304)\n",
      "... progress: 1100 (0.4404)\n",
      "... progress: 1125 (0.4504)\n",
      "... progress: 1150 (0.4604)\n",
      "... progress: 1175 (0.4704)\n",
      "... progress: 1200 (0.4804)\n",
      "... progress: 1225 (0.4904)\n",
      "... progress: 1250 (0.5004)\n",
      "... progress: 1275 (0.5104)\n",
      "... progress: 1300 (0.5204)\n",
      "... progress: 1325 (0.5304)\n",
      "... progress: 1350 (0.5404)\n",
      "... progress: 1375 (0.5504)\n",
      "... progress: 1400 (0.5604)\n",
      "... progress: 1425 (0.5704)\n",
      "... progress: 1450 (0.5804)\n",
      "... progress: 1475 (0.5904)\n",
      "... progress: 1500 (0.6004)\n",
      "... progress: 1525 (0.6104)\n",
      "... progress: 1550 (0.6204)\n",
      "... progress: 1575 (0.6304)\n",
      "... progress: 1600 (0.6404)\n",
      "... progress: 1625 (0.6504)\n",
      "... progress: 1650 (0.6604)\n",
      "... progress: 1675 (0.6704)\n",
      "... progress: 1700 (0.6804)\n",
      "... progress: 1725 (0.6904)\n",
      "... progress: 1750 (0.7004)\n",
      "... progress: 1775 (0.7104)\n",
      "... progress: 1800 (0.7204)\n",
      "... progress: 1825 (0.7304)\n",
      "... progress: 1850 (0.7404)\n",
      "... progress: 1875 (0.7504)\n",
      "... progress: 1900 (0.7604)\n",
      "... progress: 1925 (0.7704)\n",
      "... progress: 1950 (0.7804)\n",
      "... progress: 1975 (0.7904)\n",
      "... progress: 2000 (0.8004)\n",
      "... progress: 2025 (0.8104)\n",
      "... progress: 2050 (0.8204)\n",
      "... progress: 2075 (0.8304)\n",
      "... progress: 2100 (0.8404)\n",
      "... progress: 2125 (0.8504)\n",
      "... progress: 2150 (0.8604)\n",
      "... progress: 2175 (0.8704)\n",
      "... progress: 2200 (0.8804)\n",
      "... progress: 2225 (0.8904)\n",
      "... progress: 2250 (0.9004)\n",
      "... progress: 2275 (0.9104)\n",
      "... progress: 2300 (0.9204)\n",
      "... progress: 2325 (0.9304)\n",
      "... progress: 2350 (0.9404)\n",
      "... progress: 2375 (0.9504)\n",
      "... progress: 2400 (0.9604)\n",
      "... progress: 2425 (0.9704)\n",
      "... progress: 2450 (0.9804)\n",
      "... progress: 2475 (0.9904)\n",
      "----- split #0 (size: 2500) -----\n",
      " [[2435   65]\n",
      " [   0    0]]\n",
      "----- acc: 0.974 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 2500 (0.0004)\n",
      "... progress: 2525 (0.0104)\n",
      "... progress: 2550 (0.0204)\n",
      "... progress: 2575 (0.0304)\n",
      "... progress: 2600 (0.0404)\n",
      "... progress: 2625 (0.0504)\n",
      "... progress: 2650 (0.0604)\n",
      "... progress: 2675 (0.0704)\n",
      "... progress: 2700 (0.0804)\n",
      "... progress: 2725 (0.0904)\n",
      "... progress: 2750 (0.1004)\n",
      "... progress: 2775 (0.1104)\n",
      "... progress: 2800 (0.1204)\n",
      "... progress: 2825 (0.1304)\n",
      "... progress: 2850 (0.1404)\n",
      "... progress: 2875 (0.1504)\n",
      "... progress: 2900 (0.1604)\n",
      "... progress: 2925 (0.1704)\n",
      "... progress: 2950 (0.1804)\n",
      "... progress: 2975 (0.1904)\n",
      "... progress: 3000 (0.2004)\n",
      "... progress: 3025 (0.2104)\n",
      "... progress: 3050 (0.2204)\n",
      "... progress: 3075 (0.2304)\n",
      "... progress: 3100 (0.2404)\n",
      "... progress: 3125 (0.2504)\n",
      "... progress: 3150 (0.2604)\n",
      "... progress: 3175 (0.2704)\n",
      "... progress: 3200 (0.2804)\n",
      "... progress: 3225 (0.2904)\n",
      "... progress: 3250 (0.3004)\n",
      "... progress: 3275 (0.3104)\n",
      "... progress: 3300 (0.3204)\n",
      "... progress: 3325 (0.3304)\n",
      "... progress: 3350 (0.3404)\n",
      "... progress: 3375 (0.3504)\n",
      "... progress: 3400 (0.3604)\n",
      "... progress: 3425 (0.3704)\n",
      "... progress: 3450 (0.3804)\n",
      "... progress: 3475 (0.3904)\n",
      "... progress: 3500 (0.4004)\n",
      "... progress: 3525 (0.4104)\n",
      "... progress: 3550 (0.4204)\n",
      "... progress: 3575 (0.4304)\n",
      "... progress: 3600 (0.4404)\n",
      "... progress: 3625 (0.4504)\n",
      "... progress: 3650 (0.4604)\n",
      "... progress: 3675 (0.4704)\n",
      "... progress: 3700 (0.4804)\n",
      "... progress: 3725 (0.4904)\n",
      "... progress: 3750 (0.5004)\n",
      "... progress: 3775 (0.5104)\n",
      "... progress: 3800 (0.5204)\n",
      "... progress: 3825 (0.5304)\n",
      "... progress: 3850 (0.5404)\n",
      "... progress: 3875 (0.5504)\n",
      "... progress: 3900 (0.5604)\n",
      "... progress: 3925 (0.5704)\n",
      "... progress: 3950 (0.5804)\n",
      "... progress: 3975 (0.5904)\n",
      "... progress: 4000 (0.6004)\n",
      "... progress: 4025 (0.6104)\n",
      "... progress: 4050 (0.6204)\n",
      "... progress: 4075 (0.6304)\n",
      "... progress: 4100 (0.6404)\n",
      "... progress: 4125 (0.6504)\n",
      "... progress: 4150 (0.6604)\n",
      "... progress: 4175 (0.6704)\n",
      "... progress: 4200 (0.6804)\n",
      "... progress: 4225 (0.6904)\n",
      "... progress: 4250 (0.7004)\n",
      "... progress: 4275 (0.7104)\n",
      "... progress: 4300 (0.7204)\n",
      "... progress: 4325 (0.7304)\n",
      "... progress: 4350 (0.7404)\n",
      "... progress: 4375 (0.7504)\n",
      "... progress: 4400 (0.7604)\n",
      "... progress: 4425 (0.7704)\n",
      "... progress: 4450 (0.7804)\n",
      "... progress: 4475 (0.7904)\n",
      "... progress: 4500 (0.8004)\n",
      "... progress: 4525 (0.8104)\n",
      "... progress: 4550 (0.8204)\n",
      "... progress: 4575 (0.8304)\n",
      "... progress: 4600 (0.8404)\n",
      "... progress: 4625 (0.8504)\n",
      "... progress: 4650 (0.8604)\n",
      "... progress: 4675 (0.8704)\n",
      "... progress: 4700 (0.8804)\n",
      "... progress: 4725 (0.8904)\n",
      "... progress: 4750 (0.9004)\n",
      "... progress: 4775 (0.9104)\n",
      "... progress: 4800 (0.9204)\n",
      "... progress: 4825 (0.9304)\n",
      "... progress: 4850 (0.9404)\n",
      "... progress: 4875 (0.9504)\n",
      "... progress: 4900 (0.9604)\n",
      "... progress: 4925 (0.9704)\n",
      "... progress: 4950 (0.9804)\n",
      "... progress: 4975 (0.9904)\n",
      "----- split #1 (size: 2500) -----\n",
      " [[2427   73]\n",
      " [   0    0]]\n",
      "----- acc: 0.9708 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 5000 (0.0004)\n",
      "... progress: 5025 (0.0104)\n",
      "... progress: 5050 (0.0204)\n",
      "... progress: 5075 (0.0304)\n",
      "... progress: 5100 (0.0404)\n",
      "... progress: 5125 (0.0504)\n",
      "... progress: 5150 (0.0604)\n",
      "... progress: 5175 (0.0704)\n",
      "... progress: 5200 (0.0804)\n",
      "... progress: 5225 (0.0904)\n",
      "... progress: 5250 (0.1004)\n",
      "... progress: 5275 (0.1104)\n",
      "... progress: 5300 (0.1204)\n",
      "... progress: 5325 (0.1304)\n",
      "... progress: 5350 (0.1404)\n",
      "... progress: 5375 (0.1504)\n",
      "... progress: 5400 (0.1604)\n",
      "... progress: 5425 (0.1704)\n",
      "... progress: 5450 (0.1804)\n",
      "... progress: 5475 (0.1904)\n",
      "... progress: 5500 (0.2004)\n",
      "... progress: 5525 (0.2104)\n",
      "... progress: 5550 (0.2204)\n",
      "... progress: 5575 (0.2304)\n",
      "... progress: 5600 (0.2404)\n",
      "... progress: 5625 (0.2504)\n",
      "... progress: 5650 (0.2604)\n",
      "... progress: 5675 (0.2704)\n",
      "... progress: 5700 (0.2804)\n",
      "... progress: 5725 (0.2904)\n",
      "... progress: 5750 (0.3004)\n",
      "... progress: 5775 (0.3104)\n",
      "... progress: 5800 (0.3204)\n",
      "... progress: 5825 (0.3304)\n",
      "... progress: 5850 (0.3404)\n",
      "... progress: 5875 (0.3504)\n",
      "... progress: 5900 (0.3604)\n",
      "... progress: 5925 (0.3704)\n",
      "... progress: 5950 (0.3804)\n",
      "... progress: 5975 (0.3904)\n",
      "... progress: 6000 (0.4004)\n",
      "... progress: 6025 (0.4104)\n",
      "... progress: 6050 (0.4204)\n",
      "... progress: 6075 (0.4304)\n",
      "... progress: 6100 (0.4404)\n",
      "... progress: 6125 (0.4504)\n",
      "... progress: 6150 (0.4604)\n",
      "... progress: 6175 (0.4704)\n",
      "... progress: 6200 (0.4804)\n",
      "... progress: 6225 (0.4904)\n",
      "... progress: 6250 (0.5004)\n",
      "... progress: 6275 (0.5104)\n",
      "... progress: 6300 (0.5204)\n",
      "... progress: 6325 (0.5304)\n",
      "... progress: 6350 (0.5404)\n",
      "... progress: 6375 (0.5504)\n",
      "... progress: 6400 (0.5604)\n",
      "... progress: 6425 (0.5704)\n",
      "... progress: 6450 (0.5804)\n",
      "... progress: 6475 (0.5904)\n",
      "... progress: 6500 (0.6004)\n",
      "... progress: 6525 (0.6104)\n",
      "... progress: 6550 (0.6204)\n",
      "... progress: 6575 (0.6304)\n",
      "... progress: 6600 (0.6404)\n",
      "... progress: 6625 (0.6504)\n",
      "... progress: 6650 (0.6604)\n",
      "... progress: 6675 (0.6704)\n",
      "... progress: 6700 (0.6804)\n",
      "... progress: 6725 (0.6904)\n",
      "... progress: 6750 (0.7004)\n",
      "... progress: 6775 (0.7104)\n",
      "... progress: 6800 (0.7204)\n",
      "... progress: 6825 (0.7304)\n",
      "... progress: 6850 (0.7404)\n",
      "... progress: 6875 (0.7504)\n",
      "... progress: 6900 (0.7604)\n",
      "... progress: 6925 (0.7704)\n",
      "... progress: 6950 (0.7804)\n",
      "... progress: 6975 (0.7904)\n",
      "... progress: 7000 (0.8004)\n",
      "... progress: 7025 (0.8104)\n",
      "... progress: 7050 (0.8204)\n",
      "... progress: 7075 (0.8304)\n",
      "... progress: 7100 (0.8404)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... progress: 7125 (0.8504)\n",
      "... progress: 7150 (0.8604)\n",
      "... progress: 7175 (0.8704)\n",
      "... progress: 7200 (0.8804)\n",
      "... progress: 7225 (0.8904)\n",
      "... progress: 7250 (0.9004)\n",
      "... progress: 7275 (0.9104)\n",
      "... progress: 7300 (0.9204)\n",
      "... progress: 7325 (0.9304)\n",
      "... progress: 7350 (0.9404)\n",
      "... progress: 7375 (0.9504)\n",
      "... progress: 7400 (0.9604)\n",
      "... progress: 7425 (0.9704)\n",
      "... progress: 7450 (0.9804)\n",
      "... progress: 7475 (0.9904)\n",
      "----- split #2 (size: 2500) -----\n",
      " [[2415   85]\n",
      " [   0    0]]\n",
      "----- acc: 0.966 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 7500 (0.0004)\n",
      "... progress: 7525 (0.0104)\n",
      "... progress: 7550 (0.0204)\n",
      "... progress: 7575 (0.0304)\n",
      "... progress: 7600 (0.0404)\n",
      "... progress: 7625 (0.0504)\n",
      "... progress: 7650 (0.0604)\n",
      "... progress: 7675 (0.0704)\n",
      "... progress: 7700 (0.0804)\n",
      "... progress: 7725 (0.0904)\n",
      "... progress: 7750 (0.1004)\n",
      "... progress: 7775 (0.1104)\n",
      "... progress: 7800 (0.1204)\n",
      "... progress: 7825 (0.1304)\n",
      "... progress: 7850 (0.1404)\n",
      "... progress: 7875 (0.1504)\n",
      "... progress: 7900 (0.1604)\n",
      "... progress: 7925 (0.1704)\n",
      "... progress: 7950 (0.1804)\n",
      "... progress: 7975 (0.1904)\n",
      "... progress: 8000 (0.2004)\n",
      "... progress: 8025 (0.2104)\n",
      "... progress: 8050 (0.2204)\n",
      "... progress: 8075 (0.2304)\n",
      "... progress: 8100 (0.2404)\n",
      "... progress: 8125 (0.2504)\n",
      "... progress: 8150 (0.2604)\n",
      "... progress: 8175 (0.2704)\n",
      "... progress: 8200 (0.2804)\n",
      "... progress: 8225 (0.2904)\n",
      "... progress: 8250 (0.3004)\n",
      "... progress: 8275 (0.3104)\n",
      "... progress: 8300 (0.3204)\n",
      "... progress: 8325 (0.3304)\n",
      "... progress: 8350 (0.3404)\n",
      "... progress: 8375 (0.3504)\n",
      "... progress: 8400 (0.3604)\n",
      "... progress: 8425 (0.3704)\n",
      "... progress: 8450 (0.3804)\n",
      "... progress: 8475 (0.3904)\n",
      "... progress: 8500 (0.4004)\n",
      "... progress: 8525 (0.4104)\n",
      "... progress: 8550 (0.4204)\n",
      "... progress: 8575 (0.4304)\n",
      "... progress: 8600 (0.4404)\n",
      "... progress: 8625 (0.4504)\n",
      "... progress: 8650 (0.4604)\n",
      "... progress: 8675 (0.4704)\n",
      "... progress: 8700 (0.4804)\n",
      "... progress: 8725 (0.4904)\n",
      "... progress: 8750 (0.5004)\n",
      "... progress: 8775 (0.5104)\n",
      "... progress: 8800 (0.5204)\n",
      "... progress: 8825 (0.5304)\n",
      "... progress: 8850 (0.5404)\n",
      "... progress: 8875 (0.5504)\n",
      "... progress: 8900 (0.5604)\n",
      "... progress: 8925 (0.5704)\n",
      "... progress: 8950 (0.5804)\n",
      "... progress: 8975 (0.5904)\n",
      "... progress: 9000 (0.6004)\n",
      "... progress: 9025 (0.6104)\n",
      "... progress: 9050 (0.6204)\n",
      "... progress: 9075 (0.6304)\n",
      "... progress: 9100 (0.6404)\n",
      "... progress: 9125 (0.6504)\n",
      "... progress: 9150 (0.6604)\n",
      "... progress: 9175 (0.6704)\n",
      "... progress: 9200 (0.6804)\n",
      "... progress: 9225 (0.6904)\n",
      "... progress: 9250 (0.7004)\n",
      "... progress: 9275 (0.7104)\n",
      "... progress: 9300 (0.7204)\n",
      "... progress: 9325 (0.7304)\n",
      "... progress: 9350 (0.7404)\n",
      "... progress: 9375 (0.7504)\n",
      "... progress: 9400 (0.7604)\n",
      "... progress: 9425 (0.7704)\n",
      "... progress: 9450 (0.7804)\n",
      "... progress: 9475 (0.7904)\n",
      "... progress: 9500 (0.8004)\n",
      "... progress: 9525 (0.8104)\n",
      "... progress: 9550 (0.8204)\n",
      "... progress: 9575 (0.8304)\n",
      "... progress: 9600 (0.8404)\n",
      "... progress: 9625 (0.8504)\n",
      "... progress: 9650 (0.8604)\n",
      "... progress: 9675 (0.8704)\n",
      "... progress: 9700 (0.8804)\n",
      "... progress: 9725 (0.8904)\n",
      "... progress: 9750 (0.9004)\n",
      "... progress: 9775 (0.9104)\n",
      "... progress: 9800 (0.9204)\n",
      "... progress: 9825 (0.9304)\n",
      "... progress: 9850 (0.9404)\n",
      "... progress: 9875 (0.9504)\n",
      "... progress: 9900 (0.9604)\n",
      "... progress: 9925 (0.9704)\n",
      "... progress: 9950 (0.9804)\n",
      "... progress: 9975 (0.9904)\n",
      "----- split #3 (size: 2500) -----\n",
      " [[2419   81]\n",
      " [   0    0]]\n",
      "----- acc: 0.9676 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 10000 (0.0004)\n",
      "... progress: 10025 (0.0104)\n",
      "... progress: 10050 (0.0204)\n",
      "... progress: 10075 (0.0304)\n",
      "... progress: 10100 (0.0404)\n",
      "... progress: 10125 (0.0504)\n",
      "... progress: 10150 (0.0604)\n",
      "... progress: 10175 (0.0704)\n",
      "... progress: 10200 (0.0804)\n",
      "... progress: 10225 (0.0904)\n",
      "... progress: 10250 (0.1004)\n",
      "... progress: 10275 (0.1104)\n",
      "... progress: 10300 (0.1204)\n",
      "... progress: 10325 (0.1304)\n",
      "... progress: 10350 (0.1404)\n",
      "... progress: 10375 (0.1504)\n",
      "... progress: 10400 (0.1604)\n",
      "... progress: 10425 (0.1704)\n",
      "... progress: 10450 (0.1804)\n",
      "... progress: 10475 (0.1904)\n",
      "... progress: 10500 (0.2004)\n",
      "... progress: 10525 (0.2104)\n",
      "... progress: 10550 (0.2204)\n",
      "... progress: 10575 (0.2304)\n",
      "... progress: 10600 (0.2404)\n",
      "... progress: 10625 (0.2504)\n",
      "... progress: 10650 (0.2604)\n",
      "... progress: 10675 (0.2704)\n",
      "... progress: 10700 (0.2804)\n",
      "... progress: 10725 (0.2904)\n",
      "... progress: 10750 (0.3004)\n",
      "... progress: 10775 (0.3104)\n",
      "... progress: 10800 (0.3204)\n",
      "... progress: 10825 (0.3304)\n",
      "... progress: 10850 (0.3404)\n",
      "... progress: 10875 (0.3504)\n",
      "... progress: 10900 (0.3604)\n",
      "... progress: 10925 (0.3704)\n",
      "... progress: 10950 (0.3804)\n",
      "... progress: 10975 (0.3904)\n",
      "... progress: 11000 (0.4004)\n",
      "... progress: 11025 (0.4104)\n",
      "... progress: 11050 (0.4204)\n",
      "... progress: 11075 (0.4304)\n",
      "... progress: 11100 (0.4404)\n",
      "... progress: 11125 (0.4504)\n",
      "... progress: 11150 (0.4604)\n",
      "... progress: 11175 (0.4704)\n",
      "... progress: 11200 (0.4804)\n",
      "... progress: 11225 (0.4904)\n",
      "... progress: 11250 (0.5004)\n",
      "... progress: 11275 (0.5104)\n",
      "... progress: 11300 (0.5204)\n",
      "... progress: 11325 (0.5304)\n",
      "... progress: 11350 (0.5404)\n",
      "... progress: 11375 (0.5504)\n",
      "... progress: 11400 (0.5604)\n",
      "... progress: 11425 (0.5704)\n",
      "... progress: 11450 (0.5804)\n",
      "... progress: 11475 (0.5904)\n",
      "... progress: 11500 (0.6004)\n",
      "... progress: 11525 (0.6104)\n",
      "... progress: 11550 (0.6204)\n",
      "... progress: 11575 (0.6304)\n",
      "... progress: 11600 (0.6404)\n",
      "... progress: 11625 (0.6504)\n",
      "... progress: 11650 (0.6604)\n",
      "... progress: 11675 (0.6704)\n",
      "... progress: 11700 (0.6804)\n",
      "... progress: 11725 (0.6904)\n",
      "... progress: 11750 (0.7004)\n",
      "... progress: 11775 (0.7104)\n",
      "... progress: 11800 (0.7204)\n",
      "... progress: 11825 (0.7304)\n",
      "... progress: 11850 (0.7404)\n",
      "... progress: 11875 (0.7504)\n",
      "... progress: 11900 (0.7604)\n",
      "... progress: 11925 (0.7704)\n",
      "... progress: 11950 (0.7804)\n",
      "... progress: 11975 (0.7904)\n",
      "... progress: 12000 (0.8004)\n",
      "... progress: 12025 (0.8104)\n",
      "... progress: 12050 (0.8204)\n",
      "... progress: 12075 (0.8304)\n",
      "... progress: 12100 (0.8404)\n",
      "... progress: 12125 (0.8504)\n",
      "... progress: 12150 (0.8604)\n",
      "... progress: 12175 (0.8704)\n",
      "... progress: 12200 (0.8804)\n",
      "... progress: 12225 (0.8904)\n",
      "... progress: 12250 (0.9004)\n",
      "... progress: 12275 (0.9104)\n",
      "... progress: 12300 (0.9204)\n",
      "... progress: 12325 (0.9304)\n",
      "... progress: 12350 (0.9404)\n",
      "... progress: 12375 (0.9504)\n",
      "... progress: 12400 (0.9604)\n",
      "... progress: 12425 (0.9704)\n",
      "... progress: 12450 (0.9804)\n",
      "... progress: 12475 (0.9904)\n",
      "----- split #4 (size: 2500) -----\n",
      " [[2435   65]\n",
      " [   0    0]]\n",
      "----- acc: 0.974 -----\n",
      "progupd: 25; m(m(100, 25), 1)\n",
      "... progress: 12500 (0.0004)\n",
      "... progress: 12525 (0.0104)\n",
      "... progress: 12550 (0.0204)\n",
      "... progress: 12575 (0.0304)\n",
      "... progress: 12600 (0.0404)\n",
      "... progress: 12625 (0.0504)\n",
      "... progress: 12650 (0.0604)\n",
      "... progress: 12675 (0.0704)\n",
      "... progress: 12700 (0.0804)\n",
      "... progress: 12725 (0.0904)\n",
      "... progress: 12750 (0.1004)\n",
      "... progress: 12775 (0.1104)\n",
      "... progress: 12800 (0.1204)\n",
      "... progress: 12825 (0.1304)\n",
      "... progress: 12850 (0.1404)\n",
      "... progress: 12875 (0.1504)\n",
      "... progress: 12900 (0.1604)\n",
      "... progress: 12925 (0.1704)\n",
      "... progress: 12950 (0.1804)\n",
      "... progress: 12975 (0.1904)\n",
      "... progress: 13000 (0.2004)\n",
      "... progress: 13025 (0.2104)\n",
      "... progress: 13050 (0.2204)\n",
      "... progress: 13075 (0.2304)\n",
      "... progress: 13100 (0.2404)\n",
      "... progress: 13125 (0.2504)\n",
      "... progress: 13150 (0.2604)\n",
      "... progress: 13175 (0.2704)\n",
      "... progress: 13200 (0.2804)\n",
      "... progress: 13225 (0.2904)\n",
      "... progress: 13250 (0.3004)\n",
      "... progress: 13275 (0.3104)\n",
      "... progress: 13300 (0.3204)\n",
      "... progress: 13325 (0.3304)\n",
      "... progress: 13350 (0.3404)\n",
      "... progress: 13375 (0.3504)\n",
      "... progress: 13400 (0.3604)\n",
      "... progress: 13425 (0.3704)\n",
      "... progress: 13450 (0.3804)\n",
      "... progress: 13475 (0.3904)\n",
      "... progress: 13500 (0.4004)\n",
      "... progress: 13525 (0.4104)\n",
      "... progress: 13550 (0.4204)\n",
      "... progress: 13575 (0.4304)\n",
      "... progress: 13600 (0.4404)\n",
      "... progress: 13625 (0.4504)\n",
      "... progress: 13650 (0.4604)\n",
      "... progress: 13675 (0.4704)\n",
      "... progress: 13700 (0.4804)\n",
      "... progress: 13725 (0.4904)\n",
      "... progress: 13750 (0.5004)\n",
      "... progress: 13775 (0.5104)\n",
      "... progress: 13800 (0.5204)\n",
      "... progress: 13825 (0.5304)\n",
      "... progress: 13850 (0.5404)\n",
      "... progress: 13875 (0.5504)\n",
      "... progress: 13900 (0.5604)\n",
      "... progress: 13925 (0.5704)\n",
      "... progress: 13950 (0.5804)\n",
      "... progress: 13975 (0.5904)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "... progress: 14000 (0.6004)\n",
      "... progress: 14025 (0.6104)\n",
      "... progress: 14050 (0.6204)\n",
      "... progress: 14075 (0.6304)\n",
      "... progress: 14100 (0.6404)\n",
      "... progress: 14125 (0.6504)\n",
      "... progress: 14150 (0.6604)\n",
      "... progress: 14175 (0.6704)\n",
      "... progress: 14200 (0.6804)\n",
      "... progress: 14225 (0.6904)\n",
      "... progress: 14250 (0.7004)\n",
      "... progress: 14275 (0.7104)\n",
      "... progress: 14300 (0.7204)\n",
      "... progress: 14325 (0.7304)\n",
      "... progress: 14350 (0.7404)\n",
      "... progress: 14375 (0.7504)\n",
      "... progress: 14400 (0.7604)\n",
      "... progress: 14425 (0.7704)\n",
      "... progress: 14450 (0.7804)\n",
      "... progress: 14475 (0.7904)\n",
      "... progress: 14500 (0.8004)\n",
      "... progress: 14525 (0.8104)\n",
      "... progress: 14550 (0.8204)\n",
      "... progress: 14575 (0.8304)\n",
      "... progress: 14600 (0.8404)\n",
      "... progress: 14625 (0.8504)\n",
      "... progress: 14650 (0.8604)\n",
      "... progress: 14675 (0.8704)\n",
      "... progress: 14700 (0.8804)\n",
      "... progress: 14725 (0.8904)\n",
      "... progress: 14750 (0.9004)\n",
      "... progress: 14775 (0.9104)\n",
      "... progress: 14800 (0.9204)\n",
      "... progress: 14825 (0.9304)\n",
      "... progress: 14850 (0.9404)\n",
      "... progress: 14875 (0.9504)\n",
      "... progress: 14900 (0.9604)\n",
      "... progress: 14925 (0.9704)\n",
      "... progress: 14950 (0.9804)\n",
      "... progress: 14975 (0.9904)\n",
      "----- split #5 (size: 2500) -----\n",
      " [[2429   71]\n",
      " [   0    0]]\n",
      "----- acc: 0.9716 -----\n",
      "====================================\n",
      " Final confusion matrix:\n",
      " [[14560.   440.]\n",
      " [    0.     0.]]\n",
      " Final accuracy: 0.9706666666666667\n",
      "====================================\n",
      "CPU times: user 5h 47min 35s, sys: 53.5 s, total: 5h 48min 29s\n",
      "Wall time: 21min 46s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gtc_evaluate_with_checkpoints(grover_hw_pd, solaiman_detector, checkpoint=2500, score_func=safe_accuracy)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "transf291",
   "language": "python",
   "name": "transf291"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}