{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Access data from ASReview file\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "The API is still under development and can change at any time without warning. \n",
    "\n",
    "</div>\n",
    "\n",
    "Data generated using ASReview LAB is stored in an ASReview project file. Via the ASReview Python API, there are two ways to access the data in the ASReview (extension `.asreview`) file: Via the [asreview.Project](reference/asreview.rst#asreview.Project) API and the [asreview.SQLiteState](reference/asreview.rst#asreview.SQLiteState) API. The project API is for retrieving general project settings, the imported dataset, the feature matrix, etc. The state API retrieves data related directly to the reviewing process, such as the labels, the time of labeling, and the classifier used. \n",
    "\n",
    "The ASReview Python API can be used for project files obtained reviews and simulations.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example Data\n",
    "To illustrate the ASReview Python API, the benchmark dataset `van_de_Schoot_2017` is used. The project file `example.asreview` can be obtained by running the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "asreview simulate synergy:van_de_Schoot_2018 -o example.asreview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python imports\n",
    "\n",
    "Import the `asreview` module as `asr`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import asreview as asr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Project API\n",
    "\n",
    "The ASReview project file is a zipped folder with the extension `.asreview`. This makes inspection of it's contents straightforward. The `asreview` Python package offers an API to open a project file directly as an [asreview.Project](reference/asreview.rst#asreview.Project). \n",
    "\n",
    "For this example, we will create a temporary folder to unpack the project to:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tempfile import TemporaryDirectory\n",
    "\n",
    "tmpdir = TemporaryDirectory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the project with the `load` classmethod of [asreview.Project](reference/asreview.rst#asreview.Project)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "project = asr.Project.load(\"example.asreview\", tmpdir.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following files can be found in the folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "example\n",
      "example/project.json\n",
      "example/data_store.db\n",
      "example/feature_matrices\n",
      "example/data\n",
      "example/reviews\n",
      "example/data/van_de_Schoot_2018.csv\n",
      "example/reviews/3b7b6c2b3d6f487ba19ffcc4ac8adef5\n",
      "example/reviews/3b7b6c2b3d6f487ba19ffcc4ac8adef5/results.db\n"
     ]
    }
   ],
   "source": [
    "tmpdir_path = Path(tmpdir.name)\n",
    "for path in tmpdir_path.rglob(\"*\"):\n",
    "    print(path.relative_to(tmpdir_path))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "To inspect the project details in `project.json`, use the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'version': '2.0b6.dev20+gf283ffd9.d20250505',\n",
       " 'id': 'example',\n",
       " 'mode': 'simulate',\n",
       " 'name': 'van_de_Schoot_2018',\n",
       " 'created_at_unix': 1746457760,\n",
       " 'reviews': [{'id': '3b7b6c2b3d6f487ba19ffcc4ac8adef5', 'status': 'finished'}],\n",
       " 'feature_matrices': [],\n",
       " 'tags': None,\n",
       " 'datasets': [{'id': 'van_de_Schoot_2018.csv',\n",
       "   'name': 'van_de_Schoot_2018.csv'}]}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "project.config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The imported dataset is located at `data/{dataset_filename}`, and can be inspected using the following code: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The dataset contains 4544 records.\n"
     ]
    }
   ],
   "source": [
    "datastore_fp = Path(tmpdir.name) / \"example\" / \"data_store.db\"\n",
    "\n",
    "dataset = asr.DataStore(datastore_fp)\n",
    "print(f\"The dataset contains {len(dataset)} records.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dataset_row</th>\n",
       "      <th>dataset_id</th>\n",
       "      <th>duplicate_of</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>keywords</th>\n",
       "      <th>year</th>\n",
       "      <th>doi</th>\n",
       "      <th>url</th>\n",
       "      <th>included</th>\n",
       "      <th>record_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Annual Research Review: Resilience and mental ...</td>\n",
       "      <td>Researchers focused on mental health of confli...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1111/jcpp.12053</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Profiling the Trauma Related Symptoms of Bosni...</td>\n",
       "      <td>The objective of this study was to profile tra...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1097/00005053-200007000-00004</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Acute panicogenic, anxiogenic and dissociative...</td>\n",
       "      <td>Increased anxiety and panic to inhalation of c...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1016/j.jpsychires.2011.01.009</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>A Pooled Analysis of Gender and Trauma-Type Ef...</td>\n",
       "      <td>To examine effects of gender and trauma type o...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.4088/jcp.v69n1002</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Twelve-Month Use of Mental Health Services in ...</td>\n",
       "      <td>Dramatic changes have occurred in mental healt...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1001/archpsyc.62.6.629</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   dataset_row              dataset_id duplicate_of  \\\n",
       "0            0  van_de_Schoot_2018.csv         None   \n",
       "1            1  van_de_Schoot_2018.csv         None   \n",
       "2            2  van_de_Schoot_2018.csv         None   \n",
       "3            3  van_de_Schoot_2018.csv         None   \n",
       "4            4  van_de_Schoot_2018.csv         None   \n",
       "\n",
       "                                               title  \\\n",
       "0  Annual Research Review: Resilience and mental ...   \n",
       "1  Profiling the Trauma Related Symptoms of Bosni...   \n",
       "2  Acute panicogenic, anxiogenic and dissociative...   \n",
       "3  A Pooled Analysis of Gender and Trauma-Type Ef...   \n",
       "4  Twelve-Month Use of Mental Health Services in ...   \n",
       "\n",
       "                                            abstract authors keywords  year  \\\n",
       "0  Researchers focused on mental health of confli...      []       []  None   \n",
       "1  The objective of this study was to profile tra...      []       []  None   \n",
       "2  Increased anxiety and panic to inhalation of c...      []       []  None   \n",
       "3  To examine effects of gender and trauma type o...      []       []  None   \n",
       "4  Dramatic changes have occurred in mental healt...      []       []  None   \n",
       "\n",
       "                                                doi   url  included  record_id  \n",
       "0                https://doi.org/10.1111/jcpp.12053  None         0          0  \n",
       "1  https://doi.org/10.1097/00005053-200007000-00004  None         0          1  \n",
       "2  https://doi.org/10.1016/j.jpsychires.2011.01.009  None         0          2  \n",
       "3              https://doi.org/10.4088/jcp.v69n1002  None         0          3  \n",
       "4         https://doi.org/10.1001/archpsyc.62.6.629  None         0          4  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.get_df().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## State API\n",
    "\n",
    "The data stored during the review process can be accessed as a pandas DataFrame using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The state contains 322 records.\n"
     ]
    }
   ],
   "source": [
    "with asr.open_state(\"example.asreview\") as state:\n",
    "    df_results = state.get_results_table()\n",
    "    print(f\"The state contains {len(df_results)} records.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned `state` instance is of type [asreview.SQLState](reference/asreview.rst#asreview.SQLiteState). Note that the state contains less records than the original dataset. This is because by default the simulation stopped after finding all relevant records."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>record_id</th>\n",
       "      <th>label</th>\n",
       "      <th>classifier</th>\n",
       "      <th>querier</th>\n",
       "      <th>balancer</th>\n",
       "      <th>feature_extractor</th>\n",
       "      <th>training_set</th>\n",
       "      <th>time</th>\n",
       "      <th>note</th>\n",
       "      <th>tags</th>\n",
       "      <th>user_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>312</th>\n",
       "      <td>1376</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>312</td>\n",
       "      <td>1746457765.998829</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>1589</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>313</td>\n",
       "      <td>1746457766.009229</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>2368</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>314</td>\n",
       "      <td>1746457766.019135</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>315</th>\n",
       "      <td>1359</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>315</td>\n",
       "      <td>1746457766.028713</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>952</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>316</td>\n",
       "      <td>1746457766.038262</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>317</th>\n",
       "      <td>1754</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>317</td>\n",
       "      <td>1746457766.04837</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318</th>\n",
       "      <td>1985</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>318</td>\n",
       "      <td>1746457766.05749</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>1196</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>319</td>\n",
       "      <td>1746457766.06699</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>1008</td>\n",
       "      <td>0</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>320</td>\n",
       "      <td>1746457766.076556</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>321</th>\n",
       "      <td>3442</td>\n",
       "      <td>1</td>\n",
       "      <td>svm</td>\n",
       "      <td>max</td>\n",
       "      <td>balanced</td>\n",
       "      <td>tfidf</td>\n",
       "      <td>321</td>\n",
       "      <td>1746457766.086036</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     record_id  label classifier querier  balancer feature_extractor  \\\n",
       "312       1376      0        svm     max  balanced             tfidf   \n",
       "313       1589      0        svm     max  balanced             tfidf   \n",
       "314       2368      0        svm     max  balanced             tfidf   \n",
       "315       1359      0        svm     max  balanced             tfidf   \n",
       "316        952      0        svm     max  balanced             tfidf   \n",
       "317       1754      0        svm     max  balanced             tfidf   \n",
       "318       1985      0        svm     max  balanced             tfidf   \n",
       "319       1196      0        svm     max  balanced             tfidf   \n",
       "320       1008      0        svm     max  balanced             tfidf   \n",
       "321       3442      1        svm     max  balanced             tfidf   \n",
       "\n",
       "     training_set               time  note  tags  user_id  \n",
       "312           312  1746457765.998829  None  None     <NA>  \n",
       "313           313  1746457766.009229  None  None     <NA>  \n",
       "314           314  1746457766.019135  None  None     <NA>  \n",
       "315           315  1746457766.028713  None  None     <NA>  \n",
       "316           316  1746457766.038262  None  None     <NA>  \n",
       "317           317   1746457766.04837  None  None     <NA>  \n",
       "318           318   1746457766.05749  None  None     <NA>  \n",
       "319           319   1746457766.06699  None  None     <NA>  \n",
       "320           320  1746457766.076556  None  None     <NA>  \n",
       "321           321  1746457766.086036  None  None     <NA>  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results.tail(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can merge the information from the state file with the original dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>labeling_order</th>\n",
       "      <th>record_id</th>\n",
       "      <th>label</th>\n",
       "      <th>classifier</th>\n",
       "      <th>querier</th>\n",
       "      <th>balancer</th>\n",
       "      <th>feature_extractor</th>\n",
       "      <th>training_set</th>\n",
       "      <th>time</th>\n",
       "      <th>note</th>\n",
       "      <th>...</th>\n",
       "      <th>dataset_id</th>\n",
       "      <th>duplicate_of</th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "      <th>authors</th>\n",
       "      <th>keywords</th>\n",
       "      <th>year</th>\n",
       "      <th>doi</th>\n",
       "      <th>url</th>\n",
       "      <th>included</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>top_down</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "      <td>1746457762.147489</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Annual Research Review: Resilience and mental ...</td>\n",
       "      <td>Researchers focused on mental health of confli...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1111/jcpp.12053</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>top_down</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>1</td>\n",
       "      <td>1746457762.150226</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Profiling the Trauma Related Symptoms of Bosni...</td>\n",
       "      <td>The objective of this study was to profile tra...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1097/00005053-200007000-00004</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>top_down</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>2</td>\n",
       "      <td>1746457762.15211</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Acute panicogenic, anxiogenic and dissociative...</td>\n",
       "      <td>Increased anxiety and panic to inhalation of c...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1016/j.jpsychires.2011.01.009</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>top_down</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>3</td>\n",
       "      <td>1746457762.153575</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>A Pooled Analysis of Gender and Trauma-Type Ef...</td>\n",
       "      <td>To examine effects of gender and trauma type o...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.4088/jcp.v69n1002</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>None</td>\n",
       "      <td>top_down</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>4</td>\n",
       "      <td>1746457762.154858</td>\n",
       "      <td>None</td>\n",
       "      <td>...</td>\n",
       "      <td>van_de_Schoot_2018.csv</td>\n",
       "      <td>None</td>\n",
       "      <td>Twelve-Month Use of Mental Health Services in ...</td>\n",
       "      <td>Dramatic changes have occurred in mental healt...</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>None</td>\n",
       "      <td>https://doi.org/10.1001/archpsyc.62.6.629</td>\n",
       "      <td>None</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   labeling_order  record_id  label classifier   querier balancer  \\\n",
       "0               0          0      0       None  top_down     None   \n",
       "1               1          1      0       None  top_down     None   \n",
       "2               2          2      0       None  top_down     None   \n",
       "3               3          3      0       None  top_down     None   \n",
       "4               4          4      0       None  top_down     None   \n",
       "\n",
       "  feature_extractor  training_set               time  note  ...  \\\n",
       "0              None             0  1746457762.147489  None  ...   \n",
       "1              None             1  1746457762.150226  None  ...   \n",
       "2              None             2   1746457762.15211  None  ...   \n",
       "3              None             3  1746457762.153575  None  ...   \n",
       "4              None             4  1746457762.154858  None  ...   \n",
       "\n",
       "               dataset_id  duplicate_of  \\\n",
       "0  van_de_Schoot_2018.csv          None   \n",
       "1  van_de_Schoot_2018.csv          None   \n",
       "2  van_de_Schoot_2018.csv          None   \n",
       "3  van_de_Schoot_2018.csv          None   \n",
       "4  van_de_Schoot_2018.csv          None   \n",
       "\n",
       "                                               title  \\\n",
       "0  Annual Research Review: Resilience and mental ...   \n",
       "1  Profiling the Trauma Related Symptoms of Bosni...   \n",
       "2  Acute panicogenic, anxiogenic and dissociative...   \n",
       "3  A Pooled Analysis of Gender and Trauma-Type Ef...   \n",
       "4  Twelve-Month Use of Mental Health Services in ...   \n",
       "\n",
       "                                            abstract authors keywords  year  \\\n",
       "0  Researchers focused on mental health of confli...      []       []  None   \n",
       "1  The objective of this study was to profile tra...      []       []  None   \n",
       "2  Increased anxiety and panic to inhalation of c...      []       []  None   \n",
       "3  To examine effects of gender and trauma type o...      []       []  None   \n",
       "4  Dramatic changes have occurred in mental healt...      []       []  None   \n",
       "\n",
       "                                                doi   url included  \n",
       "0                https://doi.org/10.1111/jcpp.12053  None        0  \n",
       "1  https://doi.org/10.1097/00005053-200007000-00004  None        0  \n",
       "2  https://doi.org/10.1016/j.jpsychires.2011.01.009  None        0  \n",
       "3              https://doi.org/10.4088/jcp.v69n1002  None        0  \n",
       "4         https://doi.org/10.1001/archpsyc.62.6.629  None        0  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset_with_results = df_results.reset_index(names=\"labeling_order\").join(\n",
    "    dataset.get_df().set_index(\"record_id\")\n",
    ")\n",
    "dataset_with_results.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are also multiple functions to obtain one specific variable in the data. For example, to plot the labeling times in a graph, use the following code (keep in mind that this data is from a simulation): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: title={'center': 'Time of labeling'}>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGzCAYAAAD9pBdvAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/TGe4hAAAACXBIWXMAAA9hAAAPYQGoP6dpAABRq0lEQVR4nO3deVhUZf8G8HvYBgRmWGSRRVYFN1ywFPcFxaXSFjOz1HJJ09LMSvtValb4upSVhdqiaZllveabS4oLmILlWq7IJqAsKgrDIgPMPL8/kNERBhlgGJb7c13n0nnmnJnvOQ7M7TnP8xyJEEKAiIiIyEhMjF0AERERNW8MI0RERGRUDCNERERkVAwjREREZFQMI0RERGRUDCNERERkVAwjREREZFQMI0RERGRUDCNERERkVAwjRI3ApEmT4O3tbewyaiQrKwtPPfUUHB0dIZFIsGrVKp3rSiQSLFq0SO/32LBhAyQSCY4fP17zQu+zaNEiSCQSrTZvb29MmjSpzt6DiMqYGbsAoubq/i86XQ4ePGjgSgzrtddew549e7Bw4UK4urqie/fuxi6JiBoYhhEiI9m0aZPW440bNyIyMrJCe7t27fDVV19BrVbXZ3l15sCBAxg1ahTmzZtn7FJqLS4uDiYmPKFMVNcYRoiM5LnnntN6fPToUURGRlZob+yuXbsGOzs7Y5dRJ6RSqbFLIGqSGPGJGoH7+4xcvnwZEokEK1aswBdffAFfX1+0aNECQ4cORVpaGoQQWLJkCTw8PGBlZYVRo0bh5s2bFV539+7d6Nu3L6ytrWFra4uRI0fi3Llz1aopKSkJY8aMgYODA1q0aIGePXti586dmufL+3EIIfDFF19AIpFU+9JUuZSUFLz88ssICAiAlZUVHB0dMWbMGFy+fLnS9QsLC/HSSy/B0dERMpkMEyZMwK1bt+psv+/vM1K+j0eOHMHcuXPh5OQEa2trPP7447h+/brWtmq1GosWLYKbmxtatGiBgQMH4vz58+yHQgSeGSFq1H744QcUFxfjlVdewc2bN7Fs2TI8/fTTGDRoEKKiovDWW28hISEBn3/+OebNm4dvv/1Ws+2mTZswceJEhIWF4T//+Q8KCwsRERGBPn364NSpU1V2mM3KykKvXr1QWFiIV199FY6Ojvjuu+/w2GOP4ZdffsHjjz+Ofv36YdOmTXj++ecxZMgQTJgwQe/9O3bsGGJiYvDMM8/Aw8MDly9fRkREBAYMGIDz58+jRYsWWuvPmjULdnZ2WLRoEeLi4hAREYGUlBRERUVpglBt9luXV155Bfb29li4cCEuX76MVatWYdasWfjpp5806yxYsADLli3Do48+irCwMPzzzz8ICwtDUVGR3u9H1OQIImoQZs6cKXT9SE6cOFF4eXlpHicnJwsAwsnJSeTk5GjaFyxYIACIzp07i5KSEk37uHHjhIWFhSgqKhJCCJGXlyfs7OzE1KlTtd4nMzNTyOXyCu33mzNnjgAg/vzzT01bXl6e8PHxEd7e3kKlUmnaAYiZM2c++ADcWXfhwoWax4WFhRXWiY2NFQDExo0bNW3r168XAERwcLAoLi7WtC9btkwAENu3b9d7vxcuXFjh38PLy0tMnDixwvuGhoYKtVqtaX/ttdeEqamp5t8mMzNTmJmZidGjR2u93qJFiwQArdckao54mYaoERszZgzkcrnmcY8ePQCU9UcxMzPTai8uLsbVq1cBAJGRkcjJycG4ceNw48YNzWJqaooePXo8cATPrl278PDDD6NPnz6aNhsbG0ybNg2XL1/G+fPn62T/rKysNH8vKSlBdnY2/P39YWdnh5MnT1ZYf9q0aTA3N9c8njFjBszMzLBr1y4Atd9vXaZNm6Z1Capv375QqVRISUkBAOzfvx+lpaV4+eWXtbZ75ZVXavR+RE1Nkw0jH374IXr16oUWLVpUu/Nc+TXt+5fly5dXWFepVKJLly6QSCQ4ffq01nNCCKxYsQJt27aFVCqFu7s7Pvzww0rf88iRIzAzM0OXLl202svnOLh3CQwM1Dxf3megsmXr1q0A7l7Prmy5du2a1r783//9H7y8vCCVSuHt7a11Or86Tp48iSFDhsDOzg6Ojo6YNm0a8vPz9XoN0l/r1q21HpcHE09Pz0rby/tPxMfHAwAGDRoEJycnrWXv3r1an4/KpKSkICAgoEJ7u3btNM/Xhdu3b+O9996Dp6cnpFIpWrZsCScnJ+Tk5CA3N7fC+m3atNF6bGNjg1atWmn6mNR2v3W5/9/B3t4ewN3jXX48/P39tdZzcHDQrEvUnDXqPiMDBgzApEmTKu38VVxcjDFjxiAkJATffPNNtV4vIyND6/Hu3bsxefJkPPnkkxXWffPNN+Hm5oZ//vmnwnOzZ8/G3r17sWLFCnTq1Ak3b96stPNgTk4OJkyYgMGDByMrK6vC8x06dMC+ffs0j+/9n66np2eFetetW4fly5dj+PDhAICxY8di2LBhWutMmjQJRUVFcHZ21rQ9/fTTyMrKwjfffAN/f39kZGToNYw0PT0doaGhGDt2LFavXg2FQoE5c+Zg0qRJ+OWXX6r9OqQ/U1NTvdqFEACg+ffdtGkTXF1dK6x372fNmF555RWsX78ec+bMQUhICORyOSQSCZ555pkaDXU21H4/6HgTUdUaxm8cA1i8eDGAsrMD1XX/L6ft27dj4MCB8PX11WrfvXs39u7di19//RW7d+/Weu7ChQuIiIjA2bNnNf9z9PHxqfT9pk+fjmeffRampqb47bffKjxvZmZW6S9MoOyX3/3Pbdu2DU8//TRsbGwAlJ3ivvc09/Xr13HgwAGtcPbHH38gOjoaSUlJcHBwAIBKO/B9/fXXWLlyJZKTk+Ht7Y1XX31Vc8p5x44dMDc3xxdffKGZg2HNmjUICgpCQkJChf8NkvH5+fkBAJydnREaGqr39l5eXoiLi6vQfvHiRc3zdeGXX37BxIkTsXLlSk1bUVERcnJyKl0/Pj4eAwcO1DzOz89HRkYGRowYAaD2+11T5ccjISFB6/dBdnZ2paN9iJqbJnuZpraysrKwc+dOTJ48uUL71KlTsWnTpgo9+QHg999/h6+vL3bs2AEfHx94e3tjypQpFc6MrF+/HklJSVi4cKHOGuLj4+Hm5gZfX1+MHz8eqampOtc9ceIETp8+XaHee23cuBEtWrTAU089pWn73//+h+7du2PZsmVwd3dH27ZtMW/ePNy+fVuzzg8//ID33nsPH374IS5cuICPPvoI7777Lr777jsAZZd5LCwstCaDKg9Bhw8f1lkPGU9YWBhkMhk++ugjlJSUVHj+/mGp9xsxYgT+/vtvxMbGatoKCgqwbt06eHt7o3379nVSp6mpaYWzC59//jlUKlWl669bt05rfyIiIlBaWqo5W1jb/a6pwYMHw8zMDBEREVrtq1evNsj7ETU2TfbMSG199913sLW1xRNPPKFpE0Jg0qRJmD59Orp3717pXAdJSUlISUnB1q1bsXHjRqhUKrz22mt46qmncODAAQBlIWP+/Pn4888/dZ4W7tGjBzZs2ICAgABkZGRg8eLF6Nu3L86ePQtbW9sK63/zzTdo164devXqpXOfvvnmGzz77LNaZ0uSkpJw+PBhWFpaYtu2bbhx4wZefvllZGdnY/369QCAhQsXYuXKlZpj4ePjg/Pnz2Pt2rWYOHEiBg0ahLlz52L58uWYPXs2CgoKMH/+fAAVL31RwyCTyRAREYHnn38e3bp1wzPPPAMnJyekpqZi586d6N27d5VflPPnz8ePP/6I4cOH49VXX4WDgwO+++47JCcn49dff62zWUofeeQRbNq0CXK5HO3bt0dsbCz27dsHR0fHStcvLi7G4MGD8fTTTyMuLg5ffvkl+vTpg8cee6xO9rumXFxcMHv2bKxcuRKPPfYYhg0bhn/++Qe7d+9Gy5Yt9Z5/hajJMepYHj19+OGHwtraWrOYmJgIqVSq1ZaSkqK1zfr164VcLtf7vQICAsSsWbO02j799FPRu3dvUVpaKoS4O7zy1KlTmnWmTp0qAIi4uDhN24kTJwQAcfHiRVFaWiq6d+8uIiIiNM8vXLhQdO7cucp6bt26JWQymfj6668rPFdYWCjkcrlYsWKFzu1jYmIEAHH8+HGt9iFDhghLS0ut4aG//vqrkEgkorCwUOTn5wsAwsrKSus4S6VS4ezsrNnmhx9+EC4uLsLU1FRYWFiIefPmCRcXF7F06dIq94vuqsnQ3uXLl2utd/DgQQFAbN26Vau9fAjqsWPHKqwfFhYm5HK5sLS0FH5+fmLSpEkVPieVSUxMFE899ZSws7MTlpaW4uGHHxY7duyosB5qMbT31q1b4oUXXhAtW7YUNjY2IiwsTFy8eFHnENvo6Ggxbdo0YW9vL2xsbMT48eNFdnZ2hfepzn7rM7S3suMKQBw8eFDTVlpaKt59913h6uoqrKysxKBBg8SFCxeEo6OjmD59erWOD1FT1ajCSHZ2toiPj9csDz/8sPjPf/6j1Xbv3ApC1CyMHDp0SAAQp0+f1mofNWqUMDExEaamppoFgDA1NRUTJkwQQgjx3nvvCTMzM63tCgsLBQCxd+9ecevWLc025YtEItG07d+/X2dd3bt3F/Pnz6/QvnHjRmFubi6uXbumc9sXX3xRdOnSpUL7hAkThJ+fn1bb+fPnBQBx6dIlkZmZKQCI77//Xus4x8fHi6SkpAqvl5mZKfLy8kR+fr4wMTERP//8s86aiJq78t8HH3zwgbFLITKqRnWZxsHBQdPJEijrl+Ds7FznHSS/+eYbBAcHo3Pnzlrtn332GT744APN4/T0dISFheGnn37SzO/Qu3dvlJaWIjExUdNZ7tKlSwDKOrHJZDKcOXNG63W//PJLHDhwAL/88ovOzq75+flITEzE888/X2m9jz32GJycnHRu+/PPPyM8PLzCc71798bWrVuRn5+v6fh66dIlmJiYaKYSd3NzQ1JSEsaPH1/p69/LxcUFAPDtt9/C0tISQ4YMeeA2RM3B7du3tS6RAsCqVasAlI0MJGrOGlUY0Udqaipu3ryJ1NRUqFQqzVwg/v7+mi/dwMBAhIeH4/HHH9dsp1AosHXrVq3e++Xun0ug/HX8/Pzg4eEBAAgNDUW3bt3w4osvYtWqVVCr1Zg5cyaGDBmCtm3bAgA6duyo9TrOzs6wtLTUap83bx4effRReHl5IT09HQsXLoSpqSnGjRuntW1CQgIOHTqkmdSpMj/99BNKS0srvQHbs88+iyVLluCFF17A4sWLcePGDbzxxht48cUXNb84Fy9ejFdffRVyuRzDhg2DUqnE8ePHcevWLcydOxdAWUe8Xr16wcbGBpGRkXjjjTewdOnSJnODNKLa+umnn7BhwwaMGDECNjY2OHz4MH788UcMHToUvXv3NnZ5RMZl7FMztdG/f3+xfv36Sp+bOHGiAFBhufcaLoAK269du1ZYWVlp9aHQpbI+I0IIcfXqVfHEE08IGxsb4eLiIiZNmlTpdetylfUZGTt2rGjVqpWwsLAQ7u7uYuzYsSIhIaHCtgsWLBCenp5a02/fLyQkRDz77LM6n79w4YIIDQ0VVlZWwsPDQ8ydO7fCNNw//PCD6NKli7CwsBD29vaiX79+4r///a/m+eeff144ODgICwsLERQUpDVVNxGV9R0bPHiwcHR0FObm5sLDw0PMnj1b5OXlGbs0IqOTCMFZeYiIiMh4OM8IERERGRXDCBERERlVo+jAqlarkZ6eDltbW04ORERE1EgIIZCXlwc3N7cqJ0NsFGEkPT29wl1IiYiIqHFIS0vTjDqtTKMII+XTn6elpUEmkxm5GiIiIqoOhUIBT0/PSm9jcq9GEUbKL83IZDKGESIiokbmQV0s2IGViIiIjIphhIiIiIyKYYSIiIiMimGEiIiIjIphhIiIiIyKYYSIiIiMimGEiIiIjIphhIiIiIyKYYSIiIiMimGEiIiIjIphhIiIiIyKYYSIiIiMimGEiIiomRJCYMe/6Zi68ThUamG0OhrFXXuJiIiobl3MVGDR/87haNJNAMC2U1fxVLCHUWphGCEiImpGcgtL8Mm+S9h0NAUqtYDUzAQvD/DHI0GtjFYTwwgREVEzoFIL/Hw8Dcv3xOFmQTEAYHhHV/zfyHbwsG9h1NoYRoiIiJq4Eym3sOh/53Dmai4AwN/ZBose7YA+bVoaubIyendgvXr1Kp577jk4OjrCysoKnTp1wvHjx6vcJioqCt26dYNUKoW/vz82bNhQ03qJiIiomq7lFWHuz6fxZEQMzlzNha3UDO8+0h67Z/dtMEEE0PPMyK1bt9C7d28MHDgQu3fvhpOTE+Lj42Fvb69zm+TkZIwcORLTp0/HDz/8gP3792PKlClo1aoVwsLCar0DREREpK24VI0NMcn4bH8C8pWlAIAxwR54c1ggnGylRq6uIokQotpjeebPn48jR47gzz//rPYbvPXWW9i5cyfOnj2raXvmmWeQk5ODP/74o1qvoVAoIJfLkZubC5lMVu33JiIiam4OXbqORb+fQ9L1AgBAZw85Fo/qiC6edvVeS3W/v/W6TPO///0P3bt3x5gxY+Ds7IyuXbviq6++qnKb2NhYhIaGarWFhYUhNjZW5zZKpRIKhUJrISIiIt3SbhZi6sbjmPDt30i6XgBHawssezII217ubZQgog+9wkhSUhIiIiLQpk0b7NmzBzNmzMCrr76K7777Tuc2mZmZcHFx0WpzcXGBQqHA7du3K90mPDwccrlcs3h6eupTJhERUbORV1SCpbsvYvDKaESez4KpiQQv9vbBgXkD8PRDnjAxkRi7xAfSq8+IWq1G9+7d8dFHHwEAunbtirNnz2LNmjWYOHFinRW1YMECzJ07V/NYoVAwkBAREd1DpRb46VgaPo6Mw438sqG6vf0dsfDRDmjrYmvk6vSjVxhp1aoV2rdvr9XWrl07/Prrrzq3cXV1RVZWllZbVlYWZDIZrKysKt1GKpVCKm14HWyIiIgagsPxN/DBzvO4mJkHAPBtaY23R7TD4HbOkEga/pmQ++kVRnr37o24uDittkuXLsHLy0vnNiEhIdi1a5dWW2RkJEJCQvR5ayIiomYv8Xo+Ptp5AfsvXgMAyK3MMXtwGzzX0wsWZo33dnN6hZHXXnsNvXr1wkcffYSnn34af//9N9atW4d169Zp1lmwYAGuXr2KjRs3AgCmT5+O1atX480338SLL76IAwcO4Oeff8bOnTvrdk+IiIiaqNzbJVi17xI2xaagVC1gaiLB8z29MHtwG9hbWxi7vFrTK4w89NBD2LZtGxYsWID3338fPj4+WLVqFcaPH69ZJyMjA6mpqZrHPj4+2LlzJ1577TV8+umn8PDwwNdff805RoiIiB5ArRb45eQV/Gf3RWTfmcJ9UKAz3h7RDv7ONkauru7oNc+IsXCeESIiam7OXs3Fu9vP4lRqDgDAz8kaCx/tgH5tnYxbmB6q+/3Ne9MQERE1IDmFxVi+Jw6b/06FEIC1hSlmh7bBpF4+jbpfSFUYRoiIiBqA8rvqLvvjIm4VlgAAHuvshv8b2Q4uMksjV2dYDCNERERGFpN4Ax/suIDzGWUzjge42GLxqA7o6eto5MrqB8MIERGRkSRdz8dHuy5i34Wy+bhsLc0wJ7QtJoR4wdy0aV6SqQzDCBERUT3LKSzGZ/sTsDH2smao7nM9WmN2aFs4NIGhuvpiGCEiIqonJSo1NsWm4NP98ci9XdYvpCkO1dUXwwgREVE9OBh3DUt+P4+kGwUAgEBXW/zfyHbo26bxDNU1FIYRIiIiA7qep8Ti389hx78ZAICWNhZ4fWgAnu7uCdNGcEfd+sAwQkREZCD7zmdh7s+noSgqhYkEmNzHB7ND28JGyq/fe/FoEBERGcC2U1cwb+u/UKkFOrrLsPSJIHR0lxu7rAaJYYSIiKgOJV3Px5Id53Ew7joA4PGu7lj+VBDMmtFQXX0xjBAREdWBohIVPt0fj6//TEKJSsDcVIJp/Xzx+pAAmLBvSJUYRoiIiGrpaFI2Fvz3DJLvjJQZGOCEdx9pD1+n5jtcVx8MI0RERDWkKCrB0t0XsfmvVACAi0yKJaM6YmgHVyNX1rgwjBAREdXA0aRsvPbTaWTkFgEAnu3RGvOHB0JmaW7kyhofhhEiIiI9lKjU+Gx/PFYfTIAQgLdjCyx9MqjZ3NTOEBhGiIiIqul2sQpTNh7DkYRsAMCYYA8seqwDrDlvSK3w6BEREVXDzYJizNp8EjGJ2bC2MMXSJ4PwaGc3Y5fVJDCMEBERPUBMwg28uuUUbuQXw9rCFBsnP4xgLwdjl9VkMIwQERFV4XRaDqZsPI7CYhXautjg46e7cCbVOsYwQkREpMOhS9fxyo+nUFisQt82LfH1xO6Qmpkau6wmh2GEiIjoPvnKUnwSeQnrjyRDLYCure0Q8Vwwg4iBMIwQERHdIYTAzjMZWLLjPLIUSgDAMw95YtFjHWBpziBiKAwjREREAE6k3MRHuy7iRMotAICXYwu8P6oj+rd1MnJlTR/DCBERNWuJ1/Ox7I+L2HMuCwBgaW6Cl/r5YcYAP54NqScMI0RE1Cxdz1Pi0/2X8OPfaVCpBUwkwNPdPfHakLZwkVkau7xmhWGEiIialQJlKb7+MxlrDyWisFgFAAht54y3hgWijYutkatrnhhGiIioWShVqfHT8TSs2heP63llnVM7e8ixYEQ73lfGyBhGiIioSRNCIPJ8Fv7zx0UkXi8AALR2aIE3hwVgZKdWkEgkRq6QGEaIiKjJOp+uwPs7zuFo0k0AgIO1BV4d5I9ne3jBwszEyNVROYYRIiJqcrLzlfg48hJ+/DsVagFIzUwwuY8Ppg/wg8zS3Njl0X0YRoiIqMkoUamxKTYFq/ZdgqKoFAAwMqgVFgwPhId9CyNXR7rodY5q0aJFkEgkWktgYKDO9Tds2FBhfUtLDpciIqK6Vd4vZNiqQ3h/x3koikrRwU2Gn6b1xBfPdmMQaeD0PjPSoUMH7Nu37+4LmFX9EjKZDHFxcZrH7ChERER16d8rOfhw5wX8lVzWL8TR2gJvhAVgTHdPmJrwO6cx0DuMmJmZwdXVtdrrSyQSvdYHAKVSCaVSqXmsUCj02p6IiJq+K7cKsWJPHH47nQ6A/UIaM727EsfHx8PNzQ2+vr4YP348UlNTq1w/Pz8fXl5e8PT0xKhRo3Du3LkHvkd4eDjkcrlm8fT01LdMIiJqohRFJVi6+yIGrYzWBJEnurrjwLwBeHNYIINIIyQRQojqrrx7927k5+cjICAAGRkZWLx4Ma5evYqzZ8/C1rbirHWxsbGIj49HUFAQcnNzsWLFChw6dAjnzp2Dh4eHzvep7MyIp6cncnNzIZPJ9NxFIiJqCkpUavxwNAWf7o/HrcISAECIryP+b2Q7dHSXG7k6qoxCoYBcLn/g97deYeR+OTk58PLywscff4zJkyc/cP2SkhK0a9cO48aNw5IlS6r9PtXdGSIianqEENhzrmzSsuQbZZOW+Tvb4O0RgRgY4My+iA1Ydb+/azW0187ODm3btkVCQkK11jc3N0fXrl2rvT4RETVvp1Jv4aNdF3Ds8i0AQEsbC7w2pC3GdveEmSknLWsqahVG8vPzkZiYiOeff75a66tUKpw5cwYjRoyozdsSEVETl3azEMv2xOH3f8r6hFiam2BqX1+81N8PNlJOkdXU6PUvOm/ePDz66KPw8vJCeno6Fi5cCFNTU4wbNw4AMGHCBLi7uyM8PBwA8P7776Nnz57w9/dHTk4Oli9fjpSUFEyZMqXu94SIiBq93MISrD4Yj+9iUlCsUkMiAZ7s5oHXh7ZFK7mVscsjA9ErjFy5cgXjxo1DdnY2nJyc0KdPHxw9ehROTk4AgNTUVJiY3D1tduvWLUydOhWZmZmwt7dHcHAwYmJi0L59+7rdCyIiatSUpSpsik3B5wcSkHu7rHNqH/+WeHtEO7R3Y1/Bpq5WHVjrCzuwEhE1TSq1wO//pOPjyEtIvVkIAGjrYoO3R7RD/7ZO7JzayNVLB1YiIqKaKB8h83FkHC5l5QMAnGyleH1IWzwV7MHOqc0MwwgREdUbIQSiL13Hyr2XcOZqLgBAZmmGl/r7YVIvb1izc2qzxH91IiKqF38lZWPF3jjNMN0WFqaY3McHU/r6Qm7FWVObM4YRIiIyqLjMPPznj4s4cPEaAMDCzAQTenphxgA/ONpIjVwdNQQMI0REZBCZuUX4JPIStp5Ig1oAZiYSjH3IE68MagNXuaWxy6MGhGGEiIjq1O1iFdZEJ2LtoUQUlagBAMM7uuKNsAD4OtkYuTpqiBhGiIioTggh8MfZTHyw8wKu5twGAHT3sseCEe0Q7GVv5OqoIWMYISKiWjt2+SaW7r6IEyllnVPd7azw9oh2GNHJlXOF0AMxjBARUY3FZeZh2R8Xsf9O51RLcxNM6+eHGf39YGVhauTqqLFgGCEiIr1l5N7Gyr2X8OvJKxACML3TOXX24DZwkbFzKumHYYSIiKotr6gEa6IT8c3hZE3n1BGdXPH60AD4sXMq1RDDCBERPZCyVIUfjqZi9cEE3CwoBgA85G2Pt0e0Q9fW7JxKtcMwQkREOqnVAr//m44Ve+OQdrNshIyfkzXeHBaIoe1d2DmV6gTDCBERVerQpetYuvsizmcoAADOtlK8NqQtxvBGdlTHGEaIiEhLXGYeluw4j8MJNwAAtlIzTB/ghxd7+3CEDBkEwwgREQEACotL8dn+BHz9ZxJK1QLmphI839Mbswb5w8HawtjlURPGMEJERDhwMQvv/nZOM3Pq0PYuePeR9vB0aGHkyqg5YBghImrGzl7NxYq9cYiKuw6gbObURY91wJD2LkaujJoThhEiomYo6Xo+Po68hB3/ZgAou6Pu5L4+mD24DVpY8KuB6hc/cUREzUh6zm18tj8eW09cgUotIJEAj3V2w2uhbeHd0trY5VEzxTBCRNQM3MhX4suDifj+aAqKVWUzp4a2c8brQwPQrpXMyNVRc8cwQkTUhOUVleCrQ0n45nAyCopVAIAePg54c1gAgr0cjFwdURmGESKiJqiy6duDPOR4IywAffxbcuZUalAYRoiImhC1WmD7P1excu8lXLlVNkzX18kabwwNwLCOrgwh1CAxjBARNRHHLt/E+7+fx5mruQAAF5kUc0I5fTs1fAwjRESN3JVbhVi6+6JmmK6t1AwzBvrhhV6cvp0aB4YRIqJGqkBZijXRiVh3KAnKUjUkEuCZhzzx+tAAtLSRGrs8ompjGCEiamTUaoFtp65i2Z6LyFIoAQA9fR3w7iPt0cFNbuTqiPTHMEJE1Igcu3wTH+y8gH/ScgAAng5W+L8R7RDWgZ1TqfFiGCEiagTOpedixZ44HLxzDxlrC1PMGtQGL/T2hqU5+4VQ48YwQkTUgN1/DxlTEwme7u6B14a0hbOtpZGrI6obeo31WrRoESQSidYSGBhY5TZbt25FYGAgLC0t0alTJ+zatatWBRMRNQfpObcx/9d/MeSTQ5og8lhnN+yb2x/hTwQxiFCToveZkQ4dOmDfvn13X8BM90vExMRg3LhxCA8PxyOPPILNmzdj9OjROHnyJDp27FiziomImrDsfCW+jErEpqMpKC4tu4fMoEBnzBsagPZuvIcMNU16hxEzMzO4urpWa91PP/0Uw4YNwxtvvAEAWLJkCSIjI7F69WqsWbNG37cmImqy8opK8PWfyfj6zyTNPWQe9nHAm2EB6O7Ne8hQ06Z3GImPj4ebmxssLS0REhKC8PBwtG7dutJ1Y2NjMXfuXK22sLAw/Pbbb1W+h1KphFKp1DxWKBT6lklE1CgUlaiwKTYFX0Yl4FZhCQCgo7sMb4QFol8b3kOGmge9wkiPHj2wYcMGBAQEICMjA4sXL0bfvn1x9uxZ2NraVlg/MzMTLi4uWm0uLi7IzMys8n3Cw8OxePFifUojImpUSlRqbD1+BZ/tj0emoghA2T1kXh8SgOEdXWFiwhBCzYdeYWT48OGavwcFBaFHjx7w8vLCzz//jMmTJ9dZUQsWLNA6o6JQKODp6Vlnr09EZCxqtcCOMxn4eG8cLmcXAgDc5JaYE9oWT3Rz5z1kqFmq1dBeOzs7tG3bFgkJCZU+7+rqiqysLK22rKysB/Y5kUqlkEo5lTERNR1qtcCec5n4ZN8lXMrKBwA4Wltg5kB/PNujNecKoWatVhE8Pz8fiYmJaNWqVaXPh4SEYP/+/VptkZGRCAkJqc3bEhE1GkII7DufhUc+P4wZP5zEpax82FqaYe6Qtoh+cyBe7OPDIELNnl5nRubNm4dHH30UXl5eSE9Px8KFC2Fqaopx48YBACZMmAB3d3eEh4cDAGbPno3+/ftj5cqVGDlyJLZs2YLjx49j3bp1db8nREQNiBAC0Zeu45PIS/jnSi6AsllTX+zjgyl9fCFvYW7kCokaDr3CyJUrVzBu3DhkZ2fDyckJffr0wdGjR+Hk5AQASE1NhYnJ3ZMtvXr1wubNm/HOO+/g7bffRps2bfDbb79xjhEiatJiEm5gZeQlnEi5BQCwMjfFxF7emNbPFw7WFkaujqjhkQghhLGLeBCFQgG5XI7c3FzIZJz0h4gapuOXb2LF3jgcTboJAJCameD5nl54qb8fnGzZD46an+p+f/PeNEREtZSRexvL/ojDtlNXAQAWpiYY97AnXh7oDxcZp20nehCGESKiGsq9XYI10Yn49nAylKVqSCTA08GeeDW0DdztrIxdHlGjwTBCRKQnZWnZrKmrDyYg586sqQ97O+CdR9ohyMPOuMURNUIMI0RE1aRWC/zvn3Ss2BuHK7duAwDaONvgrWGBGNzOmVO3E9UQwwgRUTUcjr+B8N0XcC697F5ZLjIp5g5piye7eXDWVKJaYhghIqrCufRcLN19EX/G3wAA2ErNMH2AH17s7QMrC05WRlQXGEaIiCqRdrMQH0de0oyQMTeV4LmeXnhlUBvOFUJUxxhGiIjucaugGF8cTMDG2BQUq9QAgMc6u2He0AC0dmxh5OqImiaGESIiAEUlKqw/chlfRiUgr6gUANDLzxELhrdDJw+5kasjatoYRoioWVOpBX49eQWfRF5CRm4RACDQ1Rbzhweif1snjpAhqgcMI0TULAkhsO/CNSz74yLir+UDANzklnh9aABGd3WHqQlDCFF9YRghomZFCIEjCdlYte8Sjt+5kZ3cyhyzBvrj+RAvWJpzhAxRfWMYIaJmQQiBQ/E38Om+SziZmgOg7EZ2L/bxwfT+fpBbmRu3QKJmjGGEiJq8Eym38MHO8zh1TwgZ93BrzBjgxxvZETUADCNE1GRdyyvCf3bH4deTVwAAluYmGN/DCy/184UzQwhRg8EwQkRNTolKjY2xKVgVeQl5yrJhuk9398C8sAA42zKEEDU0DCNE1GQIIfDH2Uws2xOH5BsFAIAgDzkWP9YBXVvbG7k6ItKFYYSImoRjl2/io10XNP1CHK0tMC8sAGO7e8KEw3SJGjSGESJq1BKu5WPZHxex93wWAMDK3BRT+/pgWn8/2Ej5K46oMeBPKhE1StfzlFi17xK2HEuDSi1gIgHGPtQar4W2YedUokaGYYSIGpWiEhW+PZKMLw8mIv9O59TQdi54a1gA2rjYGrk6IqoJhhEiahRKVWr8djodq/ZdwpVbtwGUdU79vxHt0MPX0cjVEVFtMIwQUYOmUgv8/k86Pt0frxkh4yKT4q1hgRjdxZ2dU4maAIYRImqw9l/Iwke7LiDxelkIsW9hjpf6+2FCiBdaWPDXF1FTwZ9mImpwzl7NxceRl3Dg4jUAZTeym9bPFxN7eXOEDFETxJ9qImow/r2Sg8/2x2PfhbIQYmYiweQ+Ppg1yB+2lryRHVFTxTBCREYlhMBfyTfxxcEE/Bl/AwBgIgFGdXHHrEH+8HOyMXKFRGRoDCNEZBRCCERfuo7VBxJwPOUWAMDURIJRnd0wkyGEqFlhGCGieiWEwL4L1/D5gXj8eyUXAGBhaoIx3T0wvb8fPB1aGLlCIqpvDCNEVC/UaoE95zLx+YEEnM9QACibun18j9aY2s8XLpw1lajZYhghIoMqUamx/XQ61kQnIuFaPgDA2sIUE3p5Y0ofHzjaSI1cIREZG8MIERlEUYkKPx9Pw9roJFzNKZsx1dbSDBNDvDG5jw/srS2MXCERNRQmtdl46dKlkEgkmDNnjs51NmzYAIlEorVYWvJ0LFFTpSgqwZdRCejznwN4b/s5XM25jZY2FnhrWCBi5g/CvLAABhEi0lLjMyPHjh3D2rVrERQU9MB1ZTIZ4uLiNI8lEk7fTNTUXM9TYv2RZGw6moK8orIb2HnYW+Glfr4Y090TluamRq6QiBqqGoWR/Px8jB8/Hl999RU++OCDB64vkUjg6upak7ciogYuJbsA6w4lYeuJKyguVQMA2jjbYMYAPzza2Q3mprU6AUtEzUCNwsjMmTMxcuRIhIaGViuM5Ofnw8vLC2q1Gt26dcNHH32EDh066FxfqVRCqVRqHisUipqUSUQGdC49F2uik7Dz33SoRVlb19Z2eHmAPwYHOvMGdkRUbXqHkS1btuDkyZM4duxYtdYPCAjAt99+i6CgIOTm5mLFihXo1asXzp07Bw8Pj0q3CQ8Px+LFi/UtjYgMTAiBo0k3ERGdiEOXrmvaBwQ4YUZ/Pzzs48DLsESkN4kQQlR35bS0NHTv3h2RkZGaviIDBgxAly5dsGrVqmq9RklJCdq1a4dx48ZhyZIlla5T2ZkRT09P5ObmQiaTVbdcIqojarVA5IUsREQl4nRaDoCyKdsfCXLD9P5+aO/Gn0siqkihUEAulz/w+1uvMyMnTpzAtWvX0K1bN02bSqXCoUOHsHr1aiiVSpiaVt1JzdzcHF27dkVCQoLOdaRSKaRSzj1AZGzFpWpsP30Va6ITkXi9AAAgNTPB0909MbWvL1o7crZUIqo9vcLI4MGDcebMGa22F154AYGBgXjrrbceGESAsvBy5swZjBgxQr9Kiaje3C5WYcuxVHx1KAnpuUUAyuYImRDihUm9fOBky/8sEFHd0SuM2NraomPHjlpt1tbWcHR01LRPmDAB7u7uCA8PBwC8//776NmzJ/z9/ZGTk4Ply5cjJSUFU6ZMqaNdIKK6kltYgo2xl7E+5jJuFhQDAFraSDGlrw/G92gNW0tzI1dIRE1Rnc/AmpqaChOTu0P5bt26halTpyIzMxP29vYIDg5GTEwM2rdvX9dvTUQ1dE1RhG8OJ+P7oykoKFYBADwdrPBSPz88FezBOUKIyKD06sBqLNXtAENE+km6no91h5Lw35NXUawqmyMk0NUWMwb4YWSnVjDjHCFEVAsG6cBKRE3DP2k5WBOdiD/OZaL8vyPBXvZ4eYAfBgU6c3guEdUrhhGiZkIIgT/jb2BNdCJiErM17YMDnTF9gB8e8nYwYnVE1JwxjBA1cSq1wO6zGYiISsS59LLZjM1MJHisixte6ueHAFdbI1dIRM0dwwhRE6VSC+z4Nx2fH0hAwrV8AICVuSmeedgTU/r6wt3OysgVEhGVYRghamJKVGpsP52OLw8mIOlG2URlMkszvNDbB5N6ecPe2sLIFRIRaWMYIWoilKUq/HLiCiKiEnHl1m0AgNzKHFP6+GBib2/IOEcIETVQDCNEjVxRiQo//p2KtdFJyFSUzZba0sYCU/r64rmeXrCR8seciBo2/pYiaqQKlKX4/mgKvvozGTfyy24s6SqzxEv9ffHMQ61hZcGJyoiocWAYIWpkcgqLsSHmMjbEXEZOYQkAwMPeCi8P8MeTwe6QmjGEEFHjwjBC1Ehk5hbh6z+TsPnvVBTembLdt6U1Xh7oj1Fd3GDO2VKJqJFiGCFq4JJvFGBtdKLWlO3tW8kwc6A/hnV0hakJZ0slosaNYYSogTqXnouIqETsOpMB9Z0p2x/2ccDLA/zQv60Tp2wnoiaDYYSogfk7+Sa+jEpAVNx1TdvgQGe8PNAPwV6csp2Imh6GEaIGQAiBg3HX8OXBRBxPuQUAMJEAjwS5YcYAP7RrxbtVE1HTxTBCZERqtcCec5n4/EACzmeU3TfGwtQETwZ74KV+vvBuaW3kComIDI9hhMgISlRq7Pg3HRFRibiUVXbfmBYWpniupxcm9/GBi8zSyBUSEdUfhhGiepSvLMWWv1Px7eFkpOeWzZZqKzXDpN7eeLG3D+8bQ0TNEsMIUT3IKyrBukNJ2BBzGXlFpQCAljZSTOrlhedDvCG34n1jiKj5YhghMrCDcdcw7+d/kF1QDADwdbLGtL6+GN3VHZbmnC2ViIhhhMhALt8owJdRCfj5+BUAZbOlvhEWgLAOrjDhRGVERBoMI0R1LC4zD18cTMCOf9M1k5U917M13hnZnmdCiIgqwTBCVEf+vZKD1QcSsPd8lqZtUKAzZg70R7CXvRErIyJq2BhGiGrp7+SbWH0wAYculc2YKpEAwzu64uUB/ujoLjdydUREDR/DCFENCCFwOOEGPj+QgL+TbwIATE0kGNXZDS8P9IO/s62RKyQiajwYRoj0FJNwAyv2xuFkag6AuzOmzujvh9aOLYxbHBFRI8QwQlQNQggcir+BiKgEHE0qOxMiNTPBsz1a46V+fnCVc8ZUIqKaYhghqkKJSo3f/0nHukNJuJiZB6DsTMizPVrj5QF+cOa07UREtcYwQlSJvKISbPk7Dd8eSUbGnWnbW1iY4pmHWmNKXx+42VkZuUIioqaDYYToHlmKInx7JBmb/0rVTNvuZCvFpF7eeK6HF+QtOG07EVFdYxghAnBNUYRP98fj5+NpKFGVzVTm52SNaf3Kpm2XmnGyMiIiQ2EYoWatQFmKNdGJ+PrPZNwuUQEAHvK2x0v9/DAo0JnTthMR1QOGEWq2zqcrMOvHk0i6XgAA6NraDvOHBaKHr6ORKyMial5MarPx0qVLIZFIMGfOnCrX27p1KwIDA2FpaYlOnTph165dtXlbolq5WVCMd387i0dXH0bS9QK4yiyx5rlg/HdGLwYRIiIjqHEYOXbsGNauXYugoKAq14uJicG4ceMwefJknDp1CqNHj8bo0aNx9uzZmr41UY0oS1X46lAS+i8/iE1HU6BSCwzv6Ipds/tiWEdXSCS8JENEZAwSIYTQd6P8/Hx069YNX375JT744AN06dIFq1atqnTdsWPHoqCgADt27NC09ezZE126dMGaNWsq3UapVEKpVGoeKxQKeHp6Ijc3FzKZTN9yqZkTQmDPuSyE776AlOxCAEAHNxneGdkeIX48E0JEZCgKhQJyufyB3981OjMyc+ZMjBw5EqGhoQ9cNzY2tsJ6YWFhiI2N1blNeHg45HK5ZvH09KxJmdTMqdUCu89k4LHVRzD9+xNIyS6Ek60Uy54Kwv9m9WEQISJqIPTuwLplyxacPHkSx44dq9b6mZmZcHFx0WpzcXFBZmamzm0WLFiAuXPnah6Xnxkhqo7iUjV+O30Va6ITNZ1TLc1NMLWvL6b394O1lP22iYgaEr1+K6elpWH27NmIjIyEpaXhpsGWSqWQSqUGe31qmopKVNjydyrWHUpC+p1ZU2WWZpjUyxuTevvAwdrCyBUSEVFl9AojJ06cwLVr19CtWzdNm0qlwqFDh7B69WoolUqYmmpPDuXq6oqsrCyttqysLLi6utaibKK7lKUq/HwsDasPJiBLUdbXyMlWiil9fPBsj9awteSsqUREDZleYWTw4ME4c+aMVtsLL7yAwMBAvPXWWxWCCACEhIRg//79WsN/IyMjERISUrOKie7IKSzG5r9TsTEmBZmKsjMhbnJLzBjojzHBHrA056ypRESNgV5hxNbWFh07dtRqs7a2hqOjo6Z9woQJcHd3R3h4OABg9uzZ6N+/P1auXImRI0diy5YtOH78ONatW1dHu0DNTdL1fHx7JBm/nriqmTXV2VaKWYP8MfYhT07dTkTUyNR5T77U1FSYmNwdpNOrVy9s3rwZ77zzDt5++220adMGv/32W4VQQ1QVIQRiErPxzeFkHLh4TdPerpUMk/v44NHOrRhCiIgaqRrNM1LfqjtOmZoeZakK20+n49vDybiYmQcAkEiAwYHOeLGPD0J8HTlZGRFRA1Xd72+OcaQG6Ua+Et8fTcH3R1NwI78YAGBlboox3T3wQm8f+LS0NnKFRERUVxhGqEGJz8rDV38m4bfT6SguVQMAWsktMbGXN8Y91BryFhwZQ0TU1DCMUINwKvUWvoxKROT5u8PAO3vaYXIfHwzv6Apz01rd05GIiBowhhEyKkVRCd7//Tx+OXEFQFl/kKHtXTCtny+6tbZnfxAiomaAYYSMQqUW+PZwMiKiE3GzoBgSCfBkNw9M7+8Lf2dbY5dHRET1iGGE6l1uYQlm/3QKUXHXAQC+La3xn6eC8JC3g5ErIyIiY2AYoXpzTVGE74+mYH3MZeQVlcLS3ATvPdIBT3f3gBn7hBARNVsMI2RQQgicSsvBhiOXsftsBkpUZdPatHWxwcoxXdDJQ27kComIyNgYRsggSlVq7DqbiW/+TMI/V3I17cFe9nixd9kIGRMTdk4lIiKGEapjiqIS/HwsDeuPXMbVnNsAAAszEzzW2Q2TenmjozvPhBARkTaGEaoTSdfzsSHmMn45cQWFxWU3r3O0tsCEEG8817M1HG2kRq6QiIgaKoYRqjEhBA7F38D6I8makTFAWX+QF3r74PGu7rA0583riIioagwjpDdlqQq/nLiCbw8nI/F6AYC7N6+b1MsHvf158zoiIqo+hhGqtqISFX78OxVro5OQqSgCANhIzTCmuwcmhnjDmzevIyKiGmAYoQfKUhThq0NJ2HriCnJvlwAou3nd1L6+GNPdA7aWvHkdERHVHMMIVWnbqSuY/+sZKO/cQdfD3gozBvjhqWAPSM3YH4SIiGqPYYSq9OuJq1CWqtHZQ47ZoW3Qv60zTDk/CBER1SGGEapSiarsjMiUvr4YFOhi5GqIiKgp4g1BqEqibPZ2ng0hIiKDYRihKqnupBETDtUlIiIDYRihKqnU5WHEyIUQEVGTxTBCVVLfOTPCyzRERGQoDCNUJc2ZEYYRIiIyEIYRqpK6vAMr+4wQEZGBMIxQldRqXqYhIiLDYhihKpWPpuGJESIiMhSGEaqS5swI0wgRERkIwwhVScXRNEREZGAMI1Sl8qG9HE1DRESGwjBCVVKX3ZqGl2mIiMhgGEaoSndnYGUYISIiw9ArjERERCAoKAgymQwymQwhISHYvXu3zvU3bNgAiUSitVhaWta6aKo/mnvTMLYSEZGBmOmzsoeHB5YuXYo2bdpACIHvvvsOo0aNwqlTp9ChQ4dKt5HJZIiLi9M8lvB/2I0K5xkhIiJD0yuMPProo1qPP/zwQ0RERODo0aM6w4hEIoGrq2vNKySj0tybhiGSiIgMpMYn31UqFbZs2YKCggKEhIToXC8/Px9eXl7w9PTEqFGjcO7cuQe+tlKphEKh0FrIOHhvGiIiMjS9w8iZM2dgY2MDqVSK6dOnY9u2bWjfvn2l6wYEBODbb7/F9u3b8f3330OtVqNXr164cuVKle8RHh4OuVyuWTw9PfUtk+pI+b1p2IGViIgMRSLEnfPw1VRcXIzU1FTk5ubil19+wddff43o6GidgeReJSUlaNeuHcaNG4clS5boXE+pVEKpVGoeKxQKeHp6Ijc3FzKZTJ9yqZbavfsHbpeocOiNgWjt2MLY5RARUSOiUCggl8sf+P2tV58RALCwsIC/vz8AIDg4GMeOHcOnn36KtWvXPnBbc3NzdO3aFQkJCVWuJ5VKIZVK9S2NDICjaYiIyNBq/RWjVqu1zmJURaVS4cyZM2jVqlVt35bqieB08EREZGB6nRlZsGABhg8fjtatWyMvLw+bN29GVFQU9uzZAwCYMGEC3N3dER4eDgB4//330bNnT/j7+yMnJwfLly9HSkoKpkyZUvd7Qgah4o3yiIjIwPQKI9euXcOECROQkZEBuVyOoKAg7NmzB0OGDAEApKamwuSe8/m3bt3C1KlTkZmZCXt7ewQHByMmJqZa/UvI+IQQmg6snB+GiIgMRe8OrMZQ3Q4wVLdUagG/t3cBAE6+OwQO1hZGroiIiBqT6n5/s1si6VR+iQbgZRoiIjIchhHSSX3PSTOOpiEiIkPhVwzpdG8Y4WgaIiIyFIYR0uneyzScgZWIiAyFYYR0Uqvv/p1hhIiIDIVhhHRS8TINERHVA4YR0kmrAyuzCBERGQjDCOmkvtNnxETCSc+IiMhwGEZIJ81N8hhEiIjIgBhGSKfy0TQmvEZDREQGxDBCOpWPpuHsq0REZEgMI6RTeQdWjqQhIiJDYhghne72GTFyIURE1KQxjJBOavYZISKiesAwQjqVnxlhnxEiIjIkhhHSiaNpiIioPjCMkE7lE7DyzAgRERkSwwjpVH5mhKNpiIjIkBhGSKfyPiM8MUJERIbEMEI6qXlmhIiI6gHDCOmkuUzDUyNERGRADCOk050swtE0RERkUAwjpJOa84wQEVE9YBghncov0zCLEBGRITGMkE4q3iiPiIjqAcMI6cTRNEREVB8YRkgnTQdWXqchIiIDYhghnTgDKxER1QeGEdKpfDQNswgRERkSwwjppLlrLy/TEBGRATGMkE5qjqYhIqJ6wDBCOjGMEBFRfdArjERERCAoKAgymQwymQwhISHYvXt3ldts3boVgYGBsLS0RKdOnbBr165aFUz1R6Uu+5OXaYiIyJD0CiMeHh5YunQpTpw4gePHj2PQoEEYNWoUzp07V+n6MTExGDduHCZPnoxTp05h9OjRGD16NM6ePVsnxZNhqdXswEpERIYnEeLOufgacnBwwPLlyzF58uQKz40dOxYFBQXYsWOHpq1nz57o0qUL1qxZU+33UCgUkMvlyM3NhUwmq025pIcf/07Fgv+eQWg7Z3w98SFjl0NERI1Mdb+/a9xnRKVSYcuWLSgoKEBISEil68TGxiI0NFSrLSwsDLGxsVW+tlKphEKh0Fqo/nE0DRER1Qe9w8iZM2dgY2MDqVSK6dOnY9u2bWjfvn2l62ZmZsLFxUWrzcXFBZmZmVW+R3h4OORyuWbx9PTUt0yqA4IdWImIqB7oHUYCAgJw+vRp/PXXX5gxYwYmTpyI8+fP12lRCxYsQG5urmZJS0ur09en6tGcGWEYISIiAzLTdwMLCwv4+/sDAIKDg3Hs2DF8+umnWLt2bYV1XV1dkZWVpdWWlZUFV1fXKt9DKpVCKpXqWxrVMRXvTUNERPWg1vOMqNVqKJXKSp8LCQnB/v37tdoiIyN19jGhhkVz115mESIiMiC9zowsWLAAw4cPR+vWrZGXl4fNmzcjKioKe/bsAQBMmDAB7u7uCA8PBwDMnj0b/fv3x8qVKzFy5Ehs2bIFx48fx7p16+p+T6jOqQQv0xARkeHpFUauXbuGCRMmICMjA3K5HEFBQdizZw+GDBkCAEhNTYWJyd2TLb169cLmzZvxzjvv4O2330abNm3w22+/oWPHjnW7F2QQmhlYeZmGiIgMSK8w8s0331T5fFRUVIW2MWPGYMyYMXoVRQ2D5jINz4wQEZEB8d40pJNmOniGESIiMiCGEdJJ02eEWYSIiAyIYYR0ujuahmmEiIgMh2GEdOJoGiIiqg8MI6QTR9MQEVF9YBghnTiahoiI6gPDCOlUPppGwjMjRERkQAwjpJPmMg0/JUREZED8miGdVBxNQ0RE9YBhhHRSczQNERHVA4YR0omjaYiIqD4wjJBO5ZdpeGaEiIgMiWGEdNLcm4ZnRoiIyIAYRkgnjqYhIqL6wK8Z0knTgZVnRoiIyIAYRkgnFWdgJSKiesAwQjrxzAgREdUHhhHSiaNpiIioPjCMkE7lo2k4zwgRERkSwwjpJDiahoiI6gG/ZkgnFfuMEBFRPWAYIZ00fUYYRoiIyIAYRkinu5OeMYwQEZHhMIyQThxNQ0RE9YFhhHS6k0U4moaIiAyKYYR0Uqs5moaIiAyPXzOkU/loGgnPjBARkQExjJBOmjMjDCNERGRADCOkk4qjaYiIqB4wjJBO6jvTwXM0DRERGRLDCOmkmWeEl2mIiMiAGEZIp7szsBq5ECIiatL0CiPh4eF46KGHYGtrC2dnZ4wePRpxcXFVbrNhwwZIJBKtxdLSslZFU/3Q3JuGaYSIiAxIrzASHR2NmTNn4ujRo4iMjERJSQmGDh2KgoKCKreTyWTIyMjQLCkpKbUqmurH3XlGGEaIiMhwzPRZ+Y8//tB6vGHDBjg7O+PEiRPo16+fzu0kEglcXV2r/T5KpRJKpVLzWKFQ6FMm1ZHyGVh5ozwiIjKkWvUZyc3NBQA4ODhUuV5+fj68vLzg6emJUaNG4dy5c1WuHx4eDrlcrlk8PT1rUybVkIpnRoiIqB7UOIyo1WrMmTMHvXv3RseOHXWuFxAQgG+//Rbbt2/H999/D7VajV69euHKlSs6t1mwYAFyc3M1S1paWk3LpFooH03DLEJERIak12Wae82cORNnz57F4cOHq1wvJCQEISEhmse9evVCu3btsHbtWixZsqTSbaRSKaRSaU1LozpydzQN0wgRERlOjcLIrFmzsGPHDhw6dAgeHh56bWtubo6uXbsiISGhJm9N9UjNGViJiKge6HWZRgiBWbNmYdu2bThw4AB8fHz0fkOVSoUzZ86gVatWem9L9au8AyvDCBERGZJeZ0ZmzpyJzZs3Y/v27bC1tUVmZiYAQC6Xw8rKCgAwYcIEuLu7Izw8HADw/vvvo2fPnvD390dOTg6WL1+OlJQUTJkypY53heoaL9MQEVF90CuMREREAAAGDBig1b5+/XpMmjQJAJCamgoTk7snXG7duoWpU6ciMzMT9vb2CA4ORkxMDNq3b1+7ysng1JyBlYiI6oFeYUTc6UNQlaioKK3Hn3zyCT755BO9iqKGgXftJSKi+sB705BOvExDRET1gWGEdBLswEpERPWAYYR04mUaIiKqDwwjpFP5ZRpepSEiIkNiGKFKlY+kAQBTphEiIjIghhGqlOqekVO8TENERIbEMEKVUt8TRkwYRoiIyIAYRqhSavXdv/MyDRERGRLDCFXq3ss0nGeEiIgMiWGEKqVS33uZxoiFEBFRk8evGaoUR9MQEVF9YRihSqk5moaIiOoJwwhVqrzPiEQCSHhmhIiIDIhhhCpVPpqGnVeJiMjQGEaoUpr70jCMEBGRgZkZuwAyLrVaIL+4FIrbJVDcLkXu7RIoikqQml0IgCNpiIjI8BhGmoCiEhUURSVQ3C5B7u07wULzuASKolLkFpa15WqeKwseeUUluGfgTAUtLPgRISIiw+I3TQOgUgvkFWmfmci9XaIJFbn3nbXQChm3S1Bcqn7wmzyAhZkJ5FbmkFmalf1pZQ6ZpTlGdGpVB3tIRESkG8NIHSsqUSG7oBg38pS4nqfEtTwlbuQrkVNYMUgo7gSOPGVprd9XIgFkluaQWd0JE5bmd/9sURYyZFZ328r+bqYJHZbmpnWw90RERPpjGKmBG/lKXMhQ4EKGAhcz8pB6sxA38pXIzi+uVbCwMje9c1bC7L7QcDdMaAeKu2cxbCzMeEM7IiJqlBhG7lGqUuNWYYkmWNzIV95ZipGdr0SmoggXM/NwPU9Z5euYm0rgaC2Fk60UzrZlf8pbmGufsbg/ZFiaw8KMvUWJiKj5adZh5LuYyzifrsDl7AKkZBciK68IoorOnOUkEsDb0RrtWtki0FUGPycbONlK4WhjgZY2UsgszThRGBERUTU16zDy31NX8U9ajlabRAI4tLDQBAtHGykcrS3gZCtFSxsLtHGxRYCLLaylzfrQERER1Zlm/Y06JtgDgwKc4d2yBVo7tIC7vRUcraW8FwsREVE9atZh5LmeXsYugYiIqNljj0kiIiIyKoYRIiIiMiqGESIiIjIqhhEiIiIyKoYRIiIiMiqGESIiIjIqvcJIeHg4HnroIdja2sLZ2RmjR49GXFzcA7fbunUrAgMDYWlpiU6dOmHXrl01LpiIiIiaFr3CSHR0NGbOnImjR48iMjISJSUlGDp0KAoKCnRuExMTg3HjxmHy5Mk4deoURo8ejdGjR+Ps2bO1Lp6IiIgaP4kQ1bkbS+WuX78OZ2dnREdHo1+/fpWuM3bsWBQUFGDHjh2atp49e6JLly5Ys2ZNtd5HoVBALpcjNzcXMpmspuUSERFRParu93et+ozk5uYCABwcHHSuExsbi9DQUK22sLAwxMbG6txGqVRCoVBoLURERNQ01TiMqNVqzJkzB71790bHjh11rpeZmQkXFxetNhcXF2RmZurcJjw8HHK5XLN4enrWtEwiIiJq4GocRmbOnImzZ89iy5YtdVkPAGDBggXIzc3VLGlpaXX+HkRERNQw1OhGebNmzcKOHTtw6NAheHh4VLmuq6srsrKytNqysrLg6uqqcxupVAqpVFqT0oiIiKiR0SuMCCHwyiuvYNu2bYiKioKPj88DtwkJCcH+/fsxZ84cTVtkZCRCQkL0el8A7DtCRETUiJR/bz9wrIzQw4wZM4RcLhdRUVEiIyNDsxQWFmrWef7558X8+fM1j48cOSLMzMzEihUrxIULF8TChQuFubm5OHPmTLXfNy0tTQDgwoULFy5cuDTCJS0trcrveb2G9kokkkrb169fj0mTJgEABgwYAG9vb2zYsEHz/NatW/HOO+/g8uXLaNOmDZYtW4YRI0ZU922hVquRnp4OW1tbnTXUhEKhgKenJ9LS0jhk+D48NpXjcakcj0vleFwqx+OiW1M7NkII5OXlwc3NDSYmurup1mqekcaO85foxmNTOR6XyvG4VI7HpXI8Lro112PDe9MQERGRUTGMEBERkVE16zAilUqxcOFCDiOuBI9N5XhcKsfjUjkel8rxuOjWXI9Ns+4zQkRERMbXrM+MEBERkfExjBAREZFRMYwQERGRUTGMEBERkVExjBAREZFRNesw8sUXX8Db2xuWlpbo0aMH/v77b2OXVK8WLVoEiUSitQQGBmqeLyoqwsyZM+Ho6AgbGxs8+eSTFe7A3BQcOnQIjz76KNzc3CCRSPDbb79pPS+EwHvvvYdWrVrBysoKoaGhiI+P11rn5s2bGD9+PGQyGezs7DB58mTk5+fX417UvQcdl0mTJlX4/AwbNkxrnaZ4XMLDw/HQQw/B1tYWzs7OGD16NOLi4rTWqc7PTmpqKkaOHIkWLVrA2dkZb7zxBkpLS+tzV+pUdY7LgAEDKnxmpk+frrVOUzsuABAREYGgoCDIZDLIZDKEhIRg9+7dmueb4+flfs02jPz000+YO3cuFi5ciJMnT6Jz584ICwvDtWvXjF1averQoQMyMjI0y+HDhzXPvfbaa/j999+xdetWREdHIz09HU888YQRqzWMgoICdO7cGV988UWlzy9btgyfffYZ1qxZg7/++gvW1tYICwtDUVGRZp3x48fj3LlziIyMxI4dO3Do0CFMmzatvnbBIB50XABg2LBhWp+fH3/8Uev5pnhcoqOjMXPmTBw9ehSRkZEoKSnB0KFDUVBQoFnnQT87KpUKI0eORHFxMWJiYvDdd99hw4YNeO+994yxS3WiOscFAKZOnar1mVm2bJnmuaZ4XADAw8MDS5cuxYkTJ3D8+HEMGjQIo0aNwrlz5wA0z89LBdW/Z2/T8vDDD4uZM2dqHqtUKuHm5ibCw8ONWFX9WrhwoejcuXOlz+Xk5Ahzc3OxdetWTduFCxcEABEbG1tPFdY/AGLbtm2ax2q1Wri6uorly5dr2nJycoRUKhU//vijEEKI8+fPCwDi2LFjmnV2794tJBKJuHr1ar3Vbkj3HxchhJg4caIYNWqUzm2aw3ERQohr164JACI6OloIUb2fnV27dgkTExORmZmpWSciIkLIZDKhVCrrdwcM5P7jIoQQ/fv3F7Nnz9a5TXM4LuXs7e3F119/zc/LHc3yzEhxcTFOnDiB0NBQTZuJiQlCQ0MRGxtrxMrqX3x8PNzc3ODr64vx48cjNTUVAHDixAmUlJRoHaPAwEC0bt26WR2j5ORkZGZmah0HuVyOHj16aI5DbGws7Ozs0L17d806oaGhMDExwV9//VXvNdenqKgoODs7IyAgADNmzEB2drbmueZyXHJzcwEADg4OAKr3sxMbG4tOnTrBxcVFs05YWBgUCoXmf8uN3f3HpdwPP/yAli1bomPHjliwYAEKCws1zzWH46JSqbBlyxYUFBQgJCSEn5c7zIxdgDHcuHEDKpVK6x8WAFxcXHDx4kUjVVX/evTogQ0bNiAgIAAZGRlYvHgx+vbti7NnzyIzMxMWFhaws7PT2sbFxQWZmZnGKdgIyve1ss9K+XOZmZlwdnbWet7MzAwODg5N+lgNGzYMTzzxBHx8fJCYmIi3334bw4cPR2xsLExNTZvFcVGr1ZgzZw569+6Njh07AkC1fnYyMzMr/UyVP9fYVXZcAODZZ5+Fl5cX3Nzc8O+//+Ktt95CXFwc/vvf/wJo2sflzJkzCAkJQVFREWxsbLBt2za0b98ep0+fbvafF6CZhhEqM3z4cM3fg4KC0KNHD3h5eeHnn3+GlZWVESujxuCZZ57R/L1Tp04ICgqCn58foqKiMHjwYCNWVn9mzpyJs2fPavW1It3H5d7+Qp06dUKrVq0wePBgJCYmws/Pr77LrFcBAQE4ffo0cnNz8csvv2DixImIjo42dlkNRrO8TNOyZUuYmppW6K2clZUFV1dXI1VlfHZ2dmjbti0SEhLg6uqK4uJi5OTkaK3T3I5R+b5W9VlxdXWt0PG5tLQUN2/ebFbHytfXFy1btkRCQgKApn9cZs2ahR07duDgwYPw8PDQtFfnZ8fV1bXSz1T5c42ZruNSmR49egCA1memqR4XCwsL+Pv7Izg4GOHh4ejcuTM+/fTTZv95Kdcsw4iFhQWCg4Oxf/9+TZtarcb+/fsREhJixMqMKz8/H4mJiWjVqhWCg4Nhbm6udYzi4uKQmprarI6Rj48PXF1dtY6DQqHAX3/9pTkOISEhyMnJwYkTJzTrHDhwAGq1WvPLtjm4cuUKsrOz0apVKwBN97gIITBr1ixs27YNBw4cgI+Pj9bz1fnZCQkJwZkzZ7TCWmRkJGQyGdq3b18/O1LHHnRcKnP69GkA0PrMNLXjootarYZSqWy2n5cKjN2D1li2bNkipFKp2LBhgzh//ryYNm2asLOz0+qt3NS9/vrrIioqSiQnJ4sjR46I0NBQ0bJlS3Ht2jUhhBDTp08XrVu3FgcOHBDHjx8XISEhIiQkxMhV1728vDxx6tQpcerUKQFAfPzxx+LUqVMiJSVFCCHE0qVLhZ2dndi+fbv4999/xahRo4SPj4+4ffu25jWGDRsmunbtKv766y9x+PBh0aZNGzFu3Dhj7VKdqOq45OXliXnz5onY2FiRnJws9u3bJ7p16ybatGkjioqKNK/RFI/LjBkzhFwuF1FRUSIjI0OzFBYWatZ50M9OaWmp6Nixoxg6dKg4ffq0+OOPP4STk5NYsGCBMXapTjzouCQkJIj3339fHD9+XCQnJ4vt27cLX19f0a9fP81rNMXjIoQQ8+fPF9HR0SI5OVn8+++/Yv78+UIikYi9e/cKIZrn5+V+zTaMCCHE559/Llq3bi0sLCzEww8/LI4ePWrskurV2LFjRatWrYSFhYVwd3cXY8eOFQkJCZrnb9++LV5++WVhb28vWrRoIR5//HGRkZFhxIoN4+DBgwJAhWXixIlCiLLhve+++65wcXERUqlUDB48WMTFxWm9RnZ2thg3bpywsbERMplMvPDCCyIvL88Ie1N3qjouhYWFYujQocLJyUmYm5sLLy8vMXXq1Aphvikel8qOCQCxfv16zTrV+dm5fPmyGD58uLCyshItW7YUr7/+uigpKannvak7Dzouqampol+/fsLBwUFIpVLh7+8v3njjDZGbm6v1Ok3tuAghxIsvvii8vLyEhYWFcHJyEoMHD9YEESGa5+flfhIhhKi/8zBERERE2pplnxEiIiJqOBhGiIiIyKgYRoiIiMioGEaIiIjIqBhGiIiIyKgYRoiIiMioGEaIiIjIqBhGiIiIyKgYRoiIiMioGEaIiIjIqBhGiIiIyKj+H9WF+rdxKpajAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df_results[\"time\"].plot(title=\"Time of labeling\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the records that are part of the prior knowledge are included in the results. To obtain the labels use the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0      0\n",
       "1      0\n",
       "2      0\n",
       "3      0\n",
       "4      0\n",
       "      ..\n",
       "317    0\n",
       "318    0\n",
       "319    0\n",
       "320    0\n",
       "321    1\n",
       "Name: label, Length: 322, dtype: Int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_results[\"label\"]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For normal reviews, the state also contains the ranking of the last iteration of the machine learning model. To get these, use the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>record_id</th>\n",
       "      <th>ranking</th>\n",
       "      <th>classifier</th>\n",
       "      <th>querier</th>\n",
       "      <th>balancer</th>\n",
       "      <th>feature_extractor</th>\n",
       "      <th>training_set</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [record_id, ranking, classifier, querier, balancer, feature_extractor, training_set, time]\n",
       "Index: []"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with asr.open_state(\"example.asreview\") as state:\n",
    "    last_ranking = state.get_last_ranking_table()\n",
    "\n",
    "last_ranking"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "asreview-dev",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}