its_sample_annotate.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "0de7c6a9-e397-45bf-9c53-d75ab7843348",
      "metadata": {
        "id": "0de7c6a9-e397-45bf-9c53-d75ab7843348"
      },
      "source": [
        "## ‘What Destroys a Person When that Person Appears to Be Destroying Himself?’"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "af4866de-0394-4597-98a4-e3c434c0e49e",
      "metadata": {
        "id": "af4866de-0394-4597-98a4-e3c434c0e49e"
      },
      "source": [
        "_Imports, re-indexes by date, cleans, reduces, restricts by timeframe; permits regex pattern-matched purposive (Wave 1) and random (Wave 2) sampling and named entity redaction of PushShift .gzip Reddit archives for .xlsx annotation. Computes Cohen's $\\kappa$ post-annotation. Prepares training dataset for annotation post-IAA._"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f147928e-5595-4e4a-ad4e-7cb7527bd940",
      "metadata": {
        "id": "f147928e-5595-4e4a-ad4e-7cb7527bd940"
      },
      "source": [
        "> its_sample_annotate_iaa.ipynb<br>\n",
        "> Simone J. Skeen (07-20-2024)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. [Prepare](#scrollTo=lVsJPtzvMeX6)\n",
        "2. [Pre-annotation](#scrollTo=a629e1bc)\n",
        "3. [Post-annotation](#scrollTo=f0f0da81-88a0-4c6d-976e-fca33d9ba44e)"
      ],
      "metadata": {
        "id": "M6inB6BBqLR6"
      },
      "id": "M6inB6BBqLR6"
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1. Prepare\n",
        "Installs, imports, and downloads requisite models and packages.\n",
        "***"
      ],
      "metadata": {
        "id": "lVsJPtzvMeX6"
      },
      "id": "lVsJPtzvMeX6"
    },
    {
      "cell_type": "code",
      "source": [
        "#%%capture\n",
        "\n",
        "%pip install irrCAC\n",
        "\n",
        "#%pip install openai\n",
        "#%pip install --upgrade openai\n",
        "\n",
        "%python -m spacy download en_core_web_lg --user"
      ],
      "metadata": {
        "id": "pRtfPcXwMkzV"
      },
      "id": "pRtfPcXwMkzV",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Import + calibrate\n",
        "\n",
        "import gzip\n",
        "import json\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import random\n",
        "import re\n",
        "import spacy\n",
        "import warnings\n",
        "\n",
        "from collections import Counter\n",
        "from irrCAC.raw import CAC\n",
        "from sklearn.metrics import cohen_kappa_score\n",
        "\n",
        "spacy.cli.download('en_core_web_lg')\n",
        "\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "InteractiveShell.ast_node_interactivity = 'all'\n",
        "\n",
        "pd.set_option(\n",
        "              'display.max_columns',\n",
        "              None,\n",
        "              )\n",
        "pd.set_option(\n",
        "              'display.max_rows',\n",
        "              None,\n",
        "              )\n",
        "\n",
        "warnings.simplefilter(\n",
        "                      action = 'ignore',\n",
        "                      category = FutureWarning,\n",
        "                      )\n",
        "\n",
        "#!python -m prodigy stats"
      ],
      "metadata": {
        "id": "xAHrfkFoMk7z"
      },
      "id": "xAHrfkFoMk7z",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "drive.mount(\n",
        "            '/content/gdrive/',\n",
        "            force_remount = True,\n",
        "            )\n"
      ],
      "metadata": {
        "id": "va-9C2uXMlCz"
      },
      "id": "va-9C2uXMlCz",
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "id": "a629e1bc",
      "metadata": {
        "id": "a629e1bc"
      },
      "source": [
        "### 2. Pre-annotation\n",
        "Cleans Pushshift archives, samples via PB- and TB-informed regex.\n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "822337f9-96ee-4dec-b15a-066ee88fe1ff",
      "metadata": {
        "tags": [],
        "id": "822337f9-96ee-4dec-b15a-066ee88fe1ff"
      },
      "outputs": [],
      "source": [
        "# Tap archives\n",
        "\n",
        "os.chdir('<my_dir>')\n",
        "#%pwd\n",
        "\n",
        "# Import _posts_\n",
        "\n",
        "wd = '<my_dir>/sw_complete-posts.json.gz'\n",
        "d = pd.DataFrame(json.loads(l) for l in gzip.open(wd, 'rb'))\n",
        "\n",
        "d = d.drop_duplicates(subset = 'id')\n",
        "\n",
        "# Index by post date\n",
        "\n",
        "d['date'] = pd.to_datetime(\n",
        "                           d.created_utc,\n",
        "                           unit = 's',\n",
        "                           )\n",
        "\n",
        "d.set_index(\n",
        "            'date',\n",
        "            drop = False,\n",
        "            inplace = True,\n",
        "            )\n",
        "\n",
        "# Inspect - as needed\n",
        "\n",
        "#d.resample('1M').subreddit.count().plot()\n",
        "#d.shape ### N = 874,269\n",
        "#print(d.columns.tolist())\n",
        "#d.dtypes\n",
        "#d.head(3)\n",
        "#d.tail(3)\n",
        "\n",
        "# Reduce\n",
        "\n",
        "d = d[[\n",
        "       'author',\n",
        "       'created_utc',\n",
        "       'date',\n",
        "       'id',\n",
        "       'num_comments',\n",
        "       'selftext',\n",
        "       'subreddit',\n",
        "       'title',\n",
        "       ]].copy()\n",
        "\n",
        "# Rename\n",
        "\n",
        "d.rename(\n",
        "         columns = {\n",
        "                    'author': 'p_au',\n",
        "                    'created_utc': 'p_utc',\n",
        "                    'date': 'p_date',\n",
        "                    'id': 'p_id',\n",
        "                    'num_comments': 'n_cmnt',\n",
        "                    'selftext': 'text',\n",
        "                    'subreddit': 'subrddt',\n",
        "                    'title': 'p_titl',\n",
        "                    }, inplace = True,\n",
        "        )\n",
        "\n",
        "# Incl target sub/constructs, rationale cols\n",
        "\n",
        "d = d.assign(\n",
        "             lone = ' ',     ### loneliness\n",
        "             recp = ' ',     ### nonreciprocity\n",
        "             tb = ' ',       ### thwarted belonginess (TB)\n",
        "             tb_rtnl = ' ',  ### TB rationale\n",
        "             hate = ' ',     ### self-hatred\n",
        "             libl = ' ',     ### liability\n",
        "             pb = ' ',       ### perceived burdensomeness (PB)\n",
        "             pb_rtnl = ' ',  ### PB rationale\n",
        "             lgth = ' ',     ### log excess length/brevity\n",
        "             )\n",
        "\n",
        "# Drop user deletions\n",
        "\n",
        "d = d[~d['text'].isin(['[deleted]', '[removed]'])]\n",
        "\n",
        "# Verify\n",
        "\n",
        "d.shape ### n = 458,118\n",
        "d.head(3)\n",
        "d.tail(3)\n",
        "print(d.columns.tolist())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6474b4bc-d8e6-48c7-b807-ab734cc0caa0",
      "metadata": {
        "tags": [],
        "id": "6474b4bc-d8e6-48c7-b807-ab734cc0caa0"
      },
      "outputs": [],
      "source": [
        "# Purposive sample: PB\n",
        "\n",
        "'burden\\S*' ### a priori/canonical\n",
        "'unwanted\\S*|useless\\S*|nuisance|leech\\S*|parasit\\S*|piece of shit|hate myself' ### inductively derived\n",
        "\n",
        "pb = re.compile('burden\\S*|unwanted\\S*|useless\\S*|nuisance|leech\\S*|parasit\\S*|piece of shit|hate myself', re.I)\n",
        "\n",
        "d_pb = d.loc[\n",
        "             d['text'].str.contains(\n",
        "                                    pb,\n",
        "                                    regex = True,\n",
        "                                    )\n",
        "            ]\n",
        "\n",
        "# Prevalence\n",
        "\n",
        "d_pb.shape ### n = 54,335"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c6ad12bc-ffb1-46e6-8f46-8be4b2e9fe28",
      "metadata": {
        "id": "c6ad12bc-ffb1-46e6-8f46-8be4b2e9fe28"
      },
      "outputs": [],
      "source": [
        "# Anonymize + export subsample: PB\n",
        "\n",
        "d_pb = d_pb.sample(400)\n",
        "\n",
        "# Redact w/ EntityRecognizer\n",
        "\n",
        "nlp = spacy.load('en_core_web_lg')\n",
        "\n",
        "# Define Fx\n",
        "\n",
        "def redact(p_text):\n",
        "    ne = list(\n",
        "              [\n",
        "               'PERSON',   ### people, including fictional\n",
        "               'NORP',     ### nationalities or religious or political groups\n",
        "               'FAC',      ### buildings, airports, highways, bridges, etc.\n",
        "               'ORG',      ### companies, agencies, institutions, etc.\n",
        "               #'GPE',     ### countries, cities, states\n",
        "               'LOC',      ### non-GPE locations, mountain ranges, bodies of water\n",
        "               'PRODUCT',  ### objects, vehicles, foods, etc. (not services)\n",
        "               'EVENT',    ### named hurricanes, battles, wars, sports events, etc.\n",
        "               ]\n",
        "                )\n",
        "\n",
        "    doc = nlp(p_text)\n",
        "    ne_to_remove = []\n",
        "    final_string = str(p_text)\n",
        "    for sent in doc.ents:\n",
        "        if sent.label_ in ne:\n",
        "            ne_to_remove.append(str(sent.text))\n",
        "    for n in range(len(ne_to_remove)):\n",
        "        final_string = final_string.replace(\n",
        "                                            ne_to_remove[n],\n",
        "                                            '<|PII|>',\n",
        "                                            )\n",
        "    return final_string\n",
        "\n",
        "# Redact\n",
        "\n",
        "d_pb['text'] = d_pb['text'].astype(str).apply(lambda i: redact(i))\n",
        "\n",
        "# Export\n",
        "\n",
        "d_pb.to_excel('d_pb_annotate.xlsx')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4a7998e1-35a9-4217-be8e-3137817c6c07",
      "metadata": {
        "tags": [],
        "id": "4a7998e1-35a9-4217-be8e-3137817c6c07"
      },
      "outputs": [],
      "source": [
        "# Purposive sample: TB\n",
        "\n",
        "'.lone\\S*|isolat\\S*' ### a priori/canonical\n",
        "'withdraw\\S*|alienat\\S*|ostrac\\S*|shun\\S*|abandon\\S*|reject\\S*' ### inductively derived\n",
        "\n",
        "tb = re.compile('.lone\\S*|isolat\\S*|withdraw\\S*|alienat\\S*|ostrac\\S*|shun\\S*|abandon\\S*|reject\\S*', re.I)\n",
        "\n",
        "d_tb = d.loc[\n",
        "             d['text'].str.contains(\n",
        "                                    tb,\n",
        "                                    regex = True,\n",
        "                                    )\n",
        "            ]\n",
        "\n",
        "# Prevalence\n",
        "\n",
        "d_tb.shape ### n = 87,657"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "32b7a609-9cf2-4e7c-b26b-457635dd5b8e",
      "metadata": {
        "id": "32b7a609-9cf2-4e7c-b26b-457635dd5b8e"
      },
      "outputs": [],
      "source": [
        "# Anonymize + export subsample: TB\n",
        "\n",
        "d_tb = d_tb.sample(400)\n",
        "\n",
        "# Redact\n",
        "\n",
        "d_tb['text'] = d_tb['text'].astype(str).apply(lambda i: redact(i))\n",
        "\n",
        "# Export\n",
        "\n",
        "d_tb.to_excel('d_tb_annotate.xlsx')"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "f0f0da81-88a0-4c6d-976e-fca33d9ba44e",
      "metadata": {
        "id": "f0f0da81-88a0-4c6d-976e-fca33d9ba44e"
      },
      "source": [
        "### 3. Post-annotation\n",
        "Computes Cohen's $\\kappa$ cycle x sub/construct to assess IAA. Post-IAA, merges and strctures $\\mathcal{d}$<sub>annotated</sub>.\n",
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Compute IAA"
      ],
      "metadata": {
        "id": "81YWTO6aoerQ"
      },
      "id": "81YWTO6aoerQ"
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6e46b4e9-80e5-42fc-b4c7-4b9b03b0d80d",
      "metadata": {
        "id": "6e46b4e9-80e5-42fc-b4c7-4b9b03b0d80d"
      },
      "outputs": [],
      "source": [
        "# Import annotated data\n",
        "\n",
        "os.chdir('<my_dir>')\n",
        "#%pwd\n",
        "\n",
        "lc = '<my_dir>/d_cycle*_lc.xlsx' ### * = annotation cycle; lc = LMC\n",
        "d_lc = pd.read_excel(lc)\n",
        "#d_lc.dtypes\n",
        "\n",
        "d_lc = d_lc.replace(' ', 0)\n",
        "d_lc.columns = [f'{col}_lc' for col in d_lc.columns]\n",
        "\n",
        "ss = '<my_dir>/d_cycle*_ss.xlsx' ### * = annotation cycle; ss = SJS\n",
        "d_ss = pd.read_excel(ss)\n",
        "\n",
        "d_ss = d_ss.replace(' ', 0)\n",
        "d_ss.columns = [f'{col}_ss' for col in d_ss.columns]\n",
        "\n",
        "# Inspect\n",
        "\n",
        "#d_lc.head(3)\n",
        "#d_ss.head(3)\n",
        "\n",
        "# Merge\n",
        "\n",
        "iaa = pd.merge(\n",
        "               d_lc,\n",
        "               d_ss,\n",
        "               left_index = True,\n",
        "               right_index = True,\n",
        "               )\n",
        "\n",
        "targets = [\n",
        "           'lone_lc',\n",
        "           'lone_ss',\n",
        "           'recp_lc',\n",
        "           'recp_ss',\n",
        "           'tb_lc',\n",
        "           'tb_ss',\n",
        "           'hate_lc',\n",
        "           'hate_ss',\n",
        "           'libl_lc',\n",
        "           'libl_ss',\n",
        "           'pb_lc',\n",
        "           'pb_ss',\n",
        "           ]\n",
        "\n",
        "iaa = iaa[targets].copy()\n",
        "\n",
        "# IAA\n",
        "\n",
        "'Cohen K: Self-hatred (hate)'\n",
        "hate_lc = iaa['hate_lc'].to_numpy()\n",
        "hate_ss = iaa['hate_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(hate_lc, hate_ss)\n",
        "\n",
        "'Cohen K: Liability (libl)'\n",
        "libl_lc = iaa['libl_lc'].to_numpy()\n",
        "libl_ss = iaa['libl_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(libl_lc, libl_ss)\n",
        "\n",
        "'Cohen K: PB (pb)'\n",
        "pb_lc = iaa['pb_lc'].to_numpy()\n",
        "pb_ss = iaa['pb_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(pb_lc, pb_ss)\n",
        "\n",
        "'Cohen K: Loneliness (lone)'\n",
        "lone_lc = iaa['lone_lc'].to_numpy()\n",
        "lone_ss = iaa['lone_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(lone_lc, lone_ss)\n",
        "\n",
        "'Cohen K: Nonreciprocity (recp)'\n",
        "recp_lc = iaa['recp_lc'].to_numpy()\n",
        "recp_ss = iaa['recp_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(recp_lc, recp_ss)\n",
        "\n",
        "'Cohen K: TB (tb)'\n",
        "tb_lc = iaa['tb_lc'].to_numpy()\n",
        "tb_ss = iaa['tb_ss'].to_numpy()\n",
        "\n",
        "cohen_kappa_score(tb_lc, tb_ss)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "7199b3f3-d256-4bcb-869c-6447847598c2",
      "metadata": {
        "id": "7199b3f3-d256-4bcb-869c-6447847598c2"
      },
      "source": [
        "#### Build: $\\mathcal{d}$<sub>annotated</sub>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "85f6bfff-6876-47f2-b5ae-d12cb376e760",
      "metadata": {
        "id": "85f6bfff-6876-47f2-b5ae-d12cb376e760"
      },
      "outputs": [],
      "source": [
        "# Import + calibrate\n",
        "\n",
        "import gzip\n",
        "import json\n",
        "import numpy as np\n",
        "import os\n",
        "import pandas as pd\n",
        "import re\n",
        "import warnings\n",
        "\n",
        "from IPython.core.interactiveshell import InteractiveShell\n",
        "InteractiveShell.ast_node_interactivity = 'all'\n",
        "\n",
        "pd.set_option('display.max_columns', None)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2185f347",
      "metadata": {
        "tags": [],
        "id": "2185f347"
      },
      "outputs": [],
      "source": [
        "# Tap TB cycle w/ IAA benchmark achieved\n",
        "\n",
        "wd_tb = 'C:/Users/sskee/OneDrive/Documents/02_tulane/01_research/tu_ceai/its/submissions/01_savir/data/tb_crosswalk 150+negotiated.xlsx'\n",
        "d_tb = pd.read_excel(wd_tb)\n",
        "\n",
        "d_tb.fillna(\n",
        "            0,\n",
        "            inplace = True,\n",
        "            )\n",
        "\n",
        "# Inspect\n",
        "\n",
        "d_tb.dtypes\n",
        "d_tb.head(3)\n",
        "print(d_tb.columns.tolist())\n",
        "\n",
        "# Transform id of already-annotated posts to NumPy array\n",
        "\n",
        "tb_drop = d_tb['id'].to_numpy()\n",
        "print(tb_drop)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2d579469",
      "metadata": {
        "jupyter": {
          "source_hidden": true
        },
        "tags": [],
        "id": "2d579469"
      },
      "outputs": [],
      "source": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "ba004777-35b7-4b9d-b2e9-1a51a9396340",
      "metadata": {
        "id": "ba004777-35b7-4b9d-b2e9-1a51a9396340"
      },
      "outputs": [],
      "source": [
        "# Shape TB purposive annotation set\n",
        "\n",
        "os.chdir('<my_dir>')\n",
        "#%pwd\n",
        "\n",
        "# Import _posts_\n",
        "\n",
        "wd = '<my_dir>/sw_complete-posts.json.gz'\n",
        "d = pd.DataFrame(json.loads(l) for l in gzip.open(wd, 'rb'))\n",
        "\n",
        "d = d.drop_duplicates(subset = 'id')\n",
        "\n",
        "# Index by post date\n",
        "\n",
        "d['date'] = pd.to_datetime(\n",
        "                           d.created_utc,\n",
        "                           unit = 's',\n",
        "                           )\n",
        "\n",
        "d.set_index(\n",
        "            'date',\n",
        "            drop = False,\n",
        "            inplace = True,\n",
        "            )\n",
        "\n",
        "# Reduce\n",
        "\n",
        "d = d[[\n",
        "       'author',\n",
        "       'created_utc',\n",
        "       'date',\n",
        "       'id',\n",
        "       'num_comments',\n",
        "       'selftext',\n",
        "       'subreddit',\n",
        "       'title',\n",
        "       ]].copy()\n",
        "\n",
        "# Rename\n",
        "\n",
        "d.rename(\n",
        "         columns = {\n",
        "                    'author': 'p_au',\n",
        "                    'created_utc': 'p_utc',\n",
        "                    'date': 'p_date',\n",
        "                    'id': 'p_id',\n",
        "                    'num_comments': 'n_cmnt',\n",
        "                    'selftext': 'text',\n",
        "                    'subreddit': 'subrddt',\n",
        "                    'title': 'p_titl',\n",
        "                    }, inplace = True,\n",
        "        )\n",
        "\n",
        "# Incl target sub/constructs, rationale cols\n",
        "\n",
        "d = d.assign(\n",
        "             lone = ' ',     ### loneliness\n",
        "             recp = ' ',     ### nonreciprocity\n",
        "             tb = ' ',       ### thwarted belonginess (TB)\n",
        "             tb_rtnl = ' ',  ### TB rationale\n",
        "             hate = ' ',     ### self-hatred\n",
        "             libl = ' ',     ### liability\n",
        "             pb = ' ',       ### perceived burdensomeness (PB)\n",
        "             pb_rtnl = ' ',  ### PB rationale\n",
        "             lgth = ' ',     ### log excess length/brevity\n",
        "             )\n",
        "\n",
        "# Drop user deletions\n",
        "\n",
        "d = d[~d['text'].isin(['[deleted]', '[removed]'])]\n",
        "\n",
        "# Random sample\n",
        "\n",
        "rnd_antt = d.sample(500)\n",
        "\n",
        "# Purposive sample: TB\n",
        "\n",
        "'.lone\\S*|isolat\\S*' ### a priori/canonical\n",
        "'withdraw\\S*|alienat\\S*|ostrac\\S*|shun\\S*|abandon\\S*|reject\\S*' ### inductively derived\n",
        "\n",
        "tb = re.compile('.lone\\S*|isolat\\S*|withdraw\\S*|alienat\\S*|ostrac\\S*|shun\\S*|abandon\\S*|reject\\S*', re.I)\n",
        "\n",
        "d_tb = d.loc[\n",
        "             d['text'].str.contains(\n",
        "                                    tb,\n",
        "                                    regex = True,\n",
        "                                    )\n",
        "            ]\n",
        "\n",
        "# Prevalence\n",
        "\n",
        "d_tb.shape ### n = 87,657\n",
        "\n",
        "# Drop already-annotated n = 50\n",
        "\n",
        "drop = d_tb['id'].isin(tb_drop)\n",
        "d_tb = d_tb[~drop]\n",
        "\n",
        "# Random subset, n = 450\n",
        "\n",
        "tb_antt = d_tb.sample(450)\n",
        "tb_antt.count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "46c7ecd3-9592-40bf-8307-2a362ddb0688",
      "metadata": {
        "id": "46c7ecd3-9592-40bf-8307-2a362ddb0688"
      },
      "outputs": [],
      "source": [
        "# Redact\n",
        "\n",
        "rnd_antt['text'] = rnd_antt['text'].astype(str).apply(lambda i: redact(i))\n",
        "\n",
        "tb_antt['text'] = tb_antt['text'].astype(str).apply(lambda i: redact(i))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0fc423af-ca5d-4dc9-a0dd-e2ff1705b7b5",
      "metadata": {
        "id": "0fc423af-ca5d-4dc9-a0dd-e2ff1705b7b5"
      },
      "outputs": [],
      "source": [
        "# Tap PB cycle w/ IAA benchmark achieved\n",
        "\n",
        "wd_pb = 'C:/Users/sskee/OneDrive/Documents/02_tulane/01_research/tu_ceai/its/submissions/01_savir/data/pb_crosswalk 150+negotiated.xlsx'\n",
        "d_pb = pd.read_excel(wd_pb)\n",
        "\n",
        "d_pb.fillna(\n",
        "            0,\n",
        "            inplace = True,\n",
        "            )\n",
        "\n",
        "# Inspect\n",
        "\n",
        "d_pb.dtypes\n",
        "d_pb.head(3)\n",
        "print(d_pb.columns.tolist())\n",
        "\n",
        "# Transform id of already-annotated posts to NumPy array\n",
        "\n",
        "pb_drop = d_pb['id'].to_numpy()\n",
        "print(pb_drop)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "b31f4faa-d2ae-48e2-9a30-3e8d62b4c258",
      "metadata": {
        "id": "b31f4faa-d2ae-48e2-9a30-3e8d62b4c258"
      },
      "outputs": [],
      "source": [
        "# Shape PB purposive annotation set\n",
        "\n",
        "os.chdir('<my_dir>')\n",
        "#%pwd\n",
        "\n",
        "# Import _posts_\n",
        "\n",
        "wd = '<my_dir>/sw_complete-posts.json.gz'\n",
        "d = pd.DataFrame(json.loads(l) for l in gzip.open(wd, 'rb'))\n",
        "\n",
        "d = d.drop_duplicates(subset = 'id')\n",
        "\n",
        "# Index by post date\n",
        "\n",
        "d['date'] = pd.to_datetime(\n",
        "                           d.created_utc,\n",
        "                           unit = 's',\n",
        "                           )\n",
        "\n",
        "d.set_index(\n",
        "            'date',\n",
        "            drop = False,\n",
        "            inplace = True,\n",
        "            )\n",
        "\n",
        "# Reduce\n",
        "\n",
        "d = d[[\n",
        "       'author',\n",
        "       'created_utc',\n",
        "       'date',\n",
        "       'id',\n",
        "       'num_comments',\n",
        "       'selftext',\n",
        "       'subreddit',\n",
        "       'title',\n",
        "       ]].copy()\n",
        "\n",
        "# Rename\n",
        "\n",
        "d.rename(\n",
        "         columns = {\n",
        "                    'author': 'p_au',\n",
        "                    'created_utc': 'p_utc',\n",
        "                    'date': 'p_date',\n",
        "                    'id': 'p_id',\n",
        "                    'num_comments': 'n_cmnt',\n",
        "                    'selftext': 'text',\n",
        "                    'subreddit': 'subrddt',\n",
        "                    'title': 'p_titl',\n",
        "                    }, inplace = True,\n",
        "        )\n",
        "\n",
        "# Incl target sub/constructs, rationale cols\n",
        "\n",
        "d = d.assign(\n",
        "             lone = ' ',     ### loneliness\n",
        "             recp = ' ',     ### nonreciprocity\n",
        "             tb = ' ',       ### thwarted belonginess (TB)\n",
        "             tb_rtnl = ' ',  ### TB rationale\n",
        "             hate = ' ',     ### self-hatred\n",
        "             libl = ' ',     ### liability\n",
        "             pb = ' ',       ### perceived burdensomeness (PB)\n",
        "             pb_rtnl = ' ',  ### PB rationale\n",
        "             lgth = ' ',     ### log excess length/brevity\n",
        "             )\n",
        "\n",
        "# Drop user deletions\n",
        "\n",
        "d = d[~d['text'].isin(['[deleted]', '[removed]'])]\n",
        "\n",
        "# Purposive sample: PB\n",
        "\n",
        "'burden\\S*' ### a priori/canonical\n",
        "'unwanted\\S*|useless\\S*|nuisance|leech\\S*|parasit\\S*|piece of shit|hate myself' ### inductively derived\n",
        "\n",
        "pb = re.compile('burden\\S*|unwanted\\S*|useless\\S*|nuisance|leech\\S*|parasit\\S*|piece of shit|hate myself', re.I)\n",
        "\n",
        "d_pb = d.loc[\n",
        "             d['text'].str.contains(\n",
        "                                    pb,\n",
        "                                    regex = True,\n",
        "                                    )\n",
        "            ]\n",
        "\n",
        "# Prevalence\n",
        "\n",
        "d_pb.shape ### n = 54,335\n",
        "\n",
        "# Drop already-annotated n = 50\n",
        "\n",
        "drop = d_pb['id'].isin(pb_drop)\n",
        "d_pb = d_pb[~drop]\n",
        "\n",
        "# Random subset, n = 450\n",
        "\n",
        "pb_antt = d_pb.sample(450)\n",
        "pb_antt.count()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "8520ef9f-868f-4f34-8b9a-ad827297263c",
      "metadata": {
        "id": "8520ef9f-868f-4f34-8b9a-ad827297263c"
      },
      "outputs": [],
      "source": [
        "# Redact\n",
        "\n",
        "pb_antt['text'] = pb_antt['text'].astype(str).apply(lambda i: redact(i))\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "134458a7-a5ee-4ae9-92e7-2813c1a40de7",
      "metadata": {
        "id": "134458a7-a5ee-4ae9-92e7-2813c1a40de7"
      },
      "outputs": [],
      "source": [
        "# Append: PB, TB, rnd + IAA annotation benchmark cycle\n",
        "\n",
        "its_annotate = pd.concat(\n",
        "                         [\n",
        "                          pb_antt,  ### PB, regex/purposive\n",
        "                          tb_antt,  ### TB, regex/purposive\n",
        "                          rnd_antt, ### random subsample\n",
        "                          d_cycle3_pb,  ### n = 50 cycle, IAA achieved: PB\n",
        "                          d_cycle3_tb,  ### n = 50 cycle, IAA achieved: TB\n",
        "                          ], ignore_index = True,\n",
        "                         )\n"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "0abb2b3a-bbd8-4d98-b0d5-ec98e02ceb7c",
      "metadata": {
        "id": "0abb2b3a-bbd8-4d98-b0d5-ec98e02ceb7c"
      },
      "source": [
        "> End of its_sample_annotate_iaa.ipynb (07-20-2024)"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.13"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}