preprocess_corpus/generate-TRECVID-VTT-corpus.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2020 Readme\n",
    "\n",
    "TRECVID-VTT\n",
    "\n",
    "Video-to-text dataset\n",
    "\n",
    "This dataset contains short videos (ranging from 3 seconds to 10 seconds) from TRECVID VTT task from 2016 to 2019. There are 7485 videos with captions. Each video has between 2 and 5 captions, which have been written by dedicated annotators. The videos have IDs from 1 to 7485. The videos are given in the following format:\n",
    "\n",
    "1. There are 6475 URLs from Twitter Vine. These URLs are in the vtt_video_urls.txt file. Each line in the file gives the Video ID followed by its URL.\n",
    "2. There are 1010 video files available to download from our Flickr dataset. These videos are in the webm format and have the Creative Commons License.\n",
    "3. The file vtt_ground_truth.txt contains the captions for all videos. Each line in the file gives the Video ID followed by its caption.\n",
    "4. Videos with IDs from 1 to 3528 have between 2 to 5 captions. Videos from 3529 to 7485 have 5 captions each.\n",
    "\n",
    "If you use this dataset, please cite the following paper:\n",
    "\n",
    "@inproceedings{awad2019trecvid,\n",
    "  title={Trecvid 2019: An evaluation campaign to benchmark video activity detection, video captioning and matching, and video search \\& retrieval},\n",
    "  author={Awad, George and Butt, Asad and Curtis, Keith and Lee, Yooyoung and Fiscus, Jonathan and Godil, Afzal and Delgado, Andrew and others},\n",
    "  year={2019}\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate corpus and gruond-truth references of released videos\n",
    "\n",
    "### Corpus file contents\n",
    "0. train_data: captions and idxs of training videos in format [corpus_widxs, vidxs, corpus_pidxs], where:\n",
    "    - corpus_widxs is a list of lists with the index of words in the vocabulary\n",
    "    - vidxs is a list of indexes of video features in the features file\n",
    "    - corpus_pidxs is a list of lists with the index of POS tags in the POS tagging vocabulary\n",
    "1. val_data: same format of train_data. We created this set randomly sampling 20% of videos in the tv2020 released data for training.\n",
    "2. test_data: **None**. The videos for testing have not been released yet.\n",
    "3. vocabulary: in format {'word': count}.\n",
    "4. idx2word: is the vocabulary in format {idx: 'word'}.\n",
    "5. word_embeddings: are the vectors of each word. The i-th row is the word vector of the i-th word in the vocabulary.\n",
    "6. idx2pos: is the vocabulary of POS tagging in format {idx: 'POSTAG'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate split for training and validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count of videos:  7485\n",
      "original count of pairs:  28183\n",
      "\n",
      "My split:\n",
      "count of training pairs:  22491\n",
      "count of validation pairs:  5692\n"
     ]
    }
   ],
   "source": [
    "train_corpus = []\n",
    "with open('../../../data/TRECVID/tv2020/train/vtt_ground_truth.txt') as f:\n",
    "    vidxs, corpus = zip(*[l.strip().split(' ', 1) for l in f.readlines()])\n",
    "\n",
    "print('count of videos: ', len(set(vidxs)))    \n",
    "print('original count of pairs: ', len(vidxs))\n",
    "    \n",
    "## remove vids that has a video with problem\n",
    "import os\n",
    "videos_dir_path = '../../../data/TRECVID/tv2020/train/videos'\n",
    "vidxs_to_delete = [int(filename[6:filename.index('.')]) for filename in os.listdir(videos_dir_path) if not os.path.getsize(os.path.join(videos_dir_path, filename))]\n",
    "\n",
    "import random\n",
    "unique_vidxs = set([int(vidx) for vidx in vidxs]) - set(vidxs_to_delete)\n",
    "random.seed(42)\n",
    "sampled_vidxs = random.sample(vidxs, len(unique_vidxs)//5) # 20% of videos for validation\n",
    "\n",
    "train_vidxs, train_corpus = zip(*[(int(vidx)-1, corpus[i]) for (i, vidx) in enumerate(vidxs) if not vidx in sampled_vidxs])\n",
    "valid_vidxs, valid_corpus = zip(*[(int(vidx)-1, corpus[i]) for (i, vidx) in enumerate(vidxs) if vidx in sampled_vidxs])\n",
    "\n",
    "print('\\nMy split:')\n",
    "print('count of training pairs: ', len(train_vidxs))\n",
    "print('count of validation pairs: ', len(valid_vidxs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discard videos without features (video files that have problem)\n",
    "**IMPORTANT**: I don't need to remove the features of the .h5 file. I only need to remove the indices and captions from corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<KeysViewHDF5 ['c3d_features', 'cnn_features', 'cnn_globals', 'cnn_sem_globals', 'count_features', 'eco_globals', 'frames_tstamp']>\n",
      "(7485, 20, 2048)\n",
      "254  videos to dicard\n",
      "count of training pairs:  21956\n",
      "count of validation pairs:  5634\n"
     ]
    }
   ],
   "source": [
    "import h5py\n",
    "with h5py.File('../../../data/TRECVID/tv2020/features/features_linspace16_20-cnn_globals-cnn_sem_globals-cnn_features-c3d_features-eco_globals.h5', 'r+') as feats_file:\n",
    "    dataset = feats_file['TRECVID-2020']\n",
    "    print(dataset.keys())\n",
    "    cnn_fts = dataset['cnn_features'][...]\n",
    "    print(cnn_fts.shape)\n",
    "    \n",
    "import numpy as np\n",
    "vidxs_to_discard = []\n",
    "for i, v in enumerate(cnn_fts):\n",
    "    if np.all(v == np.zeros((20,2048))):\n",
    "        vidxs_to_discard.append(i)\n",
    "print(len(vidxs_to_discard), ' videos to dicard')\n",
    "\n",
    "train_vidxs, train_corpus = zip(*[(vidx, train_corpus[i]) for (i, vidx) in enumerate(train_vidxs) if not vidx in vidxs_to_discard])\n",
    "valid_vidxs, valid_corpus = zip(*[(vidx, valid_corpus[i]) for (i, vidx) in enumerate(valid_vidxs) if not vidx in vidxs_to_discard])\n",
    "\n",
    "print('count of training pairs: ', len(train_vidxs))\n",
    "print('count of validation pairs: ', len(valid_vidxs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get pretrained embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "400000\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "\n",
    "wordvectors = {}\n",
    "# with open('./glove.42B.300d.txt') as f:\n",
    "with open('./glove.6B.300d.txt') as f:\n",
    "    for line in f:\n",
    "        s = line.strip().split(' ')\n",
    "        if len(s) == 301:\n",
    "            wordvectors[s[0]] = np.array(s[1:], dtype=float)\n",
    "    print(len(wordvectors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine the vocabulary from train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/jeperez/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Avg. count of words per caption: 18.899161960284204\n",
      "Count of unique words:  11634\n",
      "missing word: toddles\n",
      "missing word: jellabiya\n",
      "missing word: gesticulates\n",
      "missing word: socket-outlet\n",
      "missing word: laundromatic\n",
      "missing word: outstretching\n",
      "missing word: peruke\n",
      "missing word: mtv-studio\n",
      "missing word: nba-playoff\n",
      "missing word: france/eu\n",
      "missing word: mewls\n",
      "missing word: tv-studio\n",
      "missing word: cacadoo\n",
      "missing word: air-kiss\n",
      "missing word: grining\n",
      "missing word: puppie\n",
      "missing word: twerking\n",
      "missing word: flic-flacs\n",
      "missing word: hoody\n",
      "missing word: rollick\n",
      "missing word: punchbag\n",
      "missing word: york-player\n",
      "missing word: bar-keeper\n",
      "missing word: overlarge\n",
      "missing word: muscly\n",
      "missing word: sidewards\n",
      "missing word: selfie\n",
      "missing word: one-move\n",
      "missing word: afflick\n",
      "missing word: fur-trimmed\n",
      "missing word: mid-court\n",
      "missing word: carooms\n",
      "missing word: face.\n",
      "missing word: martin-lewis\n",
      "missing word: break-dance\n",
      "missing word: head-butt\n",
      "missing word: venue..\n",
      "missing word: flip-phone\n",
      "missing word: spring-jumper\n",
      "missing word: back-flip\n",
      "missing word: azdrubel\n",
      "missing word: houston-cleveland\n",
      "missing word: re-cycling\n",
      "missing word: mime-game\n",
      "missing word: -a\n",
      "missing word: man-talks\n",
      "missing word: flute-like\n",
      "missing word: ancient-looking\n",
      "missing word: bad-mouthing\n",
      "missing word: break-dancing\n",
      "missing word: course..\n",
      "missing word: unfruitfully\n",
      "missing word: shirt/black\n",
      "missing word: nurse/caregiver\n",
      "missing word: unclasping\n",
      "missing word: red-trim\n",
      "missing word: chairs/sofas\n",
      "missing word: gray/white\n",
      "missing word: road/path\n",
      "missing word: imaichi\n",
      "missing word: large-mouthed\n",
      "missing word: oak-ten\n",
      "missing word: cheerleader-type\n",
      "missing word: long-tusked\n",
      "missing word: grey/white\n",
      "missing word: camera/mic\n",
      "missing word: facks\n",
      "missing word: mattress/foam\n",
      "missing word: inquiringly\n",
      "missing word: hurky\n",
      "missing word: skate-boarder\n",
      "missing word: high-five\n",
      "missing word: pyrographics\n",
      "missing word: face-time\n",
      "missing word: singing..\n",
      "missing word: baaing\n",
      "missing word: silver/black\n",
      "missing word: daces\n",
      "missing word: flower-covered\n",
      "missing word: bug-zapper\n",
      "missing word: team.\n",
      "missing word: gym..\n",
      "missing word: gray-haired\n",
      "missing word: on.d\n",
      "missing word: i-phone\n",
      "missing word: jello-looking\n",
      "missing word: blue/white\n",
      "missing word: dribbling-passing\n",
      "missing word: rods/crosses\n",
      "missing word: balloon-like\n",
      "missing word: oil-tank\n",
      "missing word: white/black\n",
      "missing word: /blue\n",
      "missing word: lightshow\n",
      "missing word: uniform-like\n",
      "missing word: 'hello\n",
      "missing word: cami-print\n",
      "missing word: hee-haw-ing\n",
      "missing word: arena..\n",
      "missing word: santa-style\n",
      "missing word: red/white\n",
      "missing word: howdies\n",
      "missing word: break-dancer\n",
      "missing word: around.\n",
      "missing word: yugyeom\n",
      "missing word: jacket/pants\n",
      "missing word: /red\n",
      "missing word: shorts/black\n",
      "missing word: cinder-block\n",
      "missing word: spring-board\n",
      "missing word: misblows\n",
      "missing word: singing/talking\n",
      "missing word: un-trained\n",
      "missing word: bed..\n",
      "missing word: earing\n",
      "missing word: sneakered\n",
      "missing word: broom/mop\n",
      "missing word: suit/costume\n",
      "missing word: cap/mask\n",
      "missing word: high-top\n",
      "missing word: military-like\n",
      "missing word: statue/award\n",
      "missing word: singing/dancing\n",
      "missing word: sound-over\n",
      "missing word: mouth-over\n",
      "missing word: tictac\n",
      "missing word: bating\n",
      "missing word: flour-laden\n",
      "missing word: chanting.\n",
      "missing word: fort-type\n",
      "missing word: jacket/white\n",
      "missing word: jacket/black\n",
      "missing word: gestures.\n",
      "missing word: hair/bangs\n",
      "missing word: motion..\n",
      "missing word: pyramid-type\n",
      "missing word: cheetah-like\n",
      "missing word: hip-hopping\n",
      "missing word: classroom-like\n",
      "missing word: same-colored\n",
      "missing word: nose-first\n",
      "missing word: half-swallows\n",
      "missing word: 'hit\n",
      "missing word: green/white\n",
      "missing word: thigh-high\n",
      "missing word: soccer-type\n",
      "missing word: monroe-style\n",
      "missing word: ledges/curbs\n",
      "missing word: embg\n",
      "missing word: coats/hoods\n",
      "missing word: crosslegged\n",
      "missing word: blue-shirted\n",
      "missing word: up..\n",
      "missing word: teeth-showing\n",
      "missing word: smiley-face\n",
      "missing word: bug-eyes\n",
      "missing word: room..\n",
      "missing word: re-play\n",
      "missing word: shirt/dress\n",
      "missing word: tan/black\n",
      "missing word: bulls-boston\n",
      "missing word: team/blue\n",
      "missing word: talking/rapping\n",
      "missing word: coffee/hot\n",
      "missing word: face/dark\n",
      "missing word: door..\n",
      "missing word: fire-crackers\n",
      "missing word: skirt/shorts\n",
      "missing word: mouth/big\n",
      "missing word: midi-top\n",
      "missing word: shirt/white\n",
      "missing word: over..\n",
      "missing word: shirt/faded\n",
      "missing word: ..\n",
      "missing word: acrobatics.\n",
      "missing word: lollypop\n",
      "missing word: instructions..\n",
      "missing word: fishtails\n",
      "missing word: play..\n",
      "missing word: usb-iphone\n",
      "missing word: light-show\n",
      "missing word: youngster..\n",
      "missing word: high-fiving\n",
      "missing word: 9stills\n",
      "missing word: snap-scissors\n",
      "missing word: brown-and-white\n",
      "missing word: same-sized\n",
      "missing word: fore-legs\n",
      "missing word: song-line\n",
      "missing word: slam-dunk\n",
      "missing word: singer/performer\n",
      "missing word: bee-like\n",
      "missing word: airport-like\n",
      "missing word: smiling.\n",
      "missing word: non-sense\n",
      "missing word: porta-potty\n",
      "missing word: table-mounted\n",
      "missing word: snow-board\n",
      "missing word: diving/throwing\n",
      "missing word: on-to-off-camera\n",
      "missing word: rooting-on\n",
      "missing word: pass-throw\n",
      "missing word: -up\n",
      "missing word: trying-out\n",
      "missing word: beiber\n",
      "missing word: foul-back\n",
      "missing word: paintballers\n",
      "missing word: box-fight\n",
      "missing word: mid-section\n",
      "missing word: summersaults\n",
      "missing word: back-flips\n",
      "missing word: -s\n",
      "missing word: swing-set\n",
      "missing word: summersault\n",
      "missing word: living/dining\n",
      "missing word: unbuttons\n",
      "missing word: reppin\n",
      "missing word: out-dribbles\n",
      "missing word: 'she\n",
      "missing word: visible..\n",
      "missing word: dining/living\n",
      "missing word: dancing.with\n",
      "missing word: unbuckles\n",
      "missing word: caution-tape\n",
      "missing word: yellow-taped\n",
      "missing word: boom-box\n",
      "missing word: make-shift\n",
      "missing word: over-the-head\n",
      "missing word: v-sign\n",
      "missing word: man-performer\n",
      "missing word: 'watch\n",
      "missing word: twin-bedded\n",
      "missing word: blue-topped\n",
      "missing word: black-jacketed\n",
      "missing word: ear-muffs\n",
      "missing word: leopard-skin\n",
      "missing word: thavasa\n",
      "missing word: passer-bye\n",
      "missing word: 'recovery\n",
      "missing word: gymtyme\n",
      "missing word: bat-man\n",
      "missing word: air-horn\n",
      "missing word: burnt-orange\n",
      "missing word: twerks\n",
      "missing word: stadium-style\n",
      "missing word: dust-covered\n",
      "missing word: many-colored\n",
      "missing word: tear-drops\n",
      "missing word: white-blonde\n",
      "missing word: ochimamiki\n",
      "missing word: man-colored\n",
      "missing word: pro-woman\n",
      "missing word: body-slamming\n",
      "missing word: topgear\n",
      "missing word: cotton-candy\n",
      "missing word: white-fleece\n",
      "missing word: grey-covered\n",
      "missing word: fair-like\n",
      "missing word: tie-dyed\n",
      "missing word: doggie-park\n",
      "missing word: loud-speaking\n",
      "missing word: pony-tailed\n",
      "missing word: boys..\n",
      "missing word: roles.one\n",
      "missing word: boxing..\n",
      "missing word: bang-bang-bang\n",
      "missing word: air-soft\n",
      "missing word: arm-locked\n",
      "missing word: autographic\n",
      "missing word: paddle-powered\n",
      "missing word: bump-like\n",
      "missing word: s___\n",
      "missing word: jacket-shirts\n",
      "missing word: color-changing\n",
      "missing word: gray-hired\n",
      "missing word: whoh\n",
      "missing word: t-line\n",
      "missing word: people..\n",
      "missing word: phones/cameras\n",
      "missing word: 2-bowled\n",
      "missing word: betway\n",
      "missing word: round.s\n",
      "missing word: black/silver\n",
      "missing word: time.s\n",
      "missing word: outdoors.\n",
      "missing word: long-wigged\n",
      "missing word: band.\n",
      "missing word: videoing\n",
      "missing word: torch-lit\n",
      "missing word: guinea-pig\n",
      "missing word: jives\n",
      "missing word: guitar..\n",
      "missing word: red/white/blue\n",
      "missing word: down..\n",
      "missing word: icy..\n",
      "missing word: floor..\n",
      "missing word: mild..\n",
      "missing word: falls.\n",
      "missing word: suv..\n",
      "missing word: coffee-like\n",
      "missing word: trowing\n",
      "missing word: sunsuit\n",
      "missing word: class.\n",
      "missing word: dr.phil\n",
      "missing word: other.in\n",
      "missing word: chest..\n",
      "missing word: shits\n",
      "missing word: vapes\n",
      "missing word: vape\n",
      "missing word: two.takes\n",
      "missing word: three-colored\n",
      "missing word: slip-and-slide\n",
      "missing word: mini-basketball\n",
      "missing word: him.\n",
      "missing word: picture-lined\n",
      "missing word: yellow-spotlighted\n",
      "missing word: card.which\n",
      "missing word: too-low\n",
      "missing word: dog/masks\n",
      "missing word: degrees..\n",
      "missing word: squirter\n",
      "missing word: towel.is\n",
      "missing word: stage.\n",
      "missing word: camera..\n",
      "missing word: green-patterned\n",
      "missing word: got7\n",
      "missing word: eating/drinking\n",
      "missing word: -white\n",
      "missing word: karate-like\n",
      "missing word: animal-costumed\n",
      "missing word: game.\n",
      "missing word: mouth-mounted\n",
      "missing word: xenoverse\n",
      "missing word: pink-clad\n",
      "missing word: questioningly\n",
      "missing word: kerchiefed\n",
      "missing word: manner.\n",
      "missing word: steps.down\n",
      "missing word: performance..\n",
      "missing word: hole..\n",
      "missing word: blow-dried\n",
      "missing word: selfies\n",
      "missing word: green-screen\n",
      "missing word: blackboard.showing\n",
      "missing word: body.\n",
      "missing word: 'snapchat\n",
      "missing word: tee-shirt\n",
      "missing word: red/orange\n",
      "missing word: badger-like\n",
      "missing word: marmet\n",
      "missing word: agame\n",
      "missing word: bakset\n",
      "missing word: /gliding\n",
      "missing word: frontwards\n",
      "missing word: tv.\n",
      "missing word: 'humping\n",
      "missing word: brexit\n",
      "missing word: floot\n",
      "missing word: mint-green\n",
      "missing word: kitchen/dinning\n",
      "missing word: silver-back\n",
      "missing word: gorila\n",
      "missing word: b.u.t\n",
      "missing word: co-contestant\n",
      "missing word: 'prancing\n",
      "missing word: high-stepping\n",
      "missing word: injured..\n",
      "missing word: combustable\n",
      "missing word: liquid-like\n",
      "missing word: hand-stands\n",
      "missing word: re-boarding\n",
      "missing word: cross-arm\n",
      "missing word: cross-leg\n",
      "missing word: tree.\n",
      "missing word: rabbit-like\n",
      "missing word: drivers\\\n",
      "missing word: no2rouhani\n",
      "missing word: oves\n",
      "missing word: air-shocks\n",
      "missing word: rock/paper/scissors\n",
      "missing word: odds-evens\n",
      "missing word: robot-like\n",
      "missing word: black-curtained\n",
      "missing word: dark-suited\n",
      "missing word: flight-announcing\n",
      "missing word: half-opened\n",
      "missing word: .a\n",
      "missing word: on-lookers\n",
      "missing word: white-neon\n",
      "missing word: white-t-shirted\n",
      "missing word: segway=like\n",
      "missing word: redsocks\n",
      "missing word: long-sleeve\n",
      "missing word: brown-skinned\n",
      "missing word: blue-jean\n",
      "missing word: white-and-grey\n",
      "missing word: black-padded\n",
      "missing word: musses\n",
      "missing word: modelpress\n",
      "missing word: grey-and-white\n",
      "missing word: corgi-like\n",
      "missing word: arm-wrestle\n",
      "missing word: black-draped\n",
      "missing word: maroon-clad\n",
      "missing word: legs..\n",
      "missing word: elf-hats\n",
      "missing word: on..\n",
      "missing word: gold-trimmed\n",
      "missing word: door-jam\n",
      "missing word: long-blonde\n",
      "missing word: bikini-girl\n",
      "missing word: california-style\n",
      "missing word: neiman-marcus\n",
      "missing word: light-grey\n",
      "missing word: colorfully-patterned\n",
      "missing word: neck-ruff\n",
      "missing word: dachshund-like\n",
      "missing word: bronx-cheer\n",
      "missing word: blow-torch\n",
      "missing word: tacket\n",
      "missing word: mid-basketball\n",
      "missing word: snow-laden\n",
      "missing word: pink-trimmed\n",
      "missing word: duck-headed\n",
      "missing word: waist-pack\n",
      "missing word: noice\n",
      "missing word: yellow-sheeted\n",
      "missing word: light-filled\n",
      "missing word: reading-part\n",
      "missing word: freethrow\n",
      "missing word: coffee-colored\n",
      "missing word: teeshirt\n",
      "missing word: spreadeagled\n",
      "missing word: buldog\n",
      "missing word: hard-wood\n",
      "missing word: black-rimmed\n",
      "missing word: ferule\n",
      "missing word: asian-language\n",
      "missing word: black-ink\n",
      "missing word: many-storied\n",
      "missing word: close-cropped\n",
      "missing word: white-walled\n",
      "missing word: smart-phone\n",
      "missing word: duct-taped\n",
      "missing word: underbrim\n",
      "missing word: yellow-sleeveless\n",
      "missing word: teeshirts\n",
      "missing word: teamin\n",
      "missing word: white-shirted\n",
      "missing word: hot-dog-like\n",
      "missing word: tank-top\n",
      "missing word: brown-\n",
      "missing word: beatles-type\n",
      "missing word: 'he\n",
      "missing word: long-black\n",
      "missing word: neon-filled\n",
      "missing word: gold-streaked\n",
      "missing word: olive-skinned\n",
      "missing word: purple-shirted\n",
      "missing word: copa90\n",
      "missing word: fifr\n",
      "missing word: also-asian\n",
      "missing word: changing-colored\n",
      "missing word: sweat-panted\n",
      "missing word: holdig\n",
      "missing word: brown-wrapped\n",
      "missing word: yellow-jacketed\n",
      "missing word: sniggers\n",
      "missing word: istands\n",
      "missing word: act-out\n",
      "missing word: navy-colored\n",
      "missing word: cloth-clad\n",
      "missing word: head-in\n",
      "missing word: close-cut\n",
      "missing word: prophet-like\n",
      "missing word: kick-boxer\n",
      "missing word: ikes\n",
      "missing word: teal-colored\n",
      "missing word: tunic-length\n",
      "missing word: grey-striped\n",
      "missing word: face-to\n",
      "missing word: latino-looking\n",
      "missing word: bank-like\n",
      "missing word: chocolate-colored\n",
      "missing word: heavily-made-up\n",
      "missing word: tanktop\n",
      "missing word: koemi\n",
      "missing word: bts_jiemin\n",
      "missing word: grey-and\n",
      "missing word: public-area\n",
      "missing word: pepperment\n",
      "missing word: v-necked\n",
      "missing word: cap-sleeved\n",
      "missing word: hearse-like\n",
      "missing word: tree-surrounded\n",
      "missing word: person..\n",
      "missing word: oval-headed\n",
      "missing word: quizno\n",
      "missing word: wnrs\n",
      "missing word: wnpb\n",
      "missing word: barstoolsports\n",
      "missing word: fan-crowded\n",
      "missing word: dc29\n",
      "missing word: right-side-up\n",
      "missing word: blunging\n",
      "missing word: dark=haired\n",
      "missing word: 'let\n",
      "missing word: wide-topped\n",
      "missing word: re-lands\n",
      "missing word: malemute\n",
      "missing word: husky-like\n",
      "missing word: pink-and-white\n",
      "missing word: off..\n",
      "missing word: orienta\n",
      "missing word: uninterpretable\n",
      "missing word: chocker\n",
      "missing word: tri-tone\n",
      "missing word: mrz\n",
      "missing word: funyuns\n",
      "missing word: black-topped\n",
      "missing word: mop-headed\n",
      "missing word: half-ball\n",
      "missing word: clse\n",
      "missing word: marble-topped\n",
      "missing word: cheersport\n",
      "missing word: unscrews\n",
      "missing word: pastic\n",
      "missing word: winter-bare\n",
      "missing word: extralarge\n",
      "missing word: black-dressed\n",
      "missing word: yellow-painted\n",
      "missing word: blue-doored\n",
      "missing word: golf-course\n",
      "missing word: asian-alphabet\n",
      "missing word: white-topped\n",
      "missing word: re-throw\n",
      "missing word: green-shirted\n",
      "missing word: camera-like\n",
      "missing word: toy-sized\n",
      "missing word: c-lo\n",
      "missing word: tile-covered\n",
      "missing word: fold-over\n",
      "missing word: picture-time\n",
      "missing word: rummer\n",
      "missing word: blackops\n",
      "missing word: head-rest\n",
      "missing word: shaven-headed\n",
      "missing word: 'show\n",
      "missing word: black-coated\n",
      "missing word: red-nailed\n",
      "missing word: black-shirted\n",
      "missing word: stubbles\n",
      "missing word: white-clad\n",
      "missing word: green-glad\n",
      "missing word: convention-style\n",
      "missing word: food-court\n",
      "missing word: upper-floors\n",
      "missing word: branch-littered\n",
      "missing word: flokati\n",
      "missing word: glue-grey\n",
      "missing word: m-audio\n",
      "missing word: chair-desks\n",
      "missing word: doubled=over\n",
      "missing word: brain-freeze\n",
      "missing word: crack-head\n",
      "missing word: institutional-looking\n",
      "missing word: grey-bearded\n",
      "missing word: 'pizza\n",
      "missing word: lutting\n",
      "missing word: pretzel-shaped\n",
      "missing word: soccer-sized\n",
      "missing word: casemiro\n",
      "missing word: calisthenic\n",
      "missing word: grey-carpeted\n",
      "missing word: toy-covered\n",
      "missing word: playsuit\n",
      "missing word: black-rooted\n",
      "missing word: backwards-forwards\n",
      "missing word: 'happy\n",
      "missing word: 'selfie\n",
      "missing word: ofter\n",
      "missing word: blue-clad\n",
      "missing word: concrete-floored\n",
      "missing word: black-and-red\n",
      "missing word: guest-participation\n",
      "missing word: large-lensed\n",
      "missing word: 'spray\n",
      "missing word: attacher\n",
      "missing word: 'move\n",
      "missing word: window-lined\n",
      "missing word: orange-jacketed\n",
      "missing word: white-suited\n",
      "missing word: blue-lighted\n",
      "missing word: colapses\n",
      "missing word: smoothy\n",
      "missing word: t-shirted\n",
      "missing word: smoke/sparks\n",
      "missing word: manuvers\n",
      "missing word: white/red\n",
      "missing word: blue/red\n",
      "missing word: red/blue\n",
      "missing word: comopresses\n",
      "missing word: carseat\n",
      "missing word: wakens\n",
      "missing word: sprs\n",
      "missing word: anasian\n",
      "missing word: underbite\n",
      "missing word: wok=man\n",
      "missing word: rollaway\n",
      "missing word: hover-board\n",
      "missing word: belly-crawls\n",
      "missing word: dog-sized\n",
      "missing word: knells\n",
      "missing word: high-backed\n",
      "missing word: lift-gate\n",
      "missing word: raised-gate\n",
      "missing word: nocking\n",
      "missing word: magcon\n",
      "missing word: streetlamp\n",
      "missing word: .movements\n",
      "missing word: hiks\n",
      "missing word: swimmies\n",
      "missing word: top\\y\n",
      "missing word: trows\n",
      "missing word: neglige\n",
      "missing word: head-butts\n",
      "missing word: multiperson\n",
      "missing word: grey-suited\n",
      "missing word: straw-covered\n",
      "missing word: *extra*\n",
      "missing word: cheted\n",
      "missing word: vidcon\n",
      "missing word: provoques\n",
      "missing word: missies\n",
      "missing word: timeother\n",
      "missing word: inner-tube\n",
      "missing word: threeasian\n",
      "missing word: one-hand\n",
      "missing word: smily\n",
      "missing word: time.\n",
      "missing word: red-carpeted\n",
      "missing word: sticklike\n",
      "missing word: orangutang\n",
      "missing word: cigaret\n",
      "missing word: things.\n",
      "missing word: arms0\n",
      "missing word: 2-bench\n",
      "missing word: mid-bite\n",
      "missing word: face-paint\n",
      "missing word: red-bearded\n",
      "missing word: fu*k\n",
      "missing word: osters\n",
      "missing word: bedrroom\n",
      "missing word: headphones/earbuds\n",
      "missing word: sh*t\n",
      "missing word: witch-like\n",
      "missing word: di*k\n",
      "missing word: under-lit\n",
      "missing word: white-framed\n",
      "missing word: spasming\n",
      "missing word: muthafukin\n",
      "missing word: hula-hoop\n",
      "missing word: housetops\n",
      "missing word: inner-tubes\n",
      "missing word: perfomer\n",
      "missing word: song.s\n",
      "missing word: ex-girl\n",
      "missing word: jaywalks\n",
      "missing word: vlive\n",
      "missing word: distance.\n",
      "missing word: red-striped\n",
      "missing word: shirrs\n",
      "missing word: skirt.s\n",
      "missing word: young-teen\n",
      "missing word: happy-face\n",
      "missing word: carousal\n",
      "missing word: paper-scissors-rock\n",
      "missing word: rock/scissor/paper\n",
      "missing word: cream-topped\n",
      "missing word: cartful\n",
      "missing word: w.coast\n",
      "missing word: oww\n",
      "missing word: si..\n",
      "missing word: fabe\n",
      "missing word: half-door\n",
      "missing word: somene\n",
      "missing word: dalls\n",
      "missing word: plataform\n",
      "missing word: peson\n",
      "missing word: rainbow-striped\n",
      "missing word: tippy-cup\n",
      "missing word: pipe-like\n",
      "missing word: chid\n",
      "missing word: dough-like\n",
      "missing word: dashcam\n",
      "missing word: six-packs\n",
      "missing word: time/\n",
      "missing word: safara\n",
      "missing word: inpark\n",
      "missing word: walkie-talkie-\n",
      "missing word: runon\n",
      "missing word: i-pad\n",
      "missing word: airpot\n",
      "missing word: redresses\n",
      "missing word: newsvideo\n",
      "missing word: pajamaed\n",
      "missing word: trolly\n",
      "missing word: skake\n",
      "missing word: band..\n",
      "missing word: water-fountains\n",
      "missing word: interwound\n",
      "missing word: uncarpeted\n",
      "missing word: suare\n",
      "missing word: ..someones\n",
      "missing word: posthole\n",
      "missing word: box/floor\n",
      "missing word: robot-girl\n",
      "missing word: puling\n",
      "missing word: surf-boarders\n",
      "missing word: burgundy-colored\n",
      "missing word: roller-skates\n",
      "missing word: raodway\n",
      "missing word: rvca\n",
      "missing word: locey\n",
      "missing word: preents\n",
      "missing word: rolll-up\n",
      "missing word: drink..\n",
      "missing word: spot-light\n",
      "missing word: high-fives\n",
      "missing word: f=group\n",
      "missing word: un-iced\n",
      "missing word: nippled\n",
      "missing word: *litter\n",
      "missing word: 2008*\n",
      "missing word: kiter\n",
      "missing word: mittened\n",
      "missing word: mid-ground\n",
      "missing word: marble-walled\n",
      "missing word: on/sit\n",
      "missing word: tonik\n",
      "missing word: copper-top\n",
      "missing word: by..\n",
      "missing word: hours.l\n",
      "missing word: underware\n",
      "missing word: sunshiney\n",
      "missing word: house-like\n",
      "missing word: umbrellaed\n",
      "missing word: goat-like\n",
      "missing word: rodie\n",
      "missing word: =s\n",
      "missing word: stopwar.ca\n",
      "missing word: animecentral\n",
      "missing word: dodge-em\n",
      "missing word: cowboy-hatted\n",
      "missing word: around..\n",
      "missing word: blue-sky\n",
      "missing word: nauticalia\n",
      "missing word: jakson\n",
      "missing word: multi-desk\n",
      "missing word: shews\n",
      "missing word: blond-looking\n",
      "missing word: lined-headdress\n",
      "missing word: ****\n",
      "missing word: *barbecues*\n",
      "missing word: thenh\n",
      "missing word: peahen\n",
      "missing word: traw\n",
      "missing word: well=mowed\n",
      "missing word: library-like\n",
      "missing word: repare\n",
      "missing word: parachuter\n",
      "missing word: balloon.\n",
      "count of missing words:  772\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('punkt')\n",
    "\n",
    "vocab, total_len = {}, 0\n",
    "for cap in train_corpus:\n",
    "    tokens = nltk.word_tokenize(cap.lower())\n",
    "    total_len += len(tokens)\n",
    "    for w in tokens:\n",
    "        try:\n",
    "            vocab[w] += 1\n",
    "        except:\n",
    "            vocab[w] = 1\n",
    "\n",
    "print('Avg. count of words per caption:', total_len/len(train_corpus))\n",
    "print('Count of unique words: ', len(vocab))\n",
    "            \n",
    "to_del = []\n",
    "for w in vocab.keys():\n",
    "    if not w in wordvectors:\n",
    "        to_del.append(w)\n",
    "        print('missing word: {}'.format(w))\n",
    "\n",
    "print('count of missing words: ', len(to_del))\n",
    "        \n",
    "for w in to_del:\n",
    "    del vocab[w]\n",
    "        \n",
    "idx2word = {idx: word for idx, word in enumerate(['<eos>', '<unk>'] + list(vocab.keys()))}\n",
    "word2idx = {word: idx for idx, word in enumerate(['<eos>', '<unk>'] + list(vocab.keys()))}\n",
    "EOS, UNK = 0, 1\n",
    "\n",
    "len(vocab), len(idx2word), len(word2idx)\n",
    "\n",
    "word_embeddings = np.zeros((len(idx2word), 300))\n",
    "for idx, word in idx2word.items():\n",
    "    if idx == EOS:\n",
    "        word_embeddings[idx] = wordvectors['eos']\n",
    "    elif idx == UNK:\n",
    "        word_embeddings[idx] = wordvectors['unk']\n",
    "    else:\n",
    "        word_embeddings[idx] = wordvectors[word]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine POS-tagging vocabulary from train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words per tag:\n",
      " DT:\t23\n",
      " NN:\t5352\n",
      " VBZ:\t939\n",
      " IN:\t126\n",
      " PRP$:\t7\n",
      " ,:\t1\n",
      " CD:\t104\n",
      " NNS:\t2165\n",
      " VBP:\t740\n",
      " JJ:\t2816\n",
      " VBG:\t1036\n",
      " TO:\t2\n",
      " VB:\t941\n",
      " RP:\t28\n",
      " .:\t3\n",
      " CC:\t19\n",
      " RB:\t522\n",
      " POS:\t4\n",
      " PRP:\t24\n",
      " VBN:\t767\n",
      " VBD:\t502\n",
      " WRB:\t7\n",
      " WDT:\t6\n",
      " (:\t1\n",
      " ):\t1\n",
      " PDT:\t14\n",
      " JJR:\t52\n",
      " WP:\t7\n",
      " MD:\t14\n",
      " ``:\t1\n",
      " '':\t5\n",
      " WP$:\t1\n",
      " JJS:\t16\n",
      " ::\t5\n",
      " UH:\t3\n",
      " EX:\t2\n",
      " #:\t1\n",
      " FW:\t16\n",
      " RBR:\t16\n",
      " NNP:\t12\n",
      " $:\t2\n",
      " RBS:\t1\n",
      "44\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "pos_vocab = {}\n",
    "pos_unique_words = {}\n",
    "for cap in train_corpus:\n",
    "    for tag in nltk.pos_tag(nltk.word_tokenize(cap.lower())):\n",
    "        try:\n",
    "            pos_vocab[tag[1]] += 1\n",
    "            try: \n",
    "                pos_unique_words[tag[1]][tag[0]] += 1\n",
    "            except:\n",
    "                pos_unique_words[tag[1]][tag[0]] = 1\n",
    "        except:\n",
    "            pos_vocab[tag[1]] = 1\n",
    "            pos_unique_words[tag[1]] = {tag[0]: 1}\n",
    "\n",
    "print('Unique words per tag:')\n",
    "print('\\n'.join([f' {k}:\\t{len(words)}' for k, words in pos_unique_words.items()]))\n",
    "            \n",
    "idx2pos = {idx: tag for idx, tag in enumerate(['eos', 'unk'] + list(pos_vocab.keys()))}\n",
    "pos2idx = {tag: idx for idx, tag in enumerate(['eos', 'unk'] + list(pos_vocab.keys()))}\n",
    "EOS, UNK = 0, 1\n",
    "print(len(idx2pos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Determine Universal POS-tagging from train split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package universal_tagset to\n",
      "[nltk_data]     /home/jeperez/nltk_data...\n",
      "[nltk_data]   Package universal_tagset is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unique words per universal tag:\n",
      " DET:\t40\n",
      " NOUN:\t7316\n",
      " VERB:\t4038\n",
      " ADP:\t126\n",
      " PRON:\t38\n",
      " .:\t20\n",
      " NUM:\t104\n",
      " ADJ:\t2878\n",
      " PRT:\t34\n",
      " CONJ:\t19\n",
      " ADV:\t542\n",
      " X:\t19\n",
      "14\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('universal_tagset')\n",
    "\n",
    "upos_vocab = {}\n",
    "upos_unique_words = {}\n",
    "for cap in train_corpus:\n",
    "    for tag in nltk.pos_tag(nltk.word_tokenize(cap.lower()), tagset='universal'):\n",
    "        try:\n",
    "            upos_vocab[tag[1]] += 1\n",
    "            try: \n",
    "                upos_unique_words[tag[1]][tag[0]] += 1\n",
    "            except:\n",
    "                upos_unique_words[tag[1]][tag[0]] = 1\n",
    "        except:\n",
    "            upos_vocab[tag[1]] = 1\n",
    "            upos_unique_words[tag[1]] = {tag[0]: 1}\n",
    "\n",
    "print('Unique words per universal tag:')\n",
    "print('\\n'.join([f' {k}:\\t{len(words)}' for k, words in upos_unique_words.items()]))\n",
    "            \n",
    "idx2upos = {idx: word for idx, word in enumerate(['eos', 'unk'] + list(upos_vocab.keys()))}\n",
    "upos2idx = {word: idx for idx, word in enumerate(['eos', 'unk'] + list(upos_vocab.keys()))}\n",
    "print(len(idx2upos))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate ground-truth references files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('../results/TRECVID-2020_val_references.txt', 'w') as f:\n",
    "    for vidx, cap in zip(valid_vidxs, valid_corpus):\n",
    "        f.write('{}\\t{}\\n'.format(vidx, cap))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### test reference files, after the ground-truth was released"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "with open('../../../data/TRECVID/tv2020/test/Description_Generation_Groundtruth/vtt.cider.gt.json') as f:\n",
    "    gt = json.load(f)\n",
    "\n",
    "with open('../results/TRECVID-2020_test_references.txt', 'w') as f:\n",
    "    for row in gt:\n",
    "        f.write('{}\\t{}\\n'.format(row['image_id'], row['caption']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate corpus.pkl file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "train_corpus_widxs = [[word2idx[w] if w in vocab else UNK for w in nltk.word_tokenize(cap.lower())] + [EOS] for cap in train_corpus]\n",
    "valid_corpus_widxs = [[word2idx[w] if w in vocab else UNK for w in nltk.word_tokenize(cap.lower())] + [EOS] for cap in valid_corpus]\n",
    "\n",
    "train_corpus_pidxs = [[pos2idx[w[1]] if w[1] in pos_vocab else UNK for w in nltk.pos_tag(nltk.word_tokenize(cap.lower()))] + [EOS] for cap in train_corpus]\n",
    "valid_corpus_pidxs = [[pos2idx[w[1]] if w[1] in pos_vocab else UNK for w in nltk.pos_tag(nltk.word_tokenize(cap.lower()))] + [EOS] for cap in valid_corpus]\n",
    "\n",
    "assert len(train_corpus_widxs) == len(train_vidxs) and len(train_vidxs) == len(train_corpus_pidxs) and len(train_vidxs) == len(train_corpus), f'{len(train_vidxs)}, {len(train_corpus_widxs)}, {len(train_corpus_pidxs)}, {len(train_corpus)}'\n",
    "assert len(valid_corpus_widxs) == len(valid_vidxs) and len(valid_vidxs) == len(valid_corpus_pidxs) and len(valid_vidxs) == len(valid_corpus), f'{len(valid_vidxs)}, {len(valid_corpus_widxs)}, {len(valid_corpus_pidxs)}, {len(valid_corpus)}'\n",
    "\n",
    "train_data = [train_corpus_widxs, train_vidxs, train_corpus_pidxs, train_corpus]\n",
    "valid_data = [valid_corpus_widxs, valid_vidxs, valid_corpus_pidxs, valid_corpus]\n",
    "\n",
    "with open('../../../data/TRECVID/tv2020/trecvid_tv2020_corpus_pos.pkl', 'wb') as outfile:\n",
    "    pickle.dump([train_data, valid_data, None, vocab, idx2word, word_embeddings, idx2pos], outfile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "a man sings in a car\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[('a', 'DT'),\n",
       " ('man', 'NN'),\n",
       " ('sings', 'VBZ'),\n",
       " ('in', 'IN'),\n",
       " ('a', 'DT'),\n",
       " ('car', 'NN')]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train_vidxs[0])\n",
    "print(train_corpus[0])\n",
    "nltk.pos_tag(nltk.word_tokenize(train_corpus[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6\n",
      "a man is scared by a picture of a boy on a mirror in a bathroom\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[2, 3, 4, 21, 5, 2, 3, 5, 2, 3, 5, 2, 3, 5, 2, 3, 0]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(valid_vidxs[0])\n",
    "print(valid_corpus[0])\n",
    "valid_corpus_pidxs[0]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}