dart_jovian_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5bb3563a-0182-46bd-a32d-dc885f8c712b",
   "metadata": {},
   "source": [
    "# DART Jovian Moons Movie Stabilization using OpenCV using Open Source Data.\n",
    "\n",
    "An ROI centered movie of the Jovian Moons observations made by the NASA Double Asteroid Redirection Test (DART) spacecraft made on 2022-214. \n",
    "The data set is public and can be accessed in PDS here: https://pdssbn.astro.umd.edu/holdings/pds4-dart-v2.0/document_draco/SUPPORT/dataset.shtml \n",
    "\n",
    "### Data Processing\n",
    "The data is downloaded from the NASA PDS archives as a set of PNGs using `wget`.\n",
    "The data in the archive includes a time period after the Jovian Moons imaging period. During the last frames in the data collection the spacecraft continued imaging, but was no longer tracking Jupiter, we exclude the last images from our processed data set.\n",
    "The data also shows a momentary period during which Jupiter (our subject) exits the frame and comes back. This happens, roughly, frames `dart_0397159786_25152_01_rad.png` through `dart_0397159918_20758_01_rad.png`. In order to keep the tracker from losing the subject in the image tracking portion of the software, we abandon those images in the ROI centered video result produced by the end of this Jupyter Notebook. \n",
    "\n",
    "At this point DART was approximately 16 million miles (26 million km) from Earth with Jupiter approximately 435 million miles (700 million km) away from the spacecraft.\n",
    "The video shows from left to right are Ganymede, Jupiter, Europa, Io, and Callisto.\n",
    "\n",
    "The total time of the data processed: \n",
    "- First frame: dart_0397157693_12079_01_rad.fits (MET='397157693'/ [sec] MET first pixel arrives at SBC)\n",
    "- Last frame: dart_0397161169_08665_01_rad.fits (MET='397161169'/ [sec] MET first pixel arrives at SBC)\n",
    "- Total time = 3476secs = 57.93mins = 57mins 55.8sec\n",
    "\n",
    "\n",
    "### Subject (Jupiter) Tracking \n",
    "\n",
    "The code's main purpose is to track a subject within a video, ensuring that the subject remains centered in each frame of the output video, while also checking that the subject's ROI remains within the frame boundaries to avoid out-of-frame subjects. Adjustments and further optimizations can be made to suit specific tracking needs and video content.\n",
    "\n",
    "- It reads a video file specified by the path using the OpenCV library.\n",
    "- It allows the user to select an initial region of interest (ROI) by clicking and dragging the mouse on the first frame of the video.\n",
    "- It initializes the OpenCV TrackerCSRT object to track the selected ROI as it moves within subsequent frames of the video.\n",
    "- It sets up an output video file using the XVID codec and initializes a VideoWriter object to save the stabilized video.\n",
    "- It processes the video frame by frame in a loop. For each frame:\n",
    "  \n",
    "    - It updates the tracker with the current frame and the tracked ROI's new position.\n",
    "    - It calculates translation offsets to keep the subject within the frame centered at all times.\n",
    "    - It checks if the subject's ROI is within the frame boundaries. If not, the frame is skipped.\n",
    "    - If the subject's ROI is within the frame, it applies a translation to the entire frame to center the subject and draws the bounding box around the subject at the new position.\n",
    "    - It writes the processed frame into the output video.\n",
    "    - It displays the processed frame with the centered subject.\n",
    "    - It checks for the 'q' key press to exit the loop.\n",
    "      \n",
    "After processing all frames, it releases the capture and video writer resources and closes any OpenCV windows.\n",
    "\n",
    "If the subject falls out of frame for more than 2 images, tracking fails. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7b58d041-1abe-4796-aa56-84d5615f050f",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/rodrilm2/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:34: NotOpenSSLWarning: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from tqdm import tqdm\n",
    "import cv2\n",
    "import os\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7841124c-9ab4-4c18-844c-a0b442ae0ce6",
   "metadata": {},
   "source": [
    "## Image Processing (PNG)\n",
    "\n",
    "We will need to maintain the natural sort key order of our images to ensure the video is made using the right image order. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1ae291d8-7235-4e2b-90e1-7a1303f55ecd",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def natural_sort_key(s, _nsre=re.compile('([0-9]+)')):\n",
    "    return [\n",
    "        int(text)\n",
    "        if text.isdigit() else text.lower()\n",
    "        for text in _nsre.split(s)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0541a6d-1d2f-40bf-b172-3fd409ad5367",
   "metadata": {},
   "source": [
    "#### Convert PNGs into AVI file using OpenCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "41190655-8ccf-4861-9963-81d2067a1ae4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3475/3475 [00:14<00:00, 239.80it/s]\n"
     ]
    }
   ],
   "source": [
    "image_folder = '/Users/rodrilm2/jupyter/dart_jovian/pdssbn.astro.umd.edu/holdings/pds4-dart-v2.0/data_dracocal/cruise/2022/214_skipped_frames/'\n",
    "video_name = 'dart_jovian_skipped_lapses.avi'\n",
    "\n",
    "images = [img for img in os.listdir(image_folder) if img.endswith(\".png\")]\n",
    "sorted_images = sorted(images, key=natural_sort_key)\n",
    "frame = cv2.imread(os.path.join(image_folder, sorted_images[0]))\n",
    "height, width, layers = frame.shape\n",
    "\n",
    "video = cv2.VideoWriter(video_name, 0, 1, (width,height))\n",
    "\n",
    "for image in tqdm(sorted_images):\n",
    "    video.write(cv2.imread(os.path.join(image_folder, image)))\n",
    "\n",
    "cv2.destroyAllWindows()\n",
    "video.release()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "facb12d0-c668-47a8-8a5f-c37a5af8c839",
   "metadata": {},
   "source": [
    "#### Convert AVI to mp4 so that we can see it on MacOS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0f23c8a4-278d-4e60-b87d-a901e0dd64b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_avi_to_mp4(avi_file_path, output_name):\n",
    "    os.popen(\"/opt/homebrew/bin/ffmpeg -i '{input}' -ac 2 -b:v 2000k -c:a aac -c:v libx264 -b:a 160k -vprofile high -bf 0 -strict experimental -f mp4 '{output}.mp4'\".format(input = avi_file_path, output = output_name))\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "69121d9d-02c0-4f6c-90f5-6279b20e11f0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ffmpeg version 6.0 Copyright (c) 2000-2023 the FFmpeg developers\n",
      "  built with Apple clang version 14.0.3 (clang-1403.0.22.14.1)\n",
      "  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/6.0_1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon\n",
      "  libavutil      58.  2.100 / 58.  2.100\n",
      "  libavcodec     60.  3.100 / 60.  3.100\n",
      "  libavformat    60.  3.100 / 60.  3.100\n",
      "  libavdevice    60.  1.100 / 60.  1.100\n",
      "  libavfilter     9.  3.100 /  9.  3.100\n",
      "  libswscale      7.  1.100 /  7.  1.100\n",
      "  libswresample   4. 10.100 /  4. 10.100\n",
      "  libpostproc    57.  1.100 / 57.  1.100\n",
      "Input #0, avi, from 'dart_jovian_skipped_lapses.avi':\n",
      "  Metadata:\n",
      "    software        : Lavf59.27.100\n",
      "  Duration: 00:57:55.00, start: 0.000000, bitrate: 12583 kb/s\n",
      "  Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 1024x1024, 12586 kb/s, 1 fps, 1 tbr, 1 tbn\n",
      "Stream mapping:\n",
      "  Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))\n",
      "Press [q] to stop, [?] for help\n",
      "[libx264 @ 0x12d606b40] using cpu capabilities: ARMv8 NEON\n",
      "[libx264 @ 0x12d606b40] profile High, level 3.2, 4:2:0, 8-bit\n",
      "[libx264 @ 0x12d606b40] 264 - core 164 r3108 31e19f9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2023 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=18 lookahead_threads=3 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=2 keyint=250 keyint_min=1 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=2000 ratetol=1.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00\n",
      "Output #0, mp4, to 'dart_jovian_skipped_lapses.mp4':\n",
      "  Metadata:\n",
      "    software        : Lavf59.27.100\n",
      "    encoder         : Lavf60.3.100\n",
      "  Stream #0:0: Video: h264 (avc1 / 0x31637661), yuv420p(progressive), 1024x1024, q=2-31, 2000 kb/s, 1 fps, 16384 tbn\n",
      "    Metadata:\n",
      "      encoder         : Lavc60.3.100 libx264\n",
      "    Side data:\n",
      "      cpb: bitrate max/min/avg: 0/0/2000000 buffer size: 0 vbv_delay: N/A\n",
      "frame= 3475 fps=1062 q=-1.0 Lsize=   13391kB time=00:57:54.00 bitrate=  31.6kbits/s speed=1.06e+03x    \n",
      "video:13377kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.109435%\n",
      "[libx264 @ 0x12d606b40] frame I:14    Avg QP: 0.00  size: 40792\n",
      "[libx264 @ 0x12d606b40] frame P:3461  Avg QP: 0.04  size:  3793\n",
      "[libx264 @ 0x12d606b40] mb I  I16..4: 80.1%  4.2% 15.7%\n",
      "[libx264 @ 0x12d606b40] mb P  I16..4:  0.2%  0.2%  0.1%  P16..4:  1.2%  0.3%  0.3%  0.0%  0.0%    skip:97.7%\n",
      "[libx264 @ 0x12d606b40] final ratefactor: -30.13\n",
      "[libx264 @ 0x12d606b40] 8x8 transform intra:24.0% inter:33.4%\n",
      "[libx264 @ 0x12d606b40] coded y,uvDC,uvAC intra: 62.8% 44.3% 44.2% inter: 0.9% 0.8% 0.8%\n",
      "[libx264 @ 0x12d606b40] i16 v,h,dc,p: 83% 11%  6%  0%\n",
      "[libx264 @ 0x12d606b40] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  5%  7% 80%  1%  2%  2%  2%  1%  1%\n",
      "[libx264 @ 0x12d606b40] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 14% 13% 64%  2%  2%  2%  1%  1%  1%\n",
      "[libx264 @ 0x12d606b40] i8c dc,h,v,p: 88%  5%  4%  3%\n",
      "[libx264 @ 0x12d606b40] Weighted P-Frames: Y:3.3% UV:0.0%\n",
      "[libx264 @ 0x12d606b40] ref P L0: 64.9%  2.3% 21.1% 11.3%  0.4%\n",
      "[libx264 @ 0x12d606b40] kb/s:31.53\n"
     ]
    }
   ],
   "source": [
    "dart_jovian_mp4=\"dart_jovian_skipped_lapses\"\n",
    "dart_jovian_centered_subject='dart_jovian_centered_subject'\n",
    "convert_avi_to_mp4(video_name, dart_jovian_mp4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0df26607-3093-4f1f-9200-21e23fb1e468",
   "metadata": {},
   "source": [
    "#### Center ROI the Movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bd348ea3-2ac6-47d3-aa9d-d7cc52f1917a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Select a ROI and then press SPACE or ENTER button!\n",
      "Cancel the selection process by pressing c button!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "OpenCV: FFMPEG: tag 0x44495658/'XVID' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)'\n",
      "OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v'\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DONE!\n"
     ]
    }
   ],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "# Read the video\n",
    "cap = cv2.VideoCapture(dart_jovian_mp4+\".mp4\")\n",
    "\n",
    "# Read the first frame\n",
    "ret, frame = cap.read()\n",
    "if not ret:\n",
    "    exit()\n",
    "\n",
    "# Select the ROI for tracking\n",
    "roi = cv2.selectROI(\"Select ROI\", frame, fromCenter=False, showCrosshair=True)\n",
    "\n",
    "# Initialize the tracker\n",
    "tracker = cv2.TrackerCSRT_create()\n",
    "tracker.init(frame, roi)\n",
    "\n",
    "# Get the frame dimensions\n",
    "height, width, _ = frame.shape\n",
    "\n",
    "# Define the codec and create a VideoWriter object\n",
    "fourcc = cv2.VideoWriter_fourcc(*'XVID')\n",
    "out = cv2.VideoWriter(dart_jovian_centered_subject+\".mp4\", fourcc, 1.0, (width, height))\n",
    "\n",
    "# Process the video frame by frame\n",
    "while True:\n",
    "    ret, frame = cap.read()\n",
    "    if not ret:\n",
    "        break\n",
    "\n",
    "    # Update the tracker\n",
    "    success, roi = tracker.update(frame)\n",
    "\n",
    "    # If tracking is successful and the subject ROI is within the frame boundaries\n",
    "    if success:\n",
    "        (x, y, w, h) = [int(i) for i in roi]\n",
    "\n",
    "        # Calculate the translation offsets\n",
    "        center_x, center_y = width // 2, height // 2\n",
    "        offset_x, offset_y = center_x - (x + w // 2), center_y - (y + h // 2)\n",
    "\n",
    "        # Check if the subject ROI is within the frame boundaries\n",
    "        if 0 <= x + offset_x <= width - w and 0 <= y + offset_y <= height - h:\n",
    "            # Apply the translation to the entire frame\n",
    "            translation_matrix = np.float32([[1, 0, offset_x], [0, 1, offset_y]])\n",
    "            frame = cv2.warpAffine(frame, translation_matrix, (width, height))\n",
    "\n",
    "            # Draw the bounding box at the new position\n",
    "            x, y = x + offset_x, y + offset_y\n",
    "\n",
    "            # Write the frame into the output video\n",
    "            out.write(frame)\n",
    "\n",
    "    # Display the frame\n",
    "    cv2.imshow('Frame', frame)\n",
    "\n",
    "    # Press 'q' to exit\n",
    "    if cv2.waitKey(1) & 0xFF == ord('q'):\n",
    "        break\n",
    "\n",
    "# Release the capture and video writer, and destroy any OpenCV windows\n",
    "cap.release()\n",
    "out.release()\n",
    "cv2.destroyAllWindows()\n",
    "print(\"DONE!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6591eb0-e298-4aae-bd48-a2d5c62823a2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}