notebooks/02_GPT3_finetune_wb.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tune GPT3 with Weights & Biases"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "import wandb\n",
    "import pandas as pd\n",
    "from dotenv import find_dotenv, load_dotenv\n",
    "dotenv_path = find_dotenv()\n",
    "load_dotenv(dotenv_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mbenneo\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "Tracking run with wandb version 0.13.5"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Run data is saved locally in <code>/Users/benedictneo/gpt3-blog-title/notebooks/wandb/run-20221130_163524-39bftkuh</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Syncing run <strong><a href=\"https://wandb.ai/benneo/GPT-3%20blog%20title/runs/39bftkuh\" target=\"_blank\">silvery-disco-3</a></strong> to <a href=\"https://wandb.ai/benneo/GPT-3%20blog%20title\" target=\"_blank\">Weights & Biases</a> (<a href=\"https://wandb.me/run\" target=\"_blank\">docs</a>)<br/>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create a job for splitting dataset\n",
    "project_name = \"GPT-3 blog title\"\n",
    "run = wandb.init(project=project_name, job_type='split dataset')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_path= \"../data/2_final/prompts.jsonl\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "let's look at a few samples of our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"prompt\":\"Title: Literally Nobody Voted in the Quincy Midterm Elections ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Neural Networks: Is Your Brain Like A Computer? ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Must have MacBook apps for productivity ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Remedial Data Science Engineering ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Verifiable Deployment of Smart Contracts ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Security Tokens vs. Fat Protocols ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: The Fundamental Problem of the Data Economy Nobody is Talking About ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Hello Triangle, Meet Swift! (And Wide Color) ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Time series analysis and its different approach in python : Part 1 ->\",\"completion\":\" good\"}\n",
      "{\"prompt\":\"Title: Making serverless variables work for you ->\",\"completion\":\" good\"}\n"
     ]
    }
   ],
   "source": [
    "!head $dataset_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "verify data is correctly formatted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## train/valid split with openai cli"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing...\n",
      "\n",
      "- Your file contains 33373 prompt-completion pairs\n",
      "- Based on your data it seems like you're trying to fine-tune a model for classification\n",
      "- For classification, we recommend you try one of the faster and cheaper models, such as `ada`\n",
      "- For classification, you can estimate the expected model performance by keeping a held out dataset, which is not used for training\n",
      "- All prompts end with suffix ` ->`\n",
      "- All prompts start with prefix `Title: `\n",
      "\n",
      "No remediations found.\n",
      "- [Recommended] Would you like to split into training and validation set? [Y/n]: Y\n",
      "\n",
      "\n",
      "Your data will be written to a new JSONL file. Proceed [Y/n]: Y\n",
      "\n",
      "Wrote modified files to `../data/2_final/prompts_prepared_train.jsonl` and `../data/2_final/prompts_prepared_valid.jsonl`\n",
      "Feel free to take a look!\n",
      "\n",
      "Now use that file when fine-tuning:\n",
      "> openai api fine_tunes.create -t \"../data/2_final/prompts_prepared_train.jsonl\" -v \"../data/2_final/prompts_prepared_valid.jsonl\" --compute_classification_metrics --classification_positive_class \" good\"\n",
      "\n",
      "After you’ve fine-tuned a model, remember that your prompt has to end with the indicator string ` ->` for the model to start generating completions, rather than continuing with the prompt. Make sure to include `stop=[\"d\"]` so that the generated texts ends at the expected place.\n",
      "Once your model starts training, it'll approximately take 13.39 hours to train a `curie` model, and less for `ada` and `babbage`. Queue will approximately take half an hour per job ahead of you.\n"
     ]
    }
   ],
   "source": [
    "!openai tools fine_tunes.prepare_data -f $dataset_path -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   32373 ../data/2_final/prompts_prepared_train.jsonl\n",
      "    1000 ../data/2_final/prompts_prepared_valid.jsonl\n"
     ]
    }
   ],
   "source": [
    "# check number of samples\n",
    "!wc -l ../data/2_final/prompts_prepared_train.jsonl\n",
    "!wc -l ../data/2_final/prompts_prepared_valid.jsonl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_train = 32373\n",
    "n_valid = 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Log train/valid split as W&B artifact"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tables for better visualization (optional)\n",
    "\n",
    "train_path = \"../data/2_final/prompts_prepared_train.jsonl\"\n",
    "valid_path = \"../data/2_final/prompts_prepared_valid.jsonl\"\n",
    "\n",
    "df_train = pd.read_json(train_path, orient='records', lines=True)\n",
    "df_valid = pd.read_json(valid_path, orient='records', lines=True)\n",
    "table_train = wandb.Table(dataframe=df_train)\n",
    "table_valid = wandb.Table(dataframe=df_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Upload artifacts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<wandb.sdk.wandb_artifacts.Artifact at 0x12547e250>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create artifacts\n",
    "artifact_train = wandb.Artifact('medium_train.jsonl', type='training_files', metadata={'samples': n_train})\n",
    "artifact_train.add_file(train_path)\n",
    "artifact_train.add(table_train, 'medium_train')\n",
    "\n",
    "artifact_valid = wandb.Artifact('medium_valid.jsonl', type='validation_files', metadata={'samples': n_valid})\n",
    "artifact_valid.add_file(valid_path)\n",
    "artifact_valid.add(table_valid, 'medium_train')\n",
    "\n",
    "# Log files\n",
    "run.log_artifact(artifact_train)\n",
    "run.log_artifact(artifact_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "entity = wandb.run.entity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6afb19f6f2d94a61845b6ff78d669557",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "VBox(children=(Label(value='5.292 MB of 5.292 MB uploaded (0.000 MB deduped)\\r'), FloatProgress(value=1.0, max…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Synced <strong style=\"color:#cdcd00\">silvery-disco-3</strong>: <a href=\"https://wandb.ai/benneo/GPT-3%20blog%20title/runs/39bftkuh\" target=\"_blank\">https://wandb.ai/benneo/GPT-3%20blog%20title/runs/39bftkuh</a><br/>Synced 6 W&B file(s), 0 media file(s), 4 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>./wandb/run-20221130_163524-39bftkuh/logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "recover training and validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('./artifacts/medium_train.jsonl:v0/prompts_prepared_train.jsonl',\n",
       " './artifacts/medium_valid.jsonl:v0/prompts_prepared_valid.jsonl')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artifact_train = run.use_artifact(f'{entity}/{project_name}/medium_train.jsonl:latest', type='training_files')\n",
    "train_file = artifact_train.get_path('prompts_prepared_train.jsonl').download()\n",
    "\n",
    "artifact_valid = run.use_artifact(f'{entity}/{project_name}/medium_valid.jsonl:latest', type='validation_files')\n",
    "valid_file = artifact_valid.get_path('prompts_prepared_valid.jsonl').download()\n",
    "\n",
    "train_file, valid_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## upload file to openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<File file id=file-ueOemwgI8SRCkiffG18K6WNK at 0x107fc2810> JSON: {\n",
       "  \"bytes\": 93362,\n",
       "  \"created_at\": 1669849603,\n",
       "  \"filename\": \"file\",\n",
       "  \"id\": \"file-ueOemwgI8SRCkiffG18K6WNK\",\n",
       "  \"object\": \"file\",\n",
       "  \"purpose\": \"fine-tune\",\n",
       "  \"status\": \"uploaded\",\n",
       "  \"status_details\": null\n",
       "}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# upload train\n",
    "openai.File.create(\n",
    "  file=open(train_file, \"rb\"),\n",
    "  purpose='fine-tune'\n",
    ")\n",
    "\n",
    "# upload validation\n",
    "openai.File.create(\n",
    "  file=open(valid_file, \"rb\"),\n",
    "  purpose='fine-tune'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<OpenAIObject list at 0x127f9ee50> JSON: {\n",
       "  \"data\": [\n",
       "    {\n",
       "      \"bytes\": 93362,\n",
       "      \"created_at\": 1669849603,\n",
       "      \"filename\": \"file\",\n",
       "      \"id\": \"file-ueOemwgI8SRCkiffG18K6WNK\",\n",
       "      \"object\": \"file\",\n",
       "      \"purpose\": \"fine-tune\",\n",
       "      \"status\": \"processed\",\n",
       "      \"status_details\": null\n",
       "    },\n",
       "    {\n",
       "      \"bytes\": 3014524,\n",
       "      \"created_at\": 1669849602,\n",
       "      \"filename\": \"file\",\n",
       "      \"id\": \"file-eRRY7tNbGl7v8c7GLqDr3NRn\",\n",
       "      \"object\": \"file\",\n",
       "      \"purpose\": \"fine-tune\",\n",
       "      \"status\": \"processed\",\n",
       "      \"status_details\": null\n",
       "    }\n",
       "  ],\n",
       "  \"object\": \"list\"\n",
       "}"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "openai.File.list()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "define GPT-3 hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created fine-tune: ft-cNrLBjqB9V6g0WLayqpRxcGK\n",
      "Streaming events until fine-tuning is complete...\n",
      "\n",
      "(Ctrl-C will interrupt the stream, but not cancel the fine-tune)\n",
      "[2022-11-30 22:57:29] Created fine-tune: ft-cNrLBjqB9V6g0WLayqpRxcGK\n",
      "[2022-11-30 22:57:33] Fine-tune costs $0.74\n",
      "[2022-11-30 22:57:34] Fine-tune enqueued. Queue number: 0\n",
      "[2022-11-30 22:57:35] Fine-tune started\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "wandb: ERROR Dropped streaming file chunk (see wandb/debug-internal.log)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "^C\n"
     ]
    }
   ],
   "source": [
    "!openai api fine_tunes.create \\\n",
    "    -t \"file-eRRY7tNbGl7v8c7GLqDr3NRn\" \\\n",
    "    -v \"file-ueOemwgI8SRCkiffG18K6WNK\" \\\n",
    "    -m \"ada\" \\\n",
    "    --n_epochs 4 \\\n",
    "    --batch_size 256 \\\n",
    "    --classification_n_classes 2 \\\n",
    "    --suffix \"blog title scorer\" \\\n",
    "    --classification_positive_class \" good\" \\\n",
    "    --compute_classification_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!openai api fine_tunes.create \\\n",
    "    -t \"file-eRRY7tNbGl7v8c7GLqDr3NRn\" \\\n",
    "    -v \"file-ueOemwgI8SRCkiffG18K6WNK\" \\\n",
    "    -m \"ada\" \\\n",
    "    --n_epochs 4 \\\n",
    "    --batch_size 256 \\\n",
    "    --classification_n_classes 2 \\\n",
    "    --suffix \"blog title scorer\" \\\n",
    "    --classification_positive_class \" good\" \\\n",
    "    --prompt_loss_weight 0.1 \\\n",
    "    --compute_classification_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "adding a learning_rate_multiplier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created fine-tune: ft-2ooqd5t0ycMwvf8TM3MUuyAt\n",
      "Streaming events until fine-tuning is complete...\n",
      "\n",
      "(Ctrl-C will interrupt the stream, but not cancel the fine-tune)\n",
      "[2022-11-30 22:26:58] Created fine-tune: ft-2ooqd5t0ycMwvf8TM3MUuyAt\n",
      "[2022-11-30 22:27:01] Fine-tune costs $0.74\n",
      "[2022-11-30 22:27:02] Fine-tune enqueued. Queue number: 2\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!openai api fine_tunes.create \\\n",
    "    -t \"file-eRRY7tNbGl7v8c7GLqDr3NRn\" \\\n",
    "    -v \"file-ueOemwgI8SRCkiffG18K6WNK\" \\\n",
    "    -m \"ada\" \\\n",
    "    --n_epochs 4 \\\n",
    "    --batch_size 256 \\\n",
    "    --classification_n_classes 2 \\\n",
    "    --suffix \"blog title scorer\" \\\n",
    "    --classification_positive_class \" good\" \\\n",
    "    --prompt_loss_weight 0.1 \\\n",
    "    --learning_rate_multiplier 0.2 \\\n",
    "    --compute_classification_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "using babbage model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created fine-tune: ft-3Svmi2GPLKTLnZAQCmHlEYql\n",
      "Streaming events until fine-tuning is complete...\n",
      "\n",
      "(Ctrl-C will interrupt the stream, but not cancel the fine-tune)\n",
      "[2022-11-30 21:27:16] Created fine-tune: ft-3Svmi2GPLKTLnZAQCmHlEYql\n",
      "[2022-11-30 21:27:25] Fine-tune costs $1.11\n",
      "[2022-11-30 21:27:25] Fine-tune enqueued. Queue number: 0\n",
      "[2022-11-30 21:27:28] Fine-tune started\n",
      "\n",
      "Stream interrupted (client disconnected).\n",
      "To resume the stream, run:\n",
      "\n",
      "  openai api fine_tunes.follow -i ft-3Svmi2GPLKTLnZAQCmHlEYql\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!openai api fine_tunes.create \\\n",
    "    -t \"file-eRRY7tNbGl7v8c7GLqDr3NRn\" \\\n",
    "    -v \"file-ueOemwgI8SRCkiffG18K6WNK\" \\\n",
    "    -m \"babbage\" \\\n",
    "    --n_epochs 4 \\\n",
    "    --batch_size 256 \\\n",
    "    --classification_n_classes 2 \\\n",
    "    --suffix \"blog title scorer\" \\\n",
    "    --classification_positive_class \" good\" \\\n",
    "    --prompt_loss_weight 0.1 \\\n",
    "    --compute_classification_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m\u001b[1mwandb\u001b[0m: Currently logged in as: \u001b[33mbenneo\u001b[0m. Use \u001b[1m`wandb login --relogin`\u001b[0m to force relogin\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Tracking run with wandb version 0.13.5\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run data is saved locally in \u001b[35m\u001b[1m/Users/benedictneo/gpt3-blog-title/notebooks/wandb/run-20221130_230647-ft-cNrLBjqB9V6g0WLayqpRxcGK\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run \u001b[1m`wandb offline`\u001b[0m to turn off syncing.\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Syncing run \u001b[33mft-cNrLBjqB9V6g0WLayqpRxcGK\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: ⭐️ View project at \u001b[34m\u001b[4mhttps://wandb.ai/benneo/GPT-3%20blog%20title\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: 🚀 View run at \u001b[34m\u001b[4mhttps://wandb.ai/benneo/GPT-3%20blog%20title/runs/ft-cNrLBjqB9V6g0WLayqpRxcGK\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Waiting for W&B process to finish... \u001b[32m(success).\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: | 0.002 MB of 0.002 MB uploaded (0.000 MB deduped)\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run history:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:      classification/accuracy ▁███\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         classification/auprc ▁█▆▄\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         classification/auroc ▁▇█▆\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:          classification/f1.0 ▁█▇▇\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:     classification/precision █▁▂▃\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:        classification/recall ▁█▆▆\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:             elapsed_examples ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:               elapsed_tokens ▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▄▄▄▄▄▅▅▅▅▅▅▆▆▆▆▆▇▇▇▇▇▇███\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:                training_loss █▁▂▂▂▂▂▂▂▂▂▂▁▂▂▂▁▁▂▂▂▂▂▁▂▂▂▁▂▂▂▂▂▂▂▁▁▁▁▂\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:   training_sequence_accuracy ▁▅▅▅▄▅▆▇▆▆▆▇▆▇▅▇▇▆▇▇▇▇▇▇▇▇▇▇▇▇▇█▇▇▇█▇█▇▇\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:      training_token_accuracy ▁▅▅▅▄▅▆▇▆▆▆▇▆▇▅▇▇▆▇▇▇▇▇▇▇▇▇▇▇▇▇█▇▇▇█▇█▇▇\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:              validation_loss █▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: validation_sequence_accuracy ▁▆▆▇▇▆▆▇▇▆▇▇▇▇▇▇▇▇█▆█▇▇▇▇█▇█▇▇██▇▇▇▇██▇▇\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:    validation_token_accuracy ▁▆▆▇▇▆▆▇▇▆▇▇▇▇▇▇▇▇█▆█▇▇▇▇█▇█▇▇██▇▇▇▇██▇▇\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Run summary:\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:      classification/accuracy 0.586\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         classification/auprc 0.5954\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:         classification/auroc 0.61994\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:          classification/f1.0 0.60269\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:     classification/precision 0.57615\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:        classification/recall 0.63179\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:             elapsed_examples 129792.0\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:               elapsed_tokens 3498752.0\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:             fine_tuned_model ada:ft-personal:blog...\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:                       status succeeded\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:                training_loss 0.04391\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:   training_sequence_accuracy 0.65625\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:      training_token_accuracy 0.65625\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:              validation_loss 0.04652\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: validation_sequence_accuracy 0.57422\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m:    validation_token_accuracy 0.57422\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: \n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Synced \u001b[33mft-cNrLBjqB9V6g0WLayqpRxcGK\u001b[0m: \u001b[34m\u001b[4mhttps://wandb.ai/benneo/GPT-3%20blog%20title/runs/ft-cNrLBjqB9V6g0WLayqpRxcGK\u001b[0m\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Synced 5 W&B file(s), 0 media file(s), 1 artifact file(s) and 0 other file(s)\n",
      "\u001b[34m\u001b[1mwandb\u001b[0m: Find logs at: \u001b[35m\u001b[1m./wandb/run-20221130_230647-ft-cNrLBjqB9V6g0WLayqpRxcGK/logs\u001b[0m\n",
      "🎉 wandb sync completed successfully\n"
     ]
    }
   ],
   "source": [
    "!openai wandb sync --project \"GPT-3 blog title\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W&B offline. Running your script from this directory will only write metadata locally. Use wandb disabled to completely turn off W&B.\n"
     ]
    }
   ],
   "source": [
    "!wandb offline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "testing the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [],
   "source": [
    "finetunes = openai.FineTune.list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>object</th>\n",
       "      <th>id</th>\n",
       "      <th>hyperparams</th>\n",
       "      <th>organization_id</th>\n",
       "      <th>model</th>\n",
       "      <th>training_files</th>\n",
       "      <th>validation_files</th>\n",
       "      <th>result_files</th>\n",
       "      <th>created_at</th>\n",
       "      <th>updated_at</th>\n",
       "      <th>status</th>\n",
       "      <th>fine_tuned_model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>fine-tune</td>\n",
       "      <td>ft-8V9ak7gQ4jv3X4zM3wZwa4ma</td>\n",
       "      <td>{'n_epochs': 4, 'batch_size': 256, 'prompt_los...</td>\n",
       "      <td>org-Kz5UVJ3lj9OBEwe4ukIaOuoU</td>\n",
       "      <td>ada</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-ueOemwgI8SRCki...</td>\n",
       "      <td>[]</td>\n",
       "      <td>1669849957</td>\n",
       "      <td>1669849995</td>\n",
       "      <td>failed</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>fine-tune</td>\n",
       "      <td>ft-uSwqlpAX9RlIcFfrYkyNI8g1</td>\n",
       "      <td>{'n_epochs': 4, 'batch_size': 256, 'prompt_los...</td>\n",
       "      <td>org-Kz5UVJ3lj9OBEwe4ukIaOuoU</td>\n",
       "      <td>ada</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-ueOemwgI8SRCki...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-AC9o2olJ4SBYSL...</td>\n",
       "      <td>1669850014</td>\n",
       "      <td>1669850837</td>\n",
       "      <td>succeeded</td>\n",
       "      <td>ada:ft-personal:blog-title-scorer-2022-11-30-2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>fine-tune</td>\n",
       "      <td>ft-3Svmi2GPLKTLnZAQCmHlEYql</td>\n",
       "      <td>{'n_epochs': 4, 'batch_size': 256, 'prompt_los...</td>\n",
       "      <td>org-Kz5UVJ3lj9OBEwe4ukIaOuoU</td>\n",
       "      <td>babbage</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-ueOemwgI8SRCki...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-mhWxYNRuWpzRWT...</td>\n",
       "      <td>1669865236</td>\n",
       "      <td>1669867527</td>\n",
       "      <td>succeeded</td>\n",
       "      <td>babbage:ft-personal:blog-title-scorer-2022-12-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>fine-tune</td>\n",
       "      <td>ft-2ooqd5t0ycMwvf8TM3MUuyAt</td>\n",
       "      <td>{'n_epochs': 4, 'batch_size': 256, 'prompt_los...</td>\n",
       "      <td>org-Kz5UVJ3lj9OBEwe4ukIaOuoU</td>\n",
       "      <td>ada</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-ueOemwgI8SRCki...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-fqRlTuGaUwFyUk...</td>\n",
       "      <td>1669868818</td>\n",
       "      <td>1669869869</td>\n",
       "      <td>succeeded</td>\n",
       "      <td>ada:ft-personal:blog-title-scorer-2022-12-01-0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>fine-tune</td>\n",
       "      <td>ft-cNrLBjqB9V6g0WLayqpRxcGK</td>\n",
       "      <td>{'n_epochs': 4, 'batch_size': 256, 'prompt_los...</td>\n",
       "      <td>org-Kz5UVJ3lj9OBEwe4ukIaOuoU</td>\n",
       "      <td>ada</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-ueOemwgI8SRCki...</td>\n",
       "      <td>[{'object': 'file', 'id': 'file-8v1xTMFu7WjWnl...</td>\n",
       "      <td>1669870649</td>\n",
       "      <td>1669871181</td>\n",
       "      <td>succeeded</td>\n",
       "      <td>ada:ft-personal:blog-title-scorer-2022-12-01-0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      object                           id  \\\n",
       "0  fine-tune  ft-8V9ak7gQ4jv3X4zM3wZwa4ma   \n",
       "1  fine-tune  ft-uSwqlpAX9RlIcFfrYkyNI8g1   \n",
       "2  fine-tune  ft-3Svmi2GPLKTLnZAQCmHlEYql   \n",
       "3  fine-tune  ft-2ooqd5t0ycMwvf8TM3MUuyAt   \n",
       "4  fine-tune  ft-cNrLBjqB9V6g0WLayqpRxcGK   \n",
       "\n",
       "                                         hyperparams  \\\n",
       "0  {'n_epochs': 4, 'batch_size': 256, 'prompt_los...   \n",
       "1  {'n_epochs': 4, 'batch_size': 256, 'prompt_los...   \n",
       "2  {'n_epochs': 4, 'batch_size': 256, 'prompt_los...   \n",
       "3  {'n_epochs': 4, 'batch_size': 256, 'prompt_los...   \n",
       "4  {'n_epochs': 4, 'batch_size': 256, 'prompt_los...   \n",
       "\n",
       "                organization_id    model  \\\n",
       "0  org-Kz5UVJ3lj9OBEwe4ukIaOuoU      ada   \n",
       "1  org-Kz5UVJ3lj9OBEwe4ukIaOuoU      ada   \n",
       "2  org-Kz5UVJ3lj9OBEwe4ukIaOuoU  babbage   \n",
       "3  org-Kz5UVJ3lj9OBEwe4ukIaOuoU      ada   \n",
       "4  org-Kz5UVJ3lj9OBEwe4ukIaOuoU      ada   \n",
       "\n",
       "                                      training_files  \\\n",
       "0  [{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...   \n",
       "1  [{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...   \n",
       "2  [{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...   \n",
       "3  [{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...   \n",
       "4  [{'object': 'file', 'id': 'file-eRRY7tNbGl7v8c...   \n",
       "\n",
       "                                    validation_files  \\\n",
       "0  [{'object': 'file', 'id': 'file-ueOemwgI8SRCki...   \n",
       "1  [{'object': 'file', 'id': 'file-ueOemwgI8SRCki...   \n",
       "2  [{'object': 'file', 'id': 'file-ueOemwgI8SRCki...   \n",
       "3  [{'object': 'file', 'id': 'file-ueOemwgI8SRCki...   \n",
       "4  [{'object': 'file', 'id': 'file-ueOemwgI8SRCki...   \n",
       "\n",
       "                                        result_files  created_at  updated_at  \\\n",
       "0                                                 []  1669849957  1669849995   \n",
       "1  [{'object': 'file', 'id': 'file-AC9o2olJ4SBYSL...  1669850014  1669850837   \n",
       "2  [{'object': 'file', 'id': 'file-mhWxYNRuWpzRWT...  1669865236  1669867527   \n",
       "3  [{'object': 'file', 'id': 'file-fqRlTuGaUwFyUk...  1669868818  1669869869   \n",
       "4  [{'object': 'file', 'id': 'file-8v1xTMFu7WjWnl...  1669870649  1669871181   \n",
       "\n",
       "      status                                   fine_tuned_model  \n",
       "0     failed                                               None  \n",
       "1  succeeded  ada:ft-personal:blog-title-scorer-2022-11-30-2...  \n",
       "2  succeeded  babbage:ft-personal:blog-title-scorer-2022-12-...  \n",
       "3  succeeded  ada:ft-personal:blog-title-scorer-2022-12-01-0...  \n",
       "4  succeeded  ada:ft-personal:blog-title-scorer-2022-12-01-0...  "
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "wandb: ERROR Dropped streaming file chunk (see wandb/debug-internal.log)\n"
     ]
    }
   ],
   "source": [
    "pd.DataFrame.from_dict(finetunes[\"data\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ada:ft-personal:blog-title-scorer-2022-11-30-23-27-16 \t ft-uSwqlpAX9RlIcFfrYkyNI8g1\n",
      "babbage:ft-personal:blog-title-scorer-2022-12-01-04-05-26 \t ft-3Svmi2GPLKTLnZAQCmHlEYql\n",
      "ada:ft-personal:blog-title-scorer-2022-12-01-04-44-27 \t ft-2ooqd5t0ycMwvf8TM3MUuyAt\n",
      "ada:ft-personal:blog-title-scorer-2022-12-01-05-06-20 \t ft-cNrLBjqB9V6g0WLayqpRxcGK\n"
     ]
    }
   ],
   "source": [
    "model_ids = []\n",
    "for run in finetunes[\"data\"]:\n",
    "    if run[\"status\"] == \"succeeded\":\n",
    "        print(f\"{run['fine_tuned_model']} \\t {run['id']}\")\n",
    "        model_ids.append(run[\"id\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "fine tuning took 0 days 00:13:43\n",
      "fine tuning took 0 days 00:38:11\n",
      "fine tuning took 0 days 00:17:31\n",
      "fine tuning took 0 days 00:08:52\n"
     ]
    }
   ],
   "source": [
    "for id in model_ids:\n",
    "    res = openai.FineTune.retrieve(id=id)\n",
    "    df = pd.DataFrame.from_dict(res[\"events\"])\n",
    "    df[\"created_at\"] = pd.to_datetime(df[\"created_at\"], unit='s')\n",
    "    total_time = df[\"created_at\"].max() - df[\"created_at\"].min()\n",
    "    print(f\"fine tuning took {total_time}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Waiting for W&B process to finish... <strong style=\"color:green\">(success).</strong>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Synced <strong style=\"color:#cdcd00\">lunar-meadow-3</strong>: <a href=\"https://wandb.ai/benneo/GPT-3/runs/314xm882\" target=\"_blank\">https://wandb.ai/benneo/GPT-3/runs/314xm882</a><br/>Synced 6 W&B file(s), 0 media file(s), 0 artifact file(s) and 0 other file(s)"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "Find logs at: <code>./wandb/run-20221130_221512-314xm882/logs</code>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "wandb.finish()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.13 ('gpt3-blog')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "98388095db23f0d270f6cd358fd032ff4feb9c08e3a138f6a6bcb71953dc310f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}