Step-1-PDF-Pre-Processing-Logic.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4f67a6a6",
   "metadata": {},
   "source": "## Notebook 1: PDF Pre-processing"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "In the series, we will be going from a PDF to Podcast using all open models. \n",
    "\n",
    "The first step in getting to the podcast is finding a script, right now our logic is:\n",
    "- Use any PDF on any topic\n",
    "- Prompt [mlx-community/Qwen2.5-1.5B-Instruct-4bit](https://huggingface.co/mlx-community/Qwen2.5-1.5B-Instruct-4bit) model to process it into a text file\n",
    "- Re-write this into a podcast transcript in next notebook.\n",
    "\n",
    "In this notebook, we will upload a PDF and save it into a `.txt` file using the `PyPDF2` library, later we will process chunks from the text file using our featherlight model."
   ],
   "id": "30b6d43441645f5b"
  },
  {
   "cell_type": "markdown",
   "id": "61cb3584",
   "metadata": {},
   "source": [
    "Most of us shift-enter pass the comments to realise later we need to install libraries. For the few that read the instructions, please remember to do so:"
   ]
  },
  {
   "cell_type": "code",
   "id": "f4fc7aef-3505-482e-a998-790b8b9d48e4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:52.273873Z",
     "start_time": "2024-10-29T16:12:52.271371Z"
    }
   },
   "source": "#!pip install -r requirements.txt",
   "outputs": [],
   "execution_count": 1
  },
  {
   "cell_type": "markdown",
   "id": "7b23d509",
   "metadata": {},
   "source": [
    "Assuming you have a PDF uploaded on the same machine, please set the path for the file. \n",
    "\n",
    "Also, if you want to flex your GPU-please switch to a bigger model although the featherlight models work perfectly for this task:"
   ]
  },
  {
   "cell_type": "code",
   "id": "60d0061b-8b8c-4353-850f-f19466a0ae2d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:52.278630Z",
     "start_time": "2024-10-29T16:12:52.275892Z"
    }
   },
   "source": [
    "pdf_path = './resources/2402.13116v4.pdf'\n",
    "DEFAULT_MODEL = \"mlx-community/Qwen2.5-1.5B-Instruct-4bit\""
   ],
   "outputs": [],
   "execution_count": 2
  },
  {
   "cell_type": "code",
   "id": "21029232-ac5f-42ca-b26b-baad5b2f49b7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:20:12.690130Z",
     "start_time": "2024-10-29T16:20:12.687947Z"
    }
   },
   "source": [
    "import os\n",
    "import warnings\n",
    "from typing import Optional\n",
    "\n",
    "import PyPDF2\n",
    "from mlx_lm import load, generate\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "warnings.filterwarnings('ignore')"
   ],
   "outputs": [],
   "execution_count": 18
  },
  {
   "cell_type": "markdown",
   "id": "203c22eb",
   "metadata": {},
   "source": [
    "Let's make sure we don't stub our toe by checking if the file exists"
   ]
  },
  {
   "cell_type": "code",
   "id": "153d9ece-37a4-4fff-a8e8-53f923a2b0a0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:52.678111Z",
     "start_time": "2024-10-29T16:12:52.676112Z"
    }
   },
   "source": [
    "def validate_pdf(file_path: str) -> bool:\n",
    "    if not os.path.exists(file_path):\n",
    "        print(f\"Error: File not found at path: {file_path}\")\n",
    "        return False\n",
    "    if not file_path.lower().endswith('.pdf'):\n",
    "        print(\"Error: File is not a PDF\")\n",
    "        return False\n",
    "    return True"
   ],
   "outputs": [],
   "execution_count": 4
  },
  {
   "cell_type": "markdown",
   "id": "5a362ac3",
   "metadata": {},
   "source": [
    "Convert PDF to a `.txt` file. This would simply read and dump the contents of the file. We set the maximum characters to 100k. \n",
    "\n",
    "For people converting their favorite novels into a podcast, they will have to add extra logic of going outside the models context length."
   ]
  },
  {
   "cell_type": "code",
   "id": "b57c2d64-3d75-4aeb-b4ee-bd1661286b66",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:52.695963Z",
     "start_time": "2024-10-29T16:12:52.692382Z"
    }
   },
   "source": [
    "def extract_text_from_pdf(file_path: str, max_chars: int = 100000) -> Optional[str]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "\n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            # Create PDF reader object\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "\n",
    "            # Get total number of pages\n",
    "            num_pages = len(pdf_reader.pages)\n",
    "            print(f\"Processing PDF with {num_pages} pages...\")\n",
    "\n",
    "            extracted_text = []\n",
    "            total_chars = 0\n",
    "\n",
    "            # Iterate through all pages\n",
    "            for page_num in range(num_pages):\n",
    "                # Extract text from page\n",
    "                page = pdf_reader.pages[page_num]\n",
    "                text = page.extract_text()\n",
    "\n",
    "                # Check if adding this page's text would exceed the limit\n",
    "                if total_chars + len(text) > max_chars:\n",
    "                    # Only add text up to the limit\n",
    "                    remaining_chars = max_chars - total_chars\n",
    "                    extracted_text.append(text[:remaining_chars])\n",
    "                    print(f\"Reached {max_chars} character limit at page {page_num + 1}\")\n",
    "                    break\n",
    "\n",
    "                extracted_text.append(text)\n",
    "                total_chars += len(text)\n",
    "                print(f\"Processed page {page_num + 1}/{num_pages}\")\n",
    "\n",
    "            final_text = '\\n'.join(extracted_text)\n",
    "            print(f\"\\nExtraction complete! Total characters: {len(final_text)}\")\n",
    "            return final_text\n",
    "\n",
    "    except PyPDF2.PdfReadError:\n",
    "        print(\"Error: Invalid or corrupted PDF file\")\n",
    "        return None\n",
    "    except Exception as e:\n",
    "        print(f\"An unexpected error occurred: {str(e)}\")\n",
    "        return None\n"
   ],
   "outputs": [],
   "execution_count": 5
  },
  {
   "cell_type": "markdown",
   "id": "e023397b",
   "metadata": {},
   "source": [
    "Helper function to grab meta info about our PDF"
   ]
  },
  {
   "cell_type": "code",
   "id": "0984bb1e-d52c-4cec-a131-67a48061fabc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:52.708858Z",
     "start_time": "2024-10-29T16:12:52.706772Z"
    }
   },
   "source": [
    "# Get PDF metadata\n",
    "def get_pdf_metadata(file_path: str) -> Optional[dict]:\n",
    "    if not validate_pdf(file_path):\n",
    "        return None\n",
    "\n",
    "    try:\n",
    "        with open(file_path, 'rb') as file:\n",
    "            pdf_reader = PyPDF2.PdfReader(file)\n",
    "            metadata = {\n",
    "                'num_pages': len(pdf_reader.pages),\n",
    "                'metadata': pdf_reader.metadata\n",
    "            }\n",
    "            return metadata\n",
    "    except Exception as e:\n",
    "        print(f\"Error extracting metadata: {str(e)}\")\n",
    "        return None"
   ],
   "outputs": [],
   "execution_count": 6
  },
  {
   "cell_type": "markdown",
   "id": "6019affc",
   "metadata": {},
   "source": [
    "Finally, we can run our logic to extract the details from the file"
   ]
  },
  {
   "cell_type": "code",
   "id": "63848943-79cc-4e21-8396-6eab5df493e0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:53.091773Z",
     "start_time": "2024-10-29T16:12:52.712922Z"
    }
   },
   "source": [
    "# Extract metadata first\n",
    "print(\"Extracting metadata...\")\n",
    "metadata = get_pdf_metadata(pdf_path)\n",
    "if metadata:\n",
    "    print(\"\\nPDF Metadata:\")\n",
    "    print(f\"Number of pages: {metadata['num_pages']}\")\n",
    "    print(\"Document info:\")\n",
    "    for key, value in metadata['metadata'].items():\n",
    "        print(f\"{key}: {value}\")\n",
    "\n",
    "# Extract text\n",
    "print(\"\\nExtracting text...\")\n",
    "extracted_text = extract_text_from_pdf(pdf_path)\n",
    "\n",
    "# Display first 500 characters of extracted text as preview\n",
    "if extracted_text:\n",
    "    print(\"\\nPreview of extracted text (first 500 characters):\")\n",
    "    print(\"-\" * 50)\n",
    "    print(extracted_text[:500])\n",
    "    print(\"-\" * 50)\n",
    "    print(f\"\\nTotal characters extracted: {len(extracted_text)}\")\n",
    "\n",
    "# Optional: Save the extracted text to a file\n",
    "if extracted_text:\n",
    "    output_file = './resources/extracted_text.txt'\n",
    "    with open(output_file, 'w', encoding='utf-8') as f:\n",
    "        f.write(extracted_text)\n",
    "    print(f\"\\nExtracted text has been saved to {output_file}\")"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting metadata...\n",
      "\n",
      "PDF Metadata:\n",
      "Number of pages: 43\n",
      "Document info:\n",
      "/Author: \n",
      "/CreationDate: D:20241022021202Z\n",
      "/Creator: LaTeX with hyperref\n",
      "/Keywords: \n",
      "/ModDate: D:20241022021202Z\n",
      "/PTEX.Fullbanner: This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5\n",
      "/Producer: pdfTeX-1.40.25\n",
      "/Subject: \n",
      "/Title: \n",
      "/Trapped: /False\n",
      "\n",
      "Extracting text...\n",
      "Processing PDF with 43 pages...\n",
      "Processed page 1/43\n",
      "Processed page 2/43\n",
      "Processed page 3/43\n",
      "Processed page 4/43\n",
      "Processed page 5/43\n",
      "Processed page 6/43\n",
      "Processed page 7/43\n",
      "Processed page 8/43\n",
      "Processed page 9/43\n",
      "Processed page 10/43\n",
      "Processed page 11/43\n",
      "Processed page 12/43\n",
      "Processed page 13/43\n",
      "Processed page 14/43\n",
      "Processed page 15/43\n",
      "Processed page 16/43\n",
      "Reached 100000 character limit at page 17\n",
      "\n",
      "Extraction complete! Total characters: 100016\n",
      "\n",
      "Preview of extracted text (first 500 characters):\n",
      "--------------------------------------------------\n",
      "1\n",
      "A Survey on Knowledge Distillation of Large\n",
      "Language Models\n",
      "Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1,\n",
      "Can Xu5, Dacheng Tao6, Tianyi Zhou2\n",
      "1The University of Hong Kong2University of Maryland3Microsoft\n",
      "4University of Technology Sydney5Peking University6The University of Sydney\n",
      "{shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu\n",
      "ckcheng@cs.hku.hk jl0725@connect.hku.hk\n",
      "Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati\n",
      "--------------------------------------------------\n",
      "\n",
      "Total characters extracted: 100016\n",
      "\n",
      "Extracted text has been saved to ./resources/extracted_text.txt\n"
     ]
    }
   ],
   "execution_count": 7
  },
  {
   "cell_type": "markdown",
   "id": "946d1f59",
   "metadata": {},
   "source": [
    "### Pre-Processing\n",
    "\n",
    "Now let's proceed to justify our distaste for writing regex and use that as a justification for a LLM instead:\n",
    "\n",
    "At this point, have a text file extracted from a PDF of a paper. Generally PDF extracts can be messy due to characters, formatting, Latex, Tables, etc. \n",
    "\n",
    "One way to handle this would be using regex, instead we can also prompt the feather light models to clean up our text for us. \n",
    "\n",
    "Please try changing the `SYS_PROMPT` below to see what improvements you can make:"
   ]
  },
  {
   "cell_type": "code",
   "id": "7c0828a5-964d-475e-b5f5-40a04e287725",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:53.097641Z",
     "start_time": "2024-10-29T16:12:53.095850Z"
    }
   },
   "source": [
    "SYS_PROMPT = \"\"\"\n",
    "You are a world class text pre-processor, here is the raw data from a PDF, please parse and return it in a way that is crispy and usable to send to a podcast writer.\n",
    "\n",
    "The raw data is messed up with new lines, Latex math and you will see fluff that we can remove completely. Basically take away any details that you think might be useless in a podcast author's transcript.\n",
    "\n",
    "Remember, the podcast could be on any topic whatsoever so the issues listed above are not exhaustive\n",
    "\n",
    "Please be smart with what you remove and be creative ok?\n",
    "\n",
    "Remember DO NOT START SUMMARIZING THIS, YOU ARE ONLY CLEANING UP THE TEXT AND RE-WRITING WHEN NEEDED\n",
    "\n",
    "Be very smart and aggressive with removing details, you will get a running portion of the text and keep returning the processed text.\n",
    "\n",
    "PLEASE DO NOT ADD MARKDOWN FORMATTING, STOP ADDING SPECIAL CHARACTERS THAT MARKDOWN CAPATILISATION ETC LIKES\n",
    "\n",
    "ALWAYS start your response directly with processed text and NO ACKNOWLEDGEMENTS about my questions ok?\n",
    "Here is the text:\n",
    "\"\"\""
   ],
   "outputs": [],
   "execution_count": 8
  },
  {
   "cell_type": "markdown",
   "id": "fd393fae",
   "metadata": {},
   "source": [
    "Instead of having the model process the entire file at once, as you noticed in the prompt-we will pass chunks of the file. \n",
    "\n",
    "One issue with passing chunks counted by characters is, we lose meaning of words so instead we chunk by words:"
   ]
  },
  {
   "cell_type": "code",
   "id": "24e8a547-9d7c-4e2f-be9e-a3aea09cce76",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:12:53.103772Z",
     "start_time": "2024-10-29T16:12:53.101483Z"
    }
   },
   "source": [
    "def create_word_bounded_chunks(text, target_chunk_size):\n",
    "    \"\"\"\n",
    "    Split text into chunks at word boundaries close to the target chunk size.\n",
    "    \"\"\"\n",
    "    words = text.split()\n",
    "    chunks = []\n",
    "    current_chunk = []\n",
    "    current_length = 0\n",
    "\n",
    "    for word in words:\n",
    "        word_length = len(word) + 1  # +1 for the space\n",
    "        if current_length + word_length > target_chunk_size and current_chunk:\n",
    "            # Join the current chunk and add it to chunks\n",
    "            chunks.append(' '.join(current_chunk))\n",
    "            current_chunk = [word]\n",
    "            current_length = word_length\n",
    "        else:\n",
    "            current_chunk.append(word)\n",
    "            current_length += word_length\n",
    "\n",
    "    # Add the last chunk if it exists\n",
    "    if current_chunk:\n",
    "        chunks.append(' '.join(current_chunk))\n",
    "\n",
    "    return chunks"
   ],
   "outputs": [],
   "execution_count": 9
  },
  {
   "cell_type": "markdown",
   "id": "5d74223f",
   "metadata": {},
   "source": [
    "Let's load in the model and start processing the text chunks"
   ]
  },
  {
   "cell_type": "code",
   "id": "d04a4f07-b0b3-45ca-8f41-a433e1abe050",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:15:09.822147Z",
     "start_time": "2024-10-29T16:12:53.110588Z"
    }
   },
   "source": "model, tokenizer = load(DEFAULT_MODEL)",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Fetching 9 files:   0%|          | 0/9 [00:00<?, ?it/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "4e7535c6b4a549f7a56233b695e31070"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "model.safetensors:   4%|3         | 31.5M/869M [00:00<?, ?B/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "8418d413786d4ab88b59d59f78d3c51f"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "execution_count": 10
  },
  {
   "cell_type": "code",
   "id": "bbda5241-e890-4402-87dd-514d6761bb9c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:15:09.831076Z",
     "start_time": "2024-10-29T16:15:09.828473Z"
    }
   },
   "source": [
    "def process_chunk(text_chunk, chunk_num):\n",
    "    \"\"\"Process a chunk of text and return both input and output for verification\"\"\"\n",
    "    conversation = [\n",
    "        {\"role\": \"system\", \"content\": SYS_PROMPT},\n",
    "        {\"role\": \"user\", \"content\": text_chunk},\n",
    "    ]\n",
    "\n",
    "    prompt = tokenizer.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)\n",
    "    processed_text = generate(\n",
    "        model=model,\n",
    "        tokenizer=tokenizer,\n",
    "        prompt=prompt,\n",
    "        max_tokens=512,\n",
    "        temp=0.7,\n",
    "    )\n",
    "\n",
    "    # Print chunk information for monitoring\n",
    "    #print(f\"\\n{'='*40} Chunk {chunk_num} {'='*40}\")\n",
    "    print(f\"INPUT TEXT:\\n{text_chunk[:500]}...\")  # Show first 500 chars of input\n",
    "    print(f\"\\nPROCESSED TEXT:\\n{processed_text[:500]}...\")  # Show first 500 chars of output\n",
    "    print(f\"{'=' * 90}\\n\")\n",
    "\n",
    "    return processed_text"
   ],
   "outputs": [],
   "execution_count": 11
  },
  {
   "cell_type": "code",
   "id": "a0183c47-339d-4041-ae83-77fc34931075",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:15:09.840870Z",
     "start_time": "2024-10-29T16:15:09.836581Z"
    }
   },
   "source": [
    "INPUT_FILE = \"./resources/extracted_text.txt\"  # Replace with your file path\n",
    "CHUNK_SIZE = 1000  # Adjust chunk size if needed\n",
    "\n",
    "# Read the file\n",
    "with open(INPUT_FILE, 'r', encoding='utf-8') as file:\n",
    "    text = file.read()\n",
    "\n",
    "chunks = create_word_bounded_chunks(extracted_text, CHUNK_SIZE)\n",
    "num_chunks = len(chunks)"
   ],
   "outputs": [],
   "execution_count": 12
  },
  {
   "cell_type": "code",
   "id": "bb36814f-9310-4734-bf54-e16a5032339e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:15:09.855389Z",
     "start_time": "2024-10-29T16:15:09.853297Z"
    }
   },
   "source": [
    "num_chunks"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "101"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 13
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:15:09.871527Z",
     "start_time": "2024-10-29T16:15:09.869985Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Cell 6: Process the file with ordered output\n",
    "# Create output file name\n",
    "output_file = f\"./resources/clean_{os.path.basename(INPUT_FILE)}\"\n",
    "\n",
    "processed_text = \"\""
   ],
   "id": "59d994601d77baa8",
   "outputs": [],
   "execution_count": 14
  },
  {
   "cell_type": "code",
   "id": "7917dfdd-b3af-44fc-a8c0-2760ace9363e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:20:12.650111Z",
     "start_time": "2024-10-29T16:15:09.884601Z"
    }
   },
   "source": [
    "with open(output_file, 'w', encoding='utf-8') as out_file:\n",
    "    for chunk_num, chunk in enumerate(tqdm(chunks, desc=\"Processing chunks\")):\n",
    "        # Process chunk and append to complete text\n",
    "        processed_chunk = process_chunk(chunk, chunk_num)\n",
    "        processed_text += processed_chunk + \"\\n\"\n",
    "\n",
    "        # Write chunk immediately to file\n",
    "        out_file.write(processed_chunk + \"\\n\")\n",
    "        out_file.flush()"
   ],
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Processing chunks:   0%|          | 0/101 [00:00<?, ?it/s]"
      ],
      "application/vnd.jupyter.widget-view+json": {
       "version_major": 2,
       "version_minor": 0,
       "model_id": "cce783fd5a9049699d8e8256f04c385e"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INPUT TEXT:\n",
      "1 A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu1, Ming Li2, Chongyang Tao3, Tao Shen4, Reynold Cheng1, Jinyang Li1, Can Xu5, Dacheng Tao6, Tianyi Zhou2 1The University of Hong Kong2University of Maryland3Microsoft 4University of Technology Sydney5Peking University6The University of Sydney {shawnxxh,chongyangtao,hishentao }@gmail.com {minglii,tianyi }@umd.edu ckcheng@cs.hku.hk jl0725@connect.hku.hk Abstract —In the era of Large Language Models (LLMs), Knowledge Distillati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "This is a survey on Knowledge Distillation (KD) within Large Language Models. It highlights the pivotal role of KD in transferring capabilities from leading proprietary LLMs to open-source models, such as GPT-4. Additionally, it underscores the crucial role of KD in compressing open-source LLMs and facilitating their self-improvement. The survey is conducted by Xiaohan Xu, Ming Li, Chongyang Tao from the University of Hong Kong and Microsoft respectively. It is titled \"Knowledge Distillation in ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "advanced knowledge to smaller models and its utility in model compression and self- improvement. Our survey is meticulously structured around three foundational pillars: algorithm ,skill, and verticalization – providing a comprehensive examination of KD mechanisms, the enhancement of specific cognitive abilities, and their practical implications across diverse fields. Crucially, the survey navigates the interaction between data augmentation (DA) and KD, illustrating how DA emerges as a powerful ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: algorithm, skill, and verticalization – providing a comprehensive examination of knowledge distillation (KD) mechanisms. The survey highlights the enhancement of specific cognitive abilities, illustrating their practical implications across diverse fields. Crucially, it navigates the interaction between data augmentation (DA) and...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distillation and proposing future research directions. By bridging the gap between proprietary and open-source LLMs, this survey underscores the potential for more accessible, efficient, and powerful AI solutions. Most importantly, we firmly advocate for compliance with the legal terms that regulate the use of LLMs, ensuring ethical and lawful application of KD of LLMs. An associated Github repository is available at https://github.com/Tebmer/Awesome-Knowledge-Distillation-of-LLMs. Index Terms —...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "distillation of the raw data to make it more readable:\n",
      "\n",
      "In the evolving landscape of artificial intelligence (AI), proprietary Large Language Models (LLMs) such as GPT-3.5, GPT-4, Gemini, and Claude have emerged as groundbreaking technologies, reshaping our understanding of natural language processing (NLP). These models are characterized by their vast scale and complexity, making them powerful tools for language understanding. However, they also pose significant challenges in terms of scalabili...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "complexity, have unlocked new realms of possibility, from generating human-like text to offering sophisticated problem-solving capabilities. The core significance of these LLMs lies in their emergent abil- ities (Wei et al., 2022a,b; Xu et al., 2024a), a phenomenon where the models display capabilities beyond their explicit training objectives, enabling them to tackle a diverse array of tasks with remarkable proficiency. These models excel in understanding and generation, driving applications fr...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The core significance of Large Language Models (LLMs) lies in their emergent abilities. These models display capabilities beyond explicit training objectives, enabling them to tackle diverse tasks with remarkable proficiency.\n",
      "\n",
      "LLMs excel in understanding and generation. They drive applications from creative to complex problem-solving, impacting various industries.\n",
      "\n",
      "In simpler terms:\n",
      "- \"Proprietary\" models like GPT-4 are versatile and closed-source, while large open-source LLMs with many paramete...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "redefine our interaction with technology. Despite the remarkable capabilities of proprietary LLMs like GPT-4 and Gemini, they are not without their shortcom- ings, particularly when viewed in light of the advantages offered by open-source models. A significant drawback is their limited accessibility and higher cost (OpenAI et al., 2023). These proprietary models often come with substantial usage fees and restricted access, making them less attain- able for individuals and smaller organizations. ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Despite the remarkable capabilities of proprietary LLMs like GPT-4 and Gemini, they have their limitations. Open-source models offer advantages but often come with higher costs (OpenAI et al., 2023). These proprietary models have substantial usage fees and restricted access, making them less accessible to individuals or smaller organizations. In terms of data privacy and security (Wu et al., 2023a), proprietary LLMs frequently send sensitive data to external servers, raising concerns about priva...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "present significant challenges in leveraging the full potential of proprietary LLMs. In contrast to proprietary LLMs, open-source models like LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023a) bring several notable advantages. One of the primaryarXiv:2402.13116v4 [cs.CL] 21 Oct 2024 2 benefits of open-source models is their accessibility and adaptability. Without the constraints of licensing fees or restrictive usage policies, these models are more readily available to a broader rang...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The challenges in leveraging proprietary Large Language Models (LLMs) are significant. In contrast to proprietary LLMs, open-source models like LLaMA (Touvron et al., 2023) and Mistral (Jiang et al., 2023a) offer several notable advantages. One primary benefit of open-source models is their accessibility and adaptability, which are absent in proprietary LLMs. Without licensing fees or restrictive usage policies, these models can be more readily available to a broader range of users, from individ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "resources compared to their proprietary counterparts. One of the most significant limitations is the smaller model scale, which often results in lower per- formance on real-world tasks with a bunch of instruc- tions (Zheng et al., 2023a). These models, with fewer pa- rameters, may struggle to capture the depth and breadth of knowledge embodied in larger models like GPT-4. Ad- ditionally, the pre-training investment in these open-source models is typically less substantial. This reduced investmen...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The limitations of open-source models compared to proprietary ones, one of the most significant being their smaller model scale. This results in lower performance on real-world tasks, especially with a large amount of instructions (Zheng et al., 2023a). These models, with fewer parameters, struggle to capture the depth and breadth of knowledge present in larger proprietary ones like GPT-4. Additionally, open-source models receive less investment during the pre-training phase, typically leading t...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "particularly evident when these models are compared to the highly fine-tuned proprietary LLMs, which are often tailored to excel in a wide array of complex scenarios (OpenAI et al., 2023). Primarily, recognizing the disparities between propri- etary and open-source LLMs, KD techniques have surged as a means to bridge the performance gap between these models (Gou et al., 2021; Gupta and Agrawal, 2022). Knowl- edge distillation, in this context, involves leveraging the more advanced capabilities o...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Here is the cleaned and processed text:\n",
      "\n",
      "**Abstract:**  \n",
      "In this paper, we discuss knowledge distillation (KD) techniques for fine-tuning proprietary large language models.  \n",
      "**Introduction:**  \n",
      "When comparing proprietary LLMs to highly fine-tuned open-source LLMs, significant disparities are evident.  \n",
      "**Background:**  \n",
      "The disparity arises from the proprietary models' focus on complex scenarios, whereas open-source LLMs strive to be general-purpose.  \n",
      "**Problem Statement:**  \n",
      "Recognizing these...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "paradigm to achieve knowledge distillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a specific skill or domain (Taori et al., 2023). Secondly, KD still retains its fundamental role in compressing LLMs, making them more efficient without significant loss in performance. (Gu et al., 2024; Agarwal et al., 2024). More recently, the strategy of employing open-source LLMs as teachers for their own self-improvement has emerged as a promisi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "paradigm to achieve knowledge distillation of LLMs, where a small seed of knowledge is used to prompt the LLM to generate more data with respect to a specific skill or domain (Taori et al., 2023). Secondly, KD still retains its fundamental role in compressing LLMs, making them more efficient without significant loss in performance. (Gu et al., 2024; Agarwal et al., 2024). More recently, the strategy of employing open-source LLMs as teachers for their own self-improvement has emerged as a promisi...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "compression for efficiency, and 3) an emerging trend of self-improvement via self-generated knowledge. (e.g., in-context learning (Huang et al., 2022a) and in- struction following (Taori et al., 2023)), improved align- ment with user intents (e.g., human values/principles (Cui et al., 2023a), and thinking patterns like chain-of-thought (CoT) (Mukherjee et al., 2023)), and NLP task specialization (e.g., semantic understanding (Ding et al., 2023a), and code generation (Chaudhary, 2023)). These ski...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "compression for efficiency, and 3) an emerging trend of self-improvement via self-generated knowledge. (e.g., in-context learning (Huang et al., 2022a) and instruction following (Taori et al., 2023)), improved alignment with user intents (e.g., human values/principles (Cui et al., 2023a)), and NLP task specialization (e.g., semantic understanding (Ding et al., 2023a), and code generation (Chaudhary, 2023)). These skills are crucial for the wide array of applications that LLMs are expected to per...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "been extensively trained and fine-tuned in these areas. The benefits of knowledge distillation in the era of LLMs are multifaceted and transformative (Gu et al., 2024). Through a suite of distillation techniques, the gap between proprietary and open-source models is significantly nar- rowed (Chiang et al., 2023; Xu et al., 2023a) and even filled (Zhao et al., 2023a). This process not only streamlines computational requirements but also enhances the environ- mental sustainability of AI operations...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "been extensively trained fine-tuned in these areas. benefits of knowledge distillation LLMs are multifaceted transformative (Gu et al., 2024). suite distillation techniques gap proprietary open-source narrowed. even filled (Zhao et al., 2023a). streamlines computational requirements enhances AI operations. open-source proficient lesser overhead foster more accessible equitable capabilities, encourage diversity innovation growth various industries...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and research domains. The escalating need for a comprehensive survey on the knowledge distillation of LLMs stems from the rapidly evolving landscape of AI (OpenAI et al., 2023; Team et al., 2023) and the increasing complexity of these models. As AI continues to penetrate various sectors, the ability to effi- ciently and effectively distill knowledge from proprietary LLMs to open-source ones becomes not just a technical aspiration but a practical necessity. This need is driven by the growing dema...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Research domains are crucial for a comprehensive survey on knowledge distillation in large language models (LLMs). The rapid evolution of AI, driven by organizations like OpenAI and others in the field (OpenAI et al., 2023; Team et al., 2023), highlights the complexity and rapid advancement of these models. As AI technologies continue to impact various sectors, distilling knowledge from proprietary LLMs into open-source ones becomes essential for practical applications. This necessity is fueled ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "ReinforcementLearningoutputsreward RM!(·)distill SupervisedFine-tuningX,Y preferenceRankOptimizationy,1y,2y3y1y2y3≻≻rank…… DataCuration X,YrawdatasynthesizefeedbackFeedback input outputSelf-Knowledge outputinputinput YlabelLabelingExpansion X,YdemonstrationsexpandFeature featureinput,outputextractSec.4Sec.5 Sec.3.1Sec.3.2①②③④ Fig. 2: An overview of this survey on knowledge distillation of large language models. Note that ‘Section’ is abbreviated as ‘Sec.’ in this figure. RM S(·)denotes the stude...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Reinforcement Learning outputs reward RM!(·). Distillation of supervised Fine-tuning X, Y preference. Optimization y1y2y3≻≻rank…… Data Curation: raw data synthesize feedback. Feedback input output Self-Knowledge: X, Y label expansion X, Y demonstrationsexpand Feature. Fig. 2: An overview of this survey on knowledge distillation of large language models. Note that ‘Section’ is abbreviated as 'Sec.' in this figure.\n",
      "\n",
      "RM S(·) denotes the student reward model. 1⃝2⃝3⃝4 denote steps in KD of LLMs.\n",
      "\n",
      "A s...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "future research. Survey Organization. The remainder of this survey is orga- nized into several comprehensive sections, each designed to offer a deep dive into the multifaceted aspects of knowledge distillation within the realm ofLLMs. Following this intro- duction, §2 provides a foundational overview of knowledge distillation, comparing traditional techniques with those emerging in the era of LLMs and highlighting the role of data augmentation (DA) in this context. §3 delves into the approaches ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The future research in knowledge distillation within the realm of Large Language Models (LLMs) is organized into several comprehensive sections. This survey introduces foundational aspects, comparing traditional techniques with those emerging in the era of LLMs to highlight data augmentation's role. Section 2 provides an overview, comparing traditional methods with LLM-specific approaches and demonstrating the importance of data augmentation. Section 3 examines eliciting knowledge from teacher L...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(NLU), genera- tion (NLG), information retrieval, recommendation systems, and the evaluation of text generation. In §5, we venture into domain-specific vertical distillation, showcasing how knowledge distillation techniques are applied within spe- cialized fields such as law, healthcare, finance, and science,illustrating the practical implications and transformative impact of these approaches. The survey suggests open problems in §6, identifying current challenges and gaps in knowledge distillat...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "In this survey, we focus on information retrieval and recommendation systems. We provide an overview of these fields in §5 along with a comparison to traditional recipe methods, demonstrating how they differ. In §6, we highlight open problems in these areas and suggest potential research directions for the future.\n",
      "\n",
      "The survey concludes with a synthesis of our findings, drawing implications for broader AI and NLP research communities. We provide an overview in §2 to set the stage, introducing a s...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "model (teacher) to a smaller, more efficient model (student) (Gou et al., 2021). This technique is pivotal in mitigating the challenges posed by the computational demands and resource constraints of deploying large-scale models in practical applications. Historically, knowledge distillation techniques, prior to the era of LLMs, primarily concentrated on transferring knowledge from complex, often cumbersome neural net- works to more compact and efficient architectures (Sanh et al., 2019; Kim and ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "model to a smaller, more efficient model (student) via knowledge distillation techniques. This technique is crucial in mitigating the challenges posed by computational demands and resource constraints when deploying large-scale models. Historically, knowledge distillation focused on transferring knowledge from complex neural networks to more compact architectures before the era of Large Language Models (LLMs). The process was driven by needing to deploy models in resource-constrained environment...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "CoT-Distill (Hsieh et al., 2023) Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023), Baize (Xu et al., 2023b), Mammoth (Yue et al., 2023a), Mixed Distill (Chenglin et al., 2023) ExpansionSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Code Alpaca (Chaudhary, 2023) Self-Align (Sun et al., 2024b), WizardLM (Xu et al., 2023a), WizardCoder (Luo et al., 2023a), WizardMath (Luo et al., 2023b), AugGPT (Dai et al., 2023a), TDG (He et al., 2023b) CurationUltraChat (Ding et al., 2...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "### Cleaned Text:\n",
      "\n",
      "**CoT-Distill (Hsieh et al., 2023)**  \n",
      "Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023)  \n",
      "Baize (Xu et al., 2023b), Mammoth (Yue et al., 2023a)  \n",
      "Mixed Distill (Chenglin et al., 2023)  \n",
      "\n",
      "**ExpansionSelf-Instruct (Wang et al., 2022a)**  \n",
      "Alpaca (Taori et al., 2023), Code Alpaca (Chaudhary, 2023)  \n",
      "Self-Align (Sun et al., 2024b), WizardLM (Xu et al., 2023a)  \n",
      "WizardCoder (Luo et al., 2023a), WizardMath (Luo et al., 2023b)  \n",
      "AugGPT (Dai et al., 2023a), TDG (He et al., 2...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(Tunstall et al., 2023), CycleAlign (Hong et al., 2023), RLAIF (Lee et al., 2023a), Lion (Jiang et al., 2023b), PERsD (Chen et al., 2023a), GKD (Agarwal et al., 2024) Self-KnowledgeSelf-Instruct (Wang et al., 2022a), Self-Align (Sun et al., 2024b), RLCD (Yang et al., 2024), ImpDistill (Jung et al., 2023), LMSI (Huang et al., 2023a), ReST (Gulcehre et al., 2023), Self-Rewarding (Yuan et al., 2024a), Baize (Xu et al., 2023b), STaR (Zelikman et al., 2022) DistillationSupervised Fine-TuningAlpaca (T...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "CycleAlign, RLAIF, Lion, PERsD, GKD, Self-KnowledgeSelf-Instruct, RLCD, ImpDistill, LMSI, ReST, Self-Rewarding, Baize, STaR, DistillationSupervised Fine-TuningAlpaca, Vicuna, WizardLM, Self-Instruct, Baize, STaR, Divergence and SimilarityDistilGPT, f-Distill, MiniLLM, TED, Cai et al., Bai et al....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "(Gu et al., 2024), GKD (Agarwal et al., 2024), GPT3 Reward (Kwon et al., 2023) Rank Optimization Zephyr (Tunstall et al., 2023), CycleAlign (Hong et al., 2023), Skill DistillationContext FollowingInstruction FollowingSelf-Instruct (Wang et al., 2022a), Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023a), Orca (Mukherjee et al., 2023), Orca 2 (Mitra et al., 2023), WizardMath (Luo et al., 2023b), Llama-GPT4 (Peng et al., 2023a), Multi-turn DialogueVicuna (Chiang ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "(Gu et al., 2024)  \n",
      "GKD (Agarwal et al., 2024)\n",
      "GPT3 Reward(Kwon et al., 2023)  \n",
      "Rank Optimization Zephyr(Tunstall et al., 2023)\n",
      "CycleAlign(Hong et al., 2023)  \n",
      "Skill DistillationContext FollowingInstruction (Wang et al., 2022a)\n",
      "Alpaca(Taori et al., 2023)  \n",
      "Vicuna(Chiang et al., 2023)\n",
      "WizardLM(Xu et al., 2023a)  \n",
      "Orca(Mukherjee et al., 2023)\n",
      "WizardMath(Luo et al., 2023b)  \n",
      "Llama-GPT4(Peng et al., 2023a)\n",
      "Multi-turn DialogueVicuna(Chiang et al., 2023)  \n",
      "Baize(Xu et al., 2023b)\n",
      "UltraLLaMA(Ding et al...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Reward (Kwon et al., 2023), ILF (Scheurer et al., 2023), ALMoST (Kim et al., 2023a), RLEF (Roit et al., 2023), RLAIF (Lee et al., 2023a), Zephy (Tunstall et al., 2023), UltraFeedback (Cui et al., 2023a), ValueCAI (Bai et al., 2022a), Align Honesty (Yang et al., 2023a), SANDBOX (Liu et al., 2023b), Self-Align (Sun et al., 2024b), UltraFeedback (Cui et al., 2023a), RLCD (Yang et al., 2024) AgentTool UsingToolformer (Schick et al., 2023), Graph-ToolFormer (Zhang, 2023), Gorilla (Patil et al., 2023)...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Reward, ILF, ALMoST, RLEF, RLAIF, Zephy, UltraFeedback, ValueCAI, Align Honesty, SANDBOX, Self-Align, UltraFeedback, RLCD, AgentTool UsingToolformer, Graph-ToolFormer, Gorilla, ToolAlpaca, ToolLLM, CRAFT, Confucius, MLLM-Tool, α-UMi, PlanningFireAct, AgentTuning, Lumos, AUTOACT, TPTU-v2, NLP Task SpecializationNLUAugGPT, GPT Annotation....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023a), Mix Distill (Chenglin et al., 2023), Annollm (He et al., 2023a), UDG (Wang et al., 2021a), ZeroGen (Ye et al., 2022), NLGInheritSumm (Xu et al., 2023c), RECOMP (Xu et al., 2024b), MaRio (Ramnath et al., 2023), ID (Jung et al., 2023), GPT-3 Labeling (Wang et al., 2021b), BioGPT (Guo et al., 2023a), ChatGPT NMT (Yang and Nicolai, 2023), Information RetrievalQUILL (Srinivasan et al., 2022), Promptgator (Dai et al., 2023b), InPars (Bonifacio et al., 2022), AugTriever (Meng et al., 2023), (Su...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Mix Distill, Chenglin et al., 2023; Annollm, He et al., 2023a; UDG, Wang et al., 2021a; ZeroGen, Ye et al., 2022; NLGInheritSumm, Xu et al., 2023c; RECOMP, Xu et al., 2024b; MaRio, Ramnath et al., 2023; ID, Jung et al., 2023; GPT-3 Labeling, Wang et al., 2021b; BioGPT, Guo et al., 2023a; ChatGPT NMT, Yang and Nicolai, 2023; Information RetrievalQUILL, Srinivasan et al., 2022; Promptgator, Dai et al., 2023b; InPars, Bonifacio et al., 2022; AugTriever, Meng et al., 2023; Sun et al., 2023a; RankVic...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023) Phi-1 (Gunasekar et al., 2023), PERsD (Chen et al., 2023a), MFTCoder (Liu et al., 2023d), WaveCoder (Yu et al., 2024), Code Clean (Jain et al., 2023), Multi-ModalityLLaVA (Liu et al., 2023e), SVIT (Zhao et al., 2023b), LVIS-Instruct4V (Wang et al., 2023e), Shikra (Chen et al., 2023c), LSKD (Park et al., 2023), DetGPT (Pi et al., 2023; Zhao et al., 2023c), LRV (Liu et al., 2023f), NExT-GPT (Wu et al., 2023b), Valley (Luo et al., 2023d), ILuvUI (Jiang et al., 2023d), StableLLaVA (Li et ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Training a small student model alongside the larger teacher for knowledge transfer....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "to mimic the output of a larger teacher network, often through techniques like soft target training, where the student learns from the softened softmax output of the teacher. Please refer to the survey (Gou et al., 2021) for more details on general knowledge distillation techniques in AI and DL. In contrast, the advent of LLMs has revolutionized the knowledge distillation landscape. The current era of knowledge distillation in LLMs shifts the focus from mere architecture compression to knowledge...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "To mimic a larger teacher network often through techniques like soft target training, where the student learns from the softened softmax output of the teacher. Please refer to the survey (Gou et al., 2021) for more details on general knowledge distillation techniques in AI and DL. In contrast, the advent of LLMs has revolutionized the knowledge distillation landscape. The current era in knowledge distillation for Large Language Models (LLMs) shifts the focus from mere architecture compression to...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "focus in LLM-based knowledge distillation is to elicit the specific knowledge these models have. The key to this modern approach lies in heuristic and carefully designed prompts, which are used to elicit specific knowledge (Ding et al., 2023b) or capabilities (Chaudhary, 2023) from the LLMs. These prompts are crafted to tap into the LLM’s understanding and capabilities in various domains, ranging from natural language understanding (He et al., 2023a) to more complex cognitive tasks like reason- ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "focus in LLM-based knowledge distillation is to elicit the specific knowledge these models have. The key to this modern approach lies in heuristic and carefully designed prompts, which are used to elicit specific knowledge (Ding et al., 2023b) or capabilities from the LLMs. These prompts are crafted to tap into the LLM’s understanding and capabilities in various domains, ranging from natural language processing (He et al., 2023a) to more complex cognitive tasks like reasoning and problem-solving...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distillation also em- phasizes the transfer of more abstract qualities such as reasoning patterns (Mitra et al., 2023), preference align- ment (Cui et al., 2023a), and value alignment (Sun et al., 2024b). This is in stark contrast to the earlier focus on output replication (Taori et al., 2023), indicating a shift towards a more holistic and comprehensive transfer of cognitive capabilities. The current techniques involve not just the replication of outputs, but also the emulation of the thought p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "distillation also emphasizes the transfer of more abstract qualities such as reasoning patterns (Mitra et al., 2023), preference alignment (Cui et al., 2023a), and value alignment (Sun et al., 2024b). This is in stark contrast to the earlier focus on output replication (Taori et al., 2023), indicating a shift towards a more holistic and comprehensive transfer of cognitive capabilities. The current techniques involve not just the replication of outputs, but also the emulation of thought processes...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "knowledge distillation. Unlike traditional DA techniques such as paraphrasing (Gangal et al., 2022) or back-translation (Longpre et al., 2019), which primarily aim at expanding the training dataset in a somewhat mechanical manner, DA within the context of LLMs focuses on the generation of novel, context-rich training data tailored to specific domains and skills.The relationship between DA and KD in LLMs is both symbiotic and foundational. By leveraging a set of seed knowledge, KD employs DA to p...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Knowledge Distillation in Language Models. Unlike traditional DA techniques such as paraphrasing or back-translation, which primarily aim at expanding the training dataset in a somewhat mechanical manner, DA within the context of LLMs focuses on generating novel and context-rich training data tailored to specific domains and skills. The relationship between DA and Knowledge Distillation (KD) in LLMs is symbiotic, foundational. By leveraging a set of seed knowledge, KD employs DA to prompt LLMs t...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "ensuring that the distilled models not only replicate the teacher model’s output behavior but also embody its deep-seated understanding and cognitive strategies. DA acts as a force multiplier, enabling the distilled mod- els to acquire and refine capabilities that would otherwise require exponentially larger datasets and computational re- sources. It facilitates a more effective transfer of knowledge, focusing on the qualitative aspects of learning rather than quantitative expansion. This strate...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Ensuring that distilled models replicate the teacher model's output behavior while embodying its deep-seated understanding and cognitive strategies is a critical task. The role of the Distillation Agent (DA) acts as a force multiplier, allowing distilled models to acquire and refine capabilities that would otherwise require exponentially larger datasets and computational resources. This facilitates a more effective transfer of knowledge, focusing on the qualitative aspects rather than quantitati...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Building on the discussions introduced earlier, this survey aims to comprehensively explore the landscape of knowl- edge distillation within the context of LLMs, following a meticulously structured taxonomy as in Figure 3. The survey’s scope is delineated through three primary facets: KD Algorithms, Skill Distillation, and Verticalization Dis- tillation. Each facet encapsulates a range of subtopics and methodologies. It’s important to note that KD algorithms provide the technical foundations for...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Building on the discussions introduced earlier, this survey aims to comprehensively explore the landscape of knowledge distillation within the context of Large Language Models (LLMs). The survey's scope is delineated through three primary facets: Knowledge Distillation Algorithms, Skill Distillation, and Verticalization Distillation. Each facet encapsulates a range of subtopics and methodologies.\n",
      "\n",
      "Key components include KD Algorithms, which provide the technical foundations for skill distillatio...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023), curation (Gu- nasekar et al., 2023), feature understanding (Agarwal et al., 2024), feedback mechanisms (Tunstall et al., 2023), and self- knowledge generation (Wang et al., 2022a). This exploration seeks to uncover the various ways in which knowledge can be identified, expanded, and curated for effective dis- tillation. The ‘ distillation ’ subsection examines learning ap- proaches like supervised fine-tuning (SFT) (Wang et al., 2022a), divergence minimization (Agarwal et al., 2024),...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "al., 2023), curation (Gu- nasekar et al., 2023), feature understanding (Agarwal et al., 2024), feedback mechanisms (Tunstall et al., 2023), and self- knowledge generation (Wang et al., 2022a). This exploration seeks to uncover the various ways in which knowledge can be identified, expanded, and curated for effective distribution. The ‘distillation’ subsection examines learning approaches like supervised fine-tuning (SFT) (Wang et al., 2022a), divergence minimization (Agarwal et al., 2024), reinf...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "retrieval-augmented generation (RAG) Capa- bility. In the realm of alignment (Mitra et al., 2023; Tun- stall et al., 2023), the survey investigates thinking patterns, persona/preference modeling, and value alignment. The ‘agent’ category delves into skills such as Tool Using and Planning. NLP task specialization (Dai et al., 2023a; Jung et al., 2023; Chaudhary, 2023) is scrutinized through lenses like natural language understanding (NLU), natural lan- guage generation (NLG), information retrieva...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "retrieval-augmented generation (RAG) Capa- bility. In the realm of alignment (Mitra et al., 2023; Tun- stall et al., 2023), the survey investigates thinking patterns, persona/preference modeling, and value alignment. The ‘agent’ category delves into skills such as Tool Using and Planning.\n",
      "\n",
      "NLP task specialization (Dai et al., 2023a; Jung et al., 2023; Chaudhary, 2023) is scrutinized through lenses like natural language understanding (NLU), natural lan- guage generation (NLG), information retriev...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023a), Finance (Zhang and Yang, 2023), Science (Zhang et al., 2024), among others. This exploration not only showcases the practical implications of KD tech- niques but also highlights their transformative impact on domain-specific AI solutions. Through these facets, this survey provides a compre- hensive analysis of KD in LLMs, guiding researchers and practitioners through methodologies, challenges, and op- portunities in this rapidly evolving domain. Declaration. This survey represents our ea...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "This exploration not only showcases the practical implications of KD techniques but also highlights their transformative impact on domain-specific AI solutions. Through these facets, this survey provides a comprehensive analysis of KD in LLMs, guiding researchers and practitioners through methodologies, challenges, and opportunities. This survey represents our earnest effort to provide a comprehensive and insightful overview of knowledge distillation techniques applied to LLMs, focusing on algor...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "foundational paradigms of knowledge dis- tillation, highlighting key methodologies and their impacts across a range of applications. 2.4 Distillation Pipeline in LLM Era SeedKnowledgeSkill/Domain TeacherLLMKnowledgeElicitationStudentModelDistillationAlgorithmsteer driveGeneratedKnowledgeLearningObjectivetrain Fig. 4: An illustration of a general pipeline to distill knowl- edge from a large language model to a student model. The general distillation pipeline of LLMs is a structured and methodical...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Foundational paradigms of knowledge distillation, highlighting key methodologies and their impacts across a range of applications. \n",
      "\n",
      "Distillation pipelines in the LLM era, seed knowledge from teachers to student models, driven by generative algorithms. Fig. 4: An illustration of a general pipeline to distill knowledge from large language models (LLMs) to student models. The LLM-specific pipeline is structured and methodical, transferring knowledge from sophisticated teacher models to less comple...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "seen in Figure 2. I. Target Skill or Domain Steering Teacher LLM. The first stage involves directing the teacher LLM towards a specific target skill or domain. This is achieved through care- fully crafted instructions or templates that guide the LLM’s focus. These instructions are designed to elicit responses that demonstrate the LLM’s proficiency in a particular area, be it a specialized domain like healthcare or law, or a skill such as reasoning or language understanding. II. Seed Knowledge as...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The first stage involves directing the teacher LLM towards a specific target skill or domain. This is achieved through carefully crafted instructions or templates that guide the LLM's focus. These instructions are designed to elicit responses demonstrating proficiency in a particular area, whether specialized like healthcare or law, or general skills such as reasoning or language understanding. Once the target area is defined, feed it to the teacher LLM with seed knowledge comprising a small dat...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "thereby creating more comprehensive and in-depth knowledge examples. III. Generation of Distillation Knowledge. In response to the seed knowledge and steering instructions, the teacher LLM generates knowledge examples. These examples are predominantly in the form of question-and-answer (QA) dialogues or narrative explanations, aligning with the nat- ural language processing/understanding capabilities of the LLM. In certain specialized cases, the outputs may also in- clude logits or hidden featur...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "the specific learning objective of the student model. The loss function is designed to measure and penalize deviations from this target, thereby enabling the student model to improve its understanding of the subject matter. The process is iterative and involves multiple cycles, where new knowledge examples are continuously fed into the system to refine both teacher LLM and student model. The goal is to achieve a state of high accuracy, where the final output reflects an advanced level of underst...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "learning objectives. The loss function quantifies the student model’s performance in replicating or adapting the knowledge from the teacher model. By minimizing this loss, the student model learns to emulate the target skills or domain knowledge of the teacher, thereby acquiring similar capabilities. The process involves iteratively adjusting the student model’s parameters to reduce the discrepancy be- tween its outputs and those of the teacher model, ensuring the effective transfer of knowledge...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Learning objectives. The loss function quantifies the student model's performance in replicating or adapting knowledge from a teacher model. By minimizing this loss, the student learns to emulate target skills or domain knowledge of the teacher, acquiring similar capabilities. The process involves iteratively adjusting parameters to reduce discrepancies between student and teacher model outputs, ensuring effective knowledge transfer. Abstracted into two formulations:\n",
      "\n",
      "1. **Knowledge Elicitation*...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "example ( e.g., (x, y)) from the teacher LLM’s output o(plus the input sin some cases), andpTrepresents the teacher LLM with parameters θT. Given the datasets D(kd) Ibuilt for distillation, we then define a learning objective as L=X ILI(D(kd) I;θS), (2) whereP Idenotes there could be multiple tasks or skills being distilled into one student model, LI(·;·)stands for a specific learning objective, and θSparameterizes the student model. Following our exploration of the distillation pipeline and the...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "This section covers the process of knowledge distillation, categorizing it into two principal steps: 'Knowledge' for eliciting teacher LLM knowledge (Eq.1), and 'Distillation,' which injects this into student models (Eq.2)....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "We will elaborate on these two processes in the subsequent sections. 3.1 Knowledge This section focuses on the approaches to elicit knowledge from teacher LLMs. According to the manners to acquire knowledge, we divided them into Labeling ,Expansion ,Data Curation ,Feature ,Feedback , and Self-Knowledge . Figure 5 shows an illustration of these knowledge elicitation meth- ods. 3.1.1 Labeling Labeling knowledge refers to using a teacher LLM to label the output yfor a given input xas the seed knowl...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "We will elaborate on these two processes in the subsequent sections. 3.1 Knowledge This section focuses on the approaches to elicit knowledge from teacher LLMs, according to their acquisition methods. According to these manners, we divide them into Labeling , Expansion , Data Curation , Feature Extraction (Feature ), Feedback Loop, and Self-Knowledge . Figure 5 illustrates these knowledge elicitation methods. **3.1 Labeling** This method involves using a teacher LLM to label the output y for eac...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "y∼pT(y|I⊕c⊕x)}. (3) Input xcould be sourced from existing NLP task datasets, which serve as typical reservoirs for distillation efforts. Numerous works have sought to harness the capa- bilities of powerful LLMs as teachers for annotating dataset samples across a range of tasks. For instance, efforts in natural language understanding involve using LLMs to cat- egorize text (Gilardi et al., 2023; Ding et al., 2023a; He et al., 2023a), while in natural language generation, LLMs assist in generating...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Rather than concentrating on specific tasks, many researchers have been exploring the use of Large Language Models (LLMs) as powerful teachers for distillation efforts. This approach aims to leverage the capabilities of LLMs, which are capable of generating high-quality text on a wide range of topics. Researchers have sought to harness the capabilities of LLMs by using them as teachers for annotating dataset samples across various tasks. This has led to significant advancements in the field of n...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "current works focus on labeling outputs based on instructions, thereby teaching student models to solve tasks in a more flexible way by following in- structions. Collections of various NLP tasks, complemented by instructional templates, serve as valuable input sources forx. For instance, FLAN-v2 collections (Longpre et al., 2023) offers extensive publicly available sets of tasks with instructions, which are labeled with responses generated by teacher LLMs in Orca (Mukherjee et al., 2023; Mitra e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The current focus in research is on classifying outputs based on instructions. This approach enables student models to solve tasks more flexibly by adhering to given instructions. Various Natural Language Processing (NLP) task collections, complemented by instructional templates, serve as valuable input sources. For instance, the FLAN-v2 collection (Longpre et al., 2023) provides extensive publicly available sets of tasks with instructions. These are labeled by responses generated from teacher L...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "instructions Ior demonstrations c. A commonly used in- struction type for guiding labeling is chain-of-thought (CoT) prompt (Hsieh et al., 2023; Fu et al., 2023; Magister et al., 2023). Mukherjee et al. (2023) add multiple system messages (e.g. “You must generate a detailed and long answer.” or “explain like I’m five, think step-by-step”) to elicit rich signals. Yue et al. (2023a) and Chenglin et al. (2023) la- bel a hybrid of knowledge of chain-of-thought (CoT) and program-of-thought (PoT) rati...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The labeling approach is simple and effective, but it faces certain limitations. Primarily, the input data has a limited scale and variety.\n",
      "\n",
      "In real-world applications involving user conversations, there are concerns regarding the privacy of data....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "involved. To address these limitations, various expansion methods have been proposed (Wang et al., 2022a; Taori et al., 2023; Chaud- hary, 2023; Si et al., 2023; Ji et al., 2023a; Luo et al., 2023b,a; Wu et al., 2023c; Sun et al., 2024b; Xu et al., 2023a; Guo et al., 2023c; Rozi `ere et al., 2023; West et al., 2022). These methods take the demonstrations as seed knowledge and aim to expand a large scale and various data by in-context learning. A key characteristic of these expansion methods is t...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Involvement of various expansion methods to address limitations. These include approaches like Wang et al., 2022a; Taori et al., 2023; Chaudhary, 2023; Si et al., 2023; Ji et al., 2023a; Luo et al., 2023b, a and Wu et al., 2023c; Sun et al., 2024b; Xu et al., 2023a; Guo et al., 2023c; Rozi `ere, et al., 2023; West et al., 2022. These methods utilize demonstrations as seed knowledge to expand a large scale and diverse data through in-context learning capabilities. A key characteristic is the use ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "𝑚Meta-Information𝑐Demonstrations𝑥𝐼 𝑦 FilterFeedback ExtractFeature𝑥𝑦 DistributionIntermediateFeature 𝑥Input𝑦Output𝐼Instruction𝑦! 𝑦\" 𝑦# 𝑥GuideFeedback𝑦#∗ 𝑦# Feedback Self-Knowledge StudentTeacher Generate≻≻𝑦\" 𝑦! 𝑦# 𝑥 𝑥& CorrectExpand𝑐 Fig. 5: An illustration of different knowledge elicitation methods from teacher LLMs. Labeling : The teacher generates the output from the input; Expansion : The teacher generates samples similar to the given demonstrations through in- context learning; Data Curatio...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "In this context, we discuss knowledge elicitation methods from teacher LLMs. The goal is to leverage the ability of large language models (LLMs) in generating human-like outputs. The methods are categorized into labeling, expansion through context learning, data synthesis according to meta-information (such as a topic or an entity), feature extraction by feeding the data into the teacher, feedback on student-generated outputs (e.g., preferences or corrections), and self-knowledge. The process in...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "xand yrepresent the new input- output pairs generated by the teacher LLM. The input x is generated based on a set of input-output demonstrations c. The output yis then generated in response to the new input xunder the guidance of an instruction I. Note that the demonstrations could be predefined or dynamically updated by adding the newly generated samples. Expansion techniques have been widely utilized to extract extensive instruction-following knowledge from teacher LLMs. Wang et al. (2022a) fi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "xand yrepresent new input-output pairs generated by a teacher's LLM. The input x is derived from predefined demonstrations c, while the output yis generated in response to new inputs under guidance I. The teacher LLM's demonstrations can be either predefined or dynamically updated by adding new samples to the pool. Expansion techniques have been widely used to extract extensive instruction-following knowledge from teacher LLMs, as demonstrated by Wang et al. (2022a) and Taori et al. (2023). Iter...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "diversity and coverage during expansion, Wu et al. (2023c) and (Sun et al., 2024b) prompt the teacher LLM to generate instructions corresponding to some specific topics. Xu et al. (2023a) propose an Evol-Instruct method to ex- pand the instructions from two dimensions: difficulty (e.g. rewriting the question to be more complex) and diversity (e.g. generating more long-tailed instructions). This Evol- Instruct method is domain-agnostic and has been used to expand the distillation of coding (Luo e...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "diversity and coverage during expansion, Wu et al. (2023c) and Sun et al., 2024b) prompt the teacher LLM to generate instructions corresponding to some specific topics. Xu et al., (2023a) propose an Evol-Instruct method to expand the instructions from two dimensions: difficulty (e.g. rewriting the question to be more complex) and diversity (e.g. generating more long-tailed instructions). This Evol-Instruct method is domain-agnostic and has been used to expand the distillation of coding (Luo et a...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "automatically identifies challenging sub- groups within data and generates new samples for these subgroups using LLMs through in-context learning. In summary, the expansion method leverages the in- context learning strengths of LLMs to produce more var- ied and extensive datasets with both inputs and outputs. However, the quality and diversity of the generated data are heavily reliant on the teacher LLMs and the initial seed demonstrations. This dependence can lead to a dataset with inherent bia...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "automatically identifies challenging sub-groups within data and generates new samples for these subgroups using LLMs through in-context learning. In summary, the expansion method leverages the in-context learning strengths of LLMs to produce more varied and extensive datasets with both inputs and outputs. However, the quality and diversity of the generated data are heavily reliant on the teacher LLMs and the initial seed demonstrations. This dependence can lead to a dataset with inherent bias fr...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "sponse to the limitations observed in both the Labeling and Expansion approaches. These methods often yield data of variable quality and face constraints in quantity. In Labeling, the seed knowledge is sourced from task datasets, leading to potential noise and dirty data. Meanwhile, in Expansion, the input xis derived from seed demonstrations, which can result in homogeneous data when generated in large quantities. To overcome these challenges, the Data Curation method curates high-quality or la...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Data Curation is a method used to create high-quality or large-scale data by incorporating extensive meta-information as seed knowledge. This approach is distinct from labeling and expansion methods, which often yield data of variable quality but have limitations in quantity. In the Labeling approach, seed knowledge is sourced from task datasets, leading to potential noise and dirty data. In the Expansion approach, input xis are derived from seed demonstrations, resulting in homogeneous data whe...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "process can be meticulously controlled to yield datasets that are not only large in scale but also of high quality. The formulation for Data Curation can be represented as: D(cur)={(x, y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}.(5) In this formulation, mrepresents the diverse meta- information used to guide the synthesis of x, and Iis the instruction guiding teacher LLMs to generate xory. Different studies primarily vary in their source and method of leveraging meta-information. UltraChat (Ding et al., 2023b)...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Data Curation formulation: D(cur)={(x,y)|x∼pT(x|I⊕m), y∼pT(y|I⊕x)}. \n",
      "\n",
      "UltraChat (Ding et al., 2023b) effectively demonstrates the process of curating high-quality, diverse data by leveraging distilled knowledge from three domains: Questions about the World, Creation and Generation , Assistance on Existing Materials . For example:\n",
      "\n",
      "- Under \"Questions about the World,\" they explore 30 meta-topics like Technology and Food & Drink.\n",
      "\n",
      "- The teacher LLMs then use this information to distill broad array...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "substantial scale of 1.5 million instances. UltraChat stands out with its lexical and topical diversity. The UltraLLaMA model, fine- tuned on this data, consistently surpasses other open-source models. Another notable series, phi(Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023), focuses on distilling smaller, high-quality datasets akin to ”textbooks.” Phi-1 (Gunasekar et al., 2023) experiments with synthesizing ”textbook qual- ity” data in the coding domain. Their approach involves distillin...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "substantial scale of **1.5 million** instances.\n",
      "UltraChat stands out with its lexical and topical diversity. The UltraLLaMA model, fine-tuned on this data, consistently surpasses other open-source models. Another notable series focuses on distilling smaller, high-quality datasets akin to \"textbooks.\" Phi(Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023) experiments with synthesizing \"textbook quality\" data in the coding domain. Their approach involves distilling clear, self-contained, instruc...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "times smaller in model size and 100 times smaller in dataset size. MFTCoder (Liu et al., 2023d) utilizes hundreds of Python knowledge points as meta-information to create a CodeExercise Dataset. In contrast, Magicoder (Wei et al., 2023) and WaveCoder (Yu et al., 2024) get raw code collections from open-source code datasets, using this as meta-information for generating instructional data. In the context of NLU tasks, certain studies (Ye et al., 2022; Gao et al., 2023a; Wang et al., 2021a) explor...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Data Curation through teacher LLMs has emerged as a promising technique for synthesizing datasets that are not only high-quality and diverse but also **smaller in model size**. This is achieved by leveraging knowledge from large models, as demonstrated by **MFTCoder (Liu et al., 2023d)**, which utilizes hundreds of Python knowledge points as meta-information to create **CodeExercise** Dataset. In contrast, **Magicoder (Wei et al., 2023)** and **WaveCoder** (Yu et al., 2024) get raw code collecti...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "large in scale. The success of models like phi-1 in specialized domains underscores the efficacy of this method. The abilityto create synthetic datasets will become a crucial technical skill and a key area of focus in AI (Li et al., 2023a). 3.1.4 Feature The previously discussed knowledge elicitation methods are typically applied to powerful black-box models, which are expensive and somewhat unreproducible due to calling API. In contrast, white-box distillation offers a more trans- parent and ac...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The success of models like phi-1 in specialized domains underscores the efficacy of this method. The ability to create synthetic datasets will become a crucial technical skill and a key area of focus in AI. Li et al., 2023a). White-box distillation offers a more transparent and accessible approach for researchers, leveraging the output distributions, intermediate features, or activations from teacher models. This method has been predominantly studied for smaller encoder-based LMs with fewer than...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023; Liang et al., 2023a; Gu et al., 2024; Agarwal et al., 2024; Liu et al., 2023a; Wen et al., 2023; Wan et al., 2024a; Zhao and Zhu, 2023; Qin et al., 2023b; Boizard et al., 2024; Zhong et al., 2024). The typical method for acquiring this feature knowledge involves teacher LLMs annotating the output sequence y with its internal representations. These annotations are then distilled into the student model using methods such as Kullback-Leibler Divergence (KLD). The process of eliciting feature ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2023; Liang et al., 2023a; Gu et al., 2024; Agarwal et al., 2024; Liu et al., 2023a; Wen et al., 2023; Wan et al., 2024a; Zhao and Zhu, 2023; Qin et al., 2023b; Boizard et al., 2024; Zhong et al., 2024. The typical method for acquiring this feature knowledge involves teacher LLMs annotating the output sequence y with its internal representations. These annotations are then distilled into the student model using methods such as Kullback-Leibler Divergence (KLD). The process of eliciting feature k...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distributions (Sanh et al., 2019; Wen et al., 2023). To leverage the rich semantic and syntactic knowledge in intermediate layers of the teacher model, TED (Liang et al., 2023a) designs task-aware layer-wise distillation. They align the student’s hidden representations with those of the teacher at each layer, selectively extracting knowledge pertinent to the target task. Gu et al. (2024) and Agarwal et al. (2024) introduce a novel approach where the student model first generates sequences, terme...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "distributions (Sanh et al., 2019; Wen et al., 2023). To leverage the rich semantic and syntactic knowledge in intermediate layers of the teacher model, TED (Liang et al., 2023a) designs task-aware layer-wise distillation. They align the student’s hidden representations with those of the teacher at each layer, selectively extracting knowledge pertinent to the target task. Gu et al. (2024) and Agarwal et al. (2024) introduce a novel approach where the student model first generates sequences, terme...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "quantizing the LLMs, ensuring minimal loss of performance. Additionally, feature knowledge could serve as a potent source for multi-teacher knowledge distil- lation. Timiryasov and Tastet (2023) leverages an ensemble of GPT-2 and LLaMA as teacher models to extract output distributions. Similarly, FuseLLM (Wan et al., 2024a) inno- vatively combines the capabilities of various LLMs through a weighted fusion of their output distributions, integrating them into a singular LLM. This approach has the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "quantizing the LLMs to ensure minimal loss of performance. Additionally, leveraging feature knowledge from teacher models can serve as a potent source for multi-teacher knowledge distillation. Timiryasov and Tastet (2023) employ an ensemble of GPT-2 and LLaMA as teacher models to extract output distributions. Similarly, FuseLLM (Wan et al., 2024a) innovatively combines the capabilities of various LLMs through a weighted fusion of their output distributions, integrating them into a singular LLM. ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "smaller models, its application is not suitable for black-box LLMs where internal parame- ters are inaccessible. Furthermore, student models distilled from white-box LLMs may underperform compared to their black-box counterparts, as the black-box teacher LLMs (e.g. GPT-4) tend to be more powerful. 3.1.5 Feedback Most previous works predominantly focus on one-way knowledge transfer from the teacher to the student for imitation, without considering feedback from the teacher on the student’s genera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "In previous works focusing on knowledge transfer, a one-way approach is predominantly used where the teacher provides guidance to the student without considering feedback from the teacher. This can lead to underperformance of distilled models compared to black-box LLMs, as the teacher's generative power may be more accessible. Feedback from teachers is typically given by ranking student-generated outputs and distilling preferences into the model through methods like Reinforcement Learning from A...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "where ydenotes the output generated by the student model in response to x, and ϕfb(·;θT))represents providing feedback from teacher LLMs. This operation evaluates the student’s output ygiven the input x, by offering assess- ment, corrective information, or other forms of guidance. This feedback knowledge can not only be distilled into the student to also generate feedback (such as creating a student preference model) but, more importantly, enable the student to refine its responses based on the ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "where \\( y \\) denotes the output generated by the student model in response to \\( x \\), and \\( \\phi_{fb}(\\cdot; \\theta_T) ) \\) represents providing feedback from teacher LLMs. This operation evaluates the student’s output \\( y \\), given input \\( x \\), by offering assessment, corrective information, or other forms of guidance. This feedback knowledge can not only be distilled into the student to also generate feedback (such as creating a preference model) but, more importantly, enable the student...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "teachers by prompting it with specific criteria. Bai et al. (2022a) in- troduce RLAIF for distilling harmlessness preferences from LLMs. This involves using an SFT-trained LLM to generate response pairs for each prompt, then ranking them for harmlessness to create a preference dataset. This dataset is distilled into a Preference Model (PM), which then guides the RL training of a more harmless LLM policy. Wizard- Math (Luo et al., 2023b) places emphasis on mathematical reasoning. They employ Chat...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "teachers by prompting it with specific criteria. Bai et al. (2022a) introduce RLAIF for distilling harmlessness preferences from LLMs. This involves using an SFT-trained LLL to generate response pairs for each prompt, then ranking them harmlessness create a preference dataset. This data is distilled into Preference Model (PM), which then guides the RL training of more harmless LLM policy. Wizard- Math employs ChatGPT as teacher to directly provide process supervision and evaluate correctness gen...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and helpfulness. Beyond merely assessing student generations, teachers can also furnish extensive feedback on instances where students underperform. In Lion (Jiang et al., 2023b), teacher model pinpoints instructions that pose challenges to the student model, generating new, more difficult instructions aimed at bolstering the student’s abilities. PERsD (Chen et al., 2023a) showcases a method where teacher offers tailored refinement feedback on incorrect code snippets gen- erated by students, gui...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "teachers can provide feedback on student generations beyond mere assessment. Teachers may offer detailed explanations, constructive criticism, and suggestions for improvement to enhance the student's understanding of the material. This feedback can be based on specific examples or patterns in students' work, helping to pinpoint areas where the student might struggle. Additionally, teachers can use various tools and techniques to provide personalized feedback on individual students' progress.\n",
      "\n",
      "##...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "wherein the student model initially generates sequences, followed by teacher model producing an output distribution as feedback. This method leverages the teacher’s insight to directly inform and refine the student model’s learning process. 3.1.6 Self-Knowledge The knowledge could also be elicited from the student itself, which we refer to as Self-Knowledge . In this setting, the same model acts both as the teacher and the student, iteratively improving itself by distilling and refining its own ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "wherein the student model initially generates sequences, followed by teacher model producing an output distribution as feedback. This method leverages the teacher’s insight to directly inform and refine the student model's learning process.\n",
      "\n",
      "3.1.6 Self-Knowledge\n",
      "\n",
      "The knowledge could also be elicited from the student itself, which we refer to as Self-Knowledge. In this setting, the same model acts both as the teacher and the student, iteratively improving itself by distilling and refining its own...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "which could include but is not limited to filtering, rewarding, or any other mechanisms for enhancing or evaluating y. It could be governed by external tools or the student itself θS. Recent research in this area has proposed various innovative methodologies to elicit self-knowledge, demonstrating its potential for creating more efficient and autonomous learn- ing systems. (Allen-Zhu and Li, 2020; Wang et al., 2022a; Sun et al., 2024b; Yang et al., 2024; Jung et al., 2023; Huang et al., 2023a; G...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "which could include but is not limited to filtering, rewarding, or any other mechanisms for enhancing or evaluating y. It could be governed by external tools or the student itself θS. Recent research in this area has proposed various innovative methodologies to elicit self-knowledge, demonstrating its potential for creating more efficient and autonomous learning systems. \n",
      "\n",
      "- Allen-Zhu and Li (2020)\n",
      "- Wang et al., 2022a\n",
      "- Sun et al., 2024b\n",
      "- Yang et al., 2024\n",
      "- Jung et al., 2023\n",
      "- Huang et al., 2...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "model. Other methods aim to elicit targeted knowledge 11 from student models by modifying prompts, and leveraging these data for further refinement. In Self-Align (Sun et al., 2024b), they find that models fine-tuned by Self-Instruct data tend to generate short or indirect responses. They prompt this model with verbose instruction to produce in- depth and detailed responses. Then, they employ context- distillation (Askell et al., 2021) to distill these responses paired with non-verbose instructi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "models. Other methods aim to elicit targeted knowledge from student models by modifying prompts and leveraging data for further refinement. In Self-Align (Sun et al., 2024b), models fine-tuned by self-instruct data tend to generate short or indirect responses. They prompt this model with verbose instructions to produce in-depth and detailed responses, then employ context-distillation (Askell et al., 2021) to distill these responses paired with non-verbal instructions back into the model. Similar...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "summarization tasks, implementing filters based on entailment, length, and diversity to screen self-generated summaries. LMSI (Huang et al., 2023a) generates multiple CoT reasoning paths and answers for each question, and then retains only those paths that lead to the most consistent answer. Note that refined self-knowledge can be iteratively ac- quired as the student model continuously improves, further enhancing the student’s capabilities. This is Gulcehre et al. (2023) introduces a Reinforced...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "summarization tasks, implementing filters based on entailment, length, and diversity to screen self-generated summaries. LMSI (Huang et al., 2023a) generates multiple CoT reasoning paths and answers for each question, then retains those leading to the most consistent answer. Refinement can iteratively be acquired as the student model continuously improves, further enhancing capabilities. Gulcehre et al. (2023) introduce a Reinforced Self-Training (ReST) framework that alternates between Grow and...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2024a) introduces a framework resembling iterative DPO, where the language model is fine-tuned to differentiate the self-generated responses from the human-annotated data. These self-generated responses could be seen as “negative knowledge” to promote the student to better align with the target distribution. Self-Rewarding (Yuan et al., 2024a) explores a novel and promising approach by utilizing the language model itself as a reward model. It employs LLM- as-a-Judge prompting to autonomously ass...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "2024a) introduces a framework resembling iterative DPO, where the language model is fine-tuned to differentiate self-generated responses from human-annotated data. These self-generated responses could be seen as \"negative knowledge\" to promote better alignment with the target distribution. Self-Rewarding (Yuan et al., 2024a) explores a novel and promising approach by utilizing the language model itself as a reward model. It employs LLM- as-a-Judge prompting to autonomously assign rewards for sel...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "and Rank Optimization , as shown in Figure 3. 3.2.1 Supervised Fine-Tuning Supervised Fine-Tuning (SFT), or called Sequence-Level KD (SeqKD) (Kim and Rush, 2016), is the simplest and one of the most effective methods for distilling powerful black-boxDivergence Type D(p, q)Function Forward KLDPp(t) logp(t) q(t) Reverse KLDPq(t) logq(t) p(t) JS Divergence1 2\u0010Pp(t) log2p(t) p(t)+q(t)+Pq(t) log2q(t) p(t)+q(t)\u0011 TABLE 1: Functional forms of Dfor various divergence types. p: reference Similarity Functi...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "and Rank Optimization, as shown in Figure 3. 3.2.1 Supervised Fine-Tuning (SFT), or called Sequence-Level KD (SeqKD) (Kim and Rush, 2016), is the simplest and one of the most effective methods for distilling powerful black-box models. SFT finetunes student model by maximizing the likelihood of sequences generated by teacher LLMs, aligning the student’s predictions with those of the teacher. This process can be mathematically represented as follows: where p(t) represents a similarity function (e....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "formulated as minimizing the objective function: LSFT=Ex∼X,y∼pT(y|x)[−logpS(y|x)], (9) where yis the output sequence produced by the teacher model. This simple yet highly effective technique forms the basis of numerous studies in the field. Numerous re- searchers have successfully employed SFT to train student models using sequences generated by teacher LLMs (Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Luo et al., 2023b). Additionally, SFT has been ex- plored in ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The objective function is minimized by the following equation:\n",
      "\n",
      "LSFT = -\\[\\log p_S(y|x)\\]\n",
      "\n",
      "where \\( y \\) represents the output sequence generated by the teacher model. This simple yet highly effective method forms a basis for numerous studies in various fields.\n",
      "\n",
      "Various researchers have employed SFT to train student models using sequences generated by teacher LLMs. For example, Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Luo et al., 2023b. Moreover, SFT has been ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "groups: those minimizing divergence in probability distributions and those aimed at enhancing the similarity of hidden states. Divergence. Divergence-based methods minimize diver- gence between the probability distributions of the teacher and student models, represented by a general divergence function D: LDiv= E x∼X,y∼Y[D(pT(y|x), pS(y|x))], (10) The specific form of Dvaries depending on the type of divergence employed. Table 1 outlines the functional forms ofDfor different divergence measures....\n",
      "\n",
      "PROCESSED TEXT:\n",
      "groups: those minimizing divergence in probability distributions and those aimed at enhancing the similarity of hidden states. Divergence.\n",
      "\n",
      "Divergence-based methods minimize divergence between the probability distributions of the teacher and student models, represented by a general divergence function \\( D \\):  \n",
      "\\[ E_{x\\sim X} y\\sim Y[D(p_T(y|x), p_S(y|x))] \\]  \n",
      "(10) The specific form of \\( D \\) varies depending on the type of divergence employed.  \n",
      "Table 1 outlines the functional forms of \\( D ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "predom- inantly on the most prominent mode, thereby exhibiting a “mode-seeking” behavior. Wen et al., 2023; Timiryasov and Tastet, 2023; Liang et al., 2023a; Chen et al., 2024d) , which forces pSto cover all the modes of pT. However, when a student model is unable to learn all modes of a highly complex teacher, the re- sultant “mode-covering” behavior might cause the student to assign probability mass to tokens with low probability under the teacher’s distribution (cf. Figure 6 blue curve). This...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "predominantly on the most prominent mode, thereby exhibiting a \"mode-seeking\" behavior. Wen et al., 2023; Timiryasov and Tastet, 2023; Liang et al., 2023a; Chen et al., 2024d) , which forces pSto cover all the modes of pT. However, when a student model is unable to learn all modes of a highly complex teacher, the resulting \"mode-covering\" behavior might cause the student to assign probability mass to tokens with low probability under the teacher's distribution (cf. Figure 6 blue curve). This mod...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "low-probability regions of the teacher’s distribution, employing Policy Gradient methods for optimization. Both Agarwal et al. (2024) and Sason and Verd ´u (2016) assess the effect of different divergence func- tions in LLM distillation, finding the optimal divergence to be task-dependent. For instance, forward KL divergence is more suitable for tasks like Machine Translation, where the output has fewer modes or variations, while reverse KL divergence is preferable for tasks like dialogue genera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "### Policy Gradient Methods for LLM Distillation\n",
      "\n",
      "Policy gradient methods are employed to optimize the distributional parameters of large language models (LLMs). Both Agarwal et al. (2024) and Sason & Verdú (2016) investigate the impact of different divergence functions in LLM distillation. The optimal divergence is found to be task-dependent, as forward KL divergence suits tasks like machine translation where the output has fewer modes or variations. Conversely, reverse KL divergence is more su...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "two models. The objective is to ensure that the student model not only produces similar outputs to the teacher but also processes information in a comparable manner. The formulation for a similarity-based objective might look like this: LSim= E x∼X,y∼Y[LF(ΦT(fT(x, y)),ΦS(fS(x, y)))],(11) where fT(x, y)andfS(x, y)are the feature maps of the teacher and student models, respectively. The transforma-tion functions ΦTandΦSare applied to these feature maps to ensure they are in the same shape, facilit...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The objective of this research project involves developing two models. The primary goal is to ensure that the student model produces outputs similar to those produced by a teacher model, while also ensuring comparable processing of information. To achieve this objective, we introduce the following formulation:\n",
      "\n",
      "\\[ L_{\\text{Sim}} = \\sum_x^X, y^Y [LF(\\Phi_T(f_{T}(x,y)), \\Phi_S(f_{S}(x, y)))], \\]\n",
      "\n",
      "where \\( f_T(x,y) \\) and \\( f_S(x,y) \\) represent the feature maps of the teacher model, while \\( \\Phi...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "discrepancy between the filtered representations in both teacher and student models. While similarity-based approaches are common in encoder-based LMs (Sun et al., 2019, 2020; Jiao et al., 2020; Hou et al., 2020; Zuo et al., 2022; Liang et al., 2021), their application in LLM knowledge distillation is not as widespread. However, considering their effectiveness, we anticipate an increase in research exploring these methods for LLM distillation in the near future. 3.2.3 Reinforcement Learning This...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**3.2.3 Reinforcement Learning**\n",
      "\n",
      "This section explores advanced methods of distilling knowledge into student models using reinforcement learning (RL). This approach is especially relevant for leveraging feedback from the teacher to train student models. \n",
      "\n",
      "**3.2.3 Reinforcement Learning**\n",
      "\n",
      "This section explores advanced methods of distilling knowledge into student models using reinforcement learning (RL). This approach is especially relevant for leveraging feedback from the teacher to train stud...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "involves training a reward model rϕusing the feedback data D(fd) generated by teacher LLMs. Preference data, as one of the typical feedback, is employed to train the student reward model (Bai et al., 2022a; Cui et al., 2023a; Lee et al., 2023a; Kim et al., 2023a). They usually consist of input-output pairs (x, yw, yl). Here, ywandylrepresent “winning” and “losing” outputs relative to the teacher’s preferences. The loss function for the reward model is defined as: LRM(rϕ,D(fd)) =− E (x,yw,yl)∼D(f...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Involves training a reward model \\( r_\\phi \\) using feedback data \\( D(fd) \\) generated by teacher LLMs. Preference data, as a typical feedback source for training the student reward model (Bai et al., 2022a; Cui et al., 2023a; Lee et al., 2023a; Kim et al., 2023a), consists of input-output pairs \\((x, y_w, y_l)\\). Here \\( y_w \\) and \\( y_l \\) represent \"winning\" and \"losing\" outputs relative to the teacher's preferences. The loss function for the reward model is defined as: \n",
      "\\[ L_{RM}(r_\\phi, D...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Learning Optimization. In the second stage, the student model, represented by a policy πθ, is optimized to maximize the expected reward as per the trained reward model. Simultaneously, it minimizes the divergence from a reference policy πref, typically the initial policy of the student model trained by SFT, controlled by a factor β. The RL objective is given by: 13 max πθE x∼X,y∼πθ(y|x)[rϕ(x, y)]−βDKL[πθ(y|x)∥πref(y|x)] (13) This RL framework not only ensures that the student model learns the ex...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Learning Optimization.\n",
      "\n",
      "In the second stage, the student model represented by a policy $\\pi\\_{θ}$ is optimized to maximize the expected reward as per the trained reward model. Simultaneously, it minimizes the divergence from a reference policy $\\pi\\_{ref}$, typically the initial policy of the student model trained by SFT. This is controlled by a factor $\\beta$. The RL objective is given by:\n",
      "\n",
      "$$\n",
      "13 \\max\\_{π\\_θ}E\\_{x, y}\\simX,y\\_\\{π\\_θ(y|x)\\}[rϕ(x, y)] - \\beta KLD[π\\_{θ}(y|x)∥ π\\_ref(y|x)]\n",
      "$$\n",
      "\n",
      "Thi...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "performance, it comes at a higher computational cost compared to employing a smaller distilled reward model. 3.2.4 Ranking Optimization Ranking optimization presents a stable and computationally efficient alternative to RL for injecting preference feedback into language models (Rafailov et al., 2023; Song et al., 2023a; Yuan et al., 2023b). This method, diverging from traditional RL approaches, directly incorporates ranking information into language models from a fixed preference dataset during ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Ranking optimization presents a stable and computationally efficient alternative to reinforcement learning (RL) for injecting preference feedback into language models. It diverges from traditional RL approaches and directly incorporates ranking information during fine-tuning, making policy updates more efficient. This method stabilizes the process without requiring sampling outputs.\n",
      "\n",
      "Recent works have explored using ranking optimization to distill teacher's preferences into student models. Zephy...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Preference Optimization (DPO) (Rafailov et al., 2023) to distill the preference alignment in teacher LLMs. DPO streamlines the objective of reinforcement learning (as in Eq. 13), which involves reward maximization with a KL-divergence constraint, into a single-stage policy training. Specifically, DPO’s training goal is to maximize the following expecta- tion: E (x,yw,yl)∼D(fd)\u0014 logσ\u0012 βlogπθ(yw|x) πref(yw|x)−βlogπθ(yl|x) πref(yl|x)\u0013\u0015 , (14) where ywis preferred over ylaccording to the teacher LLM...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Preference Optimization (DPO) streamlines the preference alignment in teacher LLMs. DPO's training goal is to maximize an expectation, defined by Eq. 14: E (x,yw,yl)∼D(fd)\u0014 logσ\u0012 βlogπθ(yw|x) πref(yw|x)−βlogπθ(yl|x) πref(yl|x)\u0013\u0015 . Here, yw is preferred over yl according to the teacher LLM. Hong et al.'s (2023) preference distillation methods use two ranking-based objectives: Rank Responses to align Human Feedback (RRHF) and Preference Ranking Optimization (PRO). RRHF focuses on a ranking loss de...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "probabilities under the policy πθ. This approach emphasizes direct comparison and ranking of responses based on the teacher’s preferences. PRO (Song et al., 2023a) expands the concept of pairwisecomparison to handle preference rankings of any length. For a given instruction xand a sequence of responses ordered by teacher preference as y1≻y2≻...≻yn, the RPO training objective is: LPRO=−n−1X k=1logexp (pk)Pn i=kexp (pi), (16) where pkrepresents the conditional log probabilities for ykunder the stu...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "In this context, the policy $\\pi_{\\theta}$ is used to evaluate responses. This method focuses on direct comparisons and rankings based on the teacher's preferences, emphasizing their impact.\n",
      "\n",
      "PRO (Song et al., 2023a) extends the concept of pairwise comparisons to handle preference rankings for any length. For an instruction $x$ and a sequence of responses ordered by teacher preference as $y_1 \\succ y_2 \\succ ... \\succ y_n$, the RPO training objective is:\n",
      "\n",
      "\\[\n",
      "L_{PRO} = -n \\sum_{k=1}^{n-1}\\log\\lef...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "range of skills exhibited by LLMs, including Context Following ,Alignment ,Agent ,NLP Task Specializa- tion and Multi-Modality .Context Following focuses on the student’s ability to comprehend and respond effectively to input information. Alignment delves into the student’s capability to align its output with the teacher’s responses. Moving forward, Agent underscores the autonomous nature of language models. NLP Task Specialization highlights the LLM’s versatility in specializing across various ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The raw data is messed up with new lines, Latex math and fluff that can be removed. Here's the cleaned-up text:\n",
      "\n",
      "\"Range of skills exhibited by LLMs, including Context Following, Alignment, Agent, NLP Task Specialization and Multi-Modality. **Context Following** focuses on the student’s ability to comprehend and respond effectively to input information. **Alignment** delves into the student's capability to align its output with the teacher’s responses. Moving forward, **Agent** underscores the au...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "transferring the ability of LLMs to handle a variety of complex contexts — such as few-shot demonstrations, intricate instructions, dia- logue history, and retrieval-augmented information — into smaller models. Many research efforts in this domain aim to imbue smaller models with these sophisticated, context- following capabilities. Our discussion here will dissect this facet of skill distillation, categorizing it based on different types of context and elaborating on how each is distilled and i...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Data acquisition, as a crucial aspect of skill distillation in LLMs, is essential to ensure the quality and relevance of instructions. In this context, researchers often use techniques such as **Data Augmentation** to enrich the dataset by creating diverse and varied examples. This process can be achieved through **In-Domain** data, where existing training datasets are augmented to include more complex scenarios or less common examples. **Out-of-Domain** data, on the other hand, involves collect...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "can be manually curated by human experts or transformed from existing NLP tasks into instructional 14 Methods Skill Seed Knowledge Teacher LLM Student Model Knowledge Elicitation Objective Context Following Self-Instruct (Wang et al., 2022a) IF 175 human-curated tasks GPT3 GPT3 Expansion + Self-Knowledge SFT Alpaca (Taori et al., 2023) IF 175 human-curated tasks GPT3 LLaMA Expansion + Self-Knowledge SFT LaMini-LM (Wu et al., 2023c) IF3.5K Wikipedia Categories + Mixed DatasetChatGPT Various Model...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**Methods for Skill Enhancement**\n",
      "\n",
      "- **Teacher-Learner Model Pair**: A teacher model and a student model are used to guide the learning process. The **Teacher** provides guidance, while the **Student** acts as a learner.\n",
      "- **Knowledge Elicitation**: The process involves gathering knowledge from various sources, often through human-curated methods or the use of models like **GPT3**, **LLaMA** expansion, and other tools for data-driven learning.\n",
      "- **Objective Context**: The context is crucial as i...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Self-Rewarding (Yuan et al., 2024a) IF Human-written Samples LLaMA LLaMA Self-Knowledge SFT + RL STaR (Zelikman et al., 2022) IF Arithmetic + CommonsenseQA + GSM8K GPT-J GPT-J Self-Knowledge SFT Llama-GPT4 (Peng et al., 2023a) IF Alpaca Dataset GPT4 LLaMA Labeling SFT Reflection-Tuning (Li et al., 2023e) IF Alpaca/WizardLM Dataset ChatGPT LLaMA Labeling SFT Selective Reflection-Tuning (Li et al., 2024d) IF Alpaca/WizardLM Dataset ChatGPT LLaMA Labeling SFT Vicuna (Chiang et al., 2023) IF/MD Huma...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "IF Human-written Samples + LLaMA Self-Knowledge SFT, IF Arithmetic + CommonsenseQA + GSM8K GPT-J, LLaMA Self-Knowledge SFT, IF Alpaca Dataset + GPT4 LLaMA Labeling SFT Reflection-Tuning (Li et al., 2023e), IF Alpaca/WizardLM Dataset + ChatGPT LLaMA Labeling SFT Selective Reflection-Tuning (Li et al., 2024d), IF Alpaca/WizardLM Dataset + ChatGPT LLaMA Labeling SFT Vicuna (Chiang et al., 2023), IF/MD Human Conversation + ChatGPT LLaMA Labeling SFT Koala (Geng et al., 2023), IF/MD Human Conversatio...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "et al., 2023) IF/TP Human Conv, Flan/Code/Math Collection ChatGPT LLaMA Labeling SFT CoT-Distill (Hsieh et al., 2023) IF/TP e-SNLI + ANLI + CQA + SVAMP PaLM T5 Labeling SFT KnowPAT (Zhang et al., 2023a) IF/TP CPKG + QA Data ChatGPT + ChatGLM + Vicuna-7B LLaMA Labeling SFT DEBATunE (Li et al., 2024e) IF/TP Controversial Topics ChatGPT LLaMA Labeling SFT Phi-1 (Gunasekar et al., 2023) IF/Code - GPT3.5 phi-1 Curation SFT Phi-1.5 (Li et al., 2023a) IF/Code 20k Topics from Web GPT3.5 phi-1 Curation +...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The data provided is a list of research papers and their corresponding authors. Here's the cleaned-up version:\n",
      "\n",
      "- IF/TP Human Conv, Flan/Code/Math Collection ChatGPT LLaMA Labeling SFT CoT-Distill (Hsieh et al., 2023)\n",
      "- IF/TP e-SNLI + ANLI + CQA + SVAMP PaLM T5 Labeling SFT KnowPAT (Zhang et al., 2023a)\n",
      "- IF/TP CPKG + QA Data ChatGPT + ChatGLM + Vicuna-7B LLaMA Labeling SFT DEBATunE (Li et al., 2024e)\n",
      "- IF/TP Controversial Topics ChatGPT LLaMA Labeling SFT Phi-1 (Gunasekar et al., 2023)\n",
      "- IF/Cod...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Human-written Prompts LLaMA LLaMA Expansion + Labeling SFT + RL RLCD (Yang et al., 2024) IF/Preference Human-written Prompts LLaMA LLaMA Labeling SFT + RL RLAIF (Lee et al., 2023a) IF/Preference Human-written Prompts PaLM 2 PaLM 2 Labeling + Feedback RL GPT3 Reward (Kwon et al., 2023) Preference Human-written Prompts GPT3 GPT3 Labeling RL ILF (Scheurer et al., 2023) Preference Task-specific Datasets GPT3 + FeedME GPT3 Labeling RL ULTRAFEEDBACK (Cui et al., 2023a) Preference Mixed Datasets GPT4 L...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Human-written prompts for LLaMA, expansion with labeling by SFT and RL. IF/Preference: Human-written prompts for LLaMA, expansion with labeling by SFT and RL.\n",
      "\n",
      "Human-written prompts using PaLM 2, PLaM 2 labeling + feedback for RL. IF/Preference: Human-written prompts using PaLM 2, PLaM 2 labeling + feedback for RL.\n",
      "\n",
      "Human-written prompts with GPT3 reward by Kwon et al. IF/Preference: Human-written prompts for GPT3, labeling + RL.\n",
      "\n",
      "Human-defined student model and Self-defined Model with expansion...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "API Documentation GPT4 LLaMA Expansion SFT GPT4Tools (Yang et al., 2023b) Tool Image Content ChatGPT LLaMA Curation + Expansion SFT ToolAlpaca (Tang et al., 2023a) Tool Public-apis Repository ChatGPT LLaMA Curation SFT ToolLLM (Qin et al., 2023a) Tool Real-world APIs ChatGPT LLaMA Curation SFT MLLM-Tool (Wang et al., 2024) Tool HuggingFace Model Cards GPT4 LLaMA Curation SFT FireAct (Chen et al., 2023b) Planning Mixed QA Dataset GPT4 LLaMA Labeling SFT AgentTuning (Zeng et al., 2023a) Planning 6...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "### API Documentation\n",
      "\n",
      "- **GPT4**: Enhanced by LLaMA Expansion SFT, Tool Alpaca\n",
      "- **LLaMA** Curation + Expansion SFT\n",
      "\n",
      "### Tools and Models Used in Research Papers\n",
      "\n",
      "- **GPT4**: Expanded by LLaMA Expansion SFT, Tool Alpaca\n",
      "  - **Content ChatGPT**: Curation + Expansion SFT\n",
      "\n",
      "- **LLaMA** LLaMA Curation + Expansion SFT\n",
      "  - **Content ChatGPT**: Expanded by LLaMA Expansion SFT\n",
      "\n",
      "- **LLaMA** Curation + Expansion SFT\n",
      "  - **Content ChatGPT**: Expanded by LLaMA Expansion SFT\n",
      "\n",
      "- **LLaMA** Curation + Expansion...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2021a) NLU NLU Tasks GPT3 BERT Expansion SFT InheritSumm (Xu et al., 2023c) NLG Pile + ArXiv + CNN/DM + WikiHow GPT3.5 ZCode++ Label SFT DIMSUM+ (Jung et al., 2023) NLG None GPT2 + CTRL + BioGPT T5 Curation + Self-Knowledge SFT Genie (Yehudai et al., 2024) NLG ELI5 + ASQA + NQ + CNN/DM Falcon + LLaMA FLAN + LLaMA Label SFT GKD (Agarwal et al., 2024) NLG/NLU/IF XSum+WMT14 en-de+GSM8K+FLAN2021 T5-XL T5 Feature + Feedback D&S + RL QUILL (Srinivasan et al., 2022) IR IR Datasets T5 4-layer Trans...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The NLU tasks that involve GPT3, BERT expansion and Self-Knowledge SFT include InheritSumm. The NLG tasks that involve Pile, ArXiv, CNN/DM and WikiHow GPT3.5 ZCode++ SFT include DIMSUM+. The NLG tasks that involve ELI5, ASQA, NQ and CNN/DM Falcon + LLaMA FLAN + LLaMA Label SFT include GKD. The IR tasks that involve T5 4-layer Transformer Internal Knowledge D&S are ranked by Vicuna. The recommendation tasks that involve GPT3 MPnet-110M Labeling SFT are ranked by InstrcutRec....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "PandaLM (Wang et al., 2023b) Evaluation Alpaca Data ChatGPT LLaMA Labeling SFT Prometheus (Kim et al., 2024) Evaluation 50 Seed Rubrics GPT4 LLaMA Labeling SFT InstructScore (Xu et al., 2023d) Evaluation Mixed Dataset GPT4 LLaMA Labeling SFT WizardMath (Luo et al., 2023b) Math GSM8k + MATH ChatGPT LLaMA Expansion + Feedback SFT + RL Mammoth (Yue et al., 2023a) Math/TP Mixed Math Dataset GPT4 LLaMA Labeling SFT Mixed Distill (Chenglin et al., 2023) Math/TP SVAMP + GSM8K + ASDIV + StrategyQA ChatG...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "PandaLM evaluation: Wang et al., 2023b  \n",
      "Alpaca, Data ChatGPT LLaMA, Labeling SFT Prometheus (Kim et al., 2024)  \n",
      "Evaluation: GPT4 LLaMA, Labeling SFT InstructScore (Xu et al., 2023d)  \n",
      "Mixed Dataset GPT4 LLaMA, Labeling SFT WizardMath (Luo et al., 2023b)  \n",
      "SVMG8k + MATH, ChatGPT LLaMA Expansion + Feedback SFT  \n",
      "Mammoth (Yue et al., 2023a) Math/TP Mixed Dataset GPT4 LLaMA, Labeling SFT  \n",
      "Mixed Distill (Chenglin et al., 2023) Math/TP SVAMP + GSM8K  \n",
      "ASDIV + StrategyQA, ChatGPT LLaMA Labeling SFT ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "al., 2023) Code Code Datasets ChatGPT LLaMA Labeling SFT Multi-Modality LLaVA (Liu et al., 2023e) Vision-Language COCO GPT4 LLaMA Labeling SFT SVIT (Zhao et al., 2023b) Vision-Language Visual Genome + COCO GPT4 LLaMA Labeling SFT LVIS-Instruct4V (Wang et al., 2023e) Vision-Language LVIS GPT4V LLaMA Labeling SFT LLaVAR (Zhang et al., 2023d) Vision-Language LAION GPT4 LLaMA Labeling SFT Macaw-LLM (Lyu et al., 2023) Multiple Modalities Image/Video with Caption ChatGPT LLaMA Labeling SFT MIMIC-IT (L...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The text has been cleaned and formatted for easier use in a podcast. Here is the processed version:\n",
      "\n",
      "al., 2023) Code Code Datasets ChatGPT LLaMA Labeling SFT Multi-Modality LLAVA (Liu et al., 2023e) Vision-Language COCO GPT4 LLaMA Labeling SFT SVIT (Zhao et al., 2023b) Vision-Language Visual Genome + COCO GPT4 LLaMA Labeling SFT LVIS-Instruct4V (Wang et al., 2023e) Vision-Language LVIS GPT4V LLaMA Labeling SFT LLAVAR (Zhang et al., 2023d) Vision-Language LAION GPT4 LLaMA Labeling SFT Macaw-LLM (...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Divergence and Similarity, RL: Reinforcement Learning, RO: Ranking Optimization. formats with templates, such as prefacing machine transla- tion data with ”Translate this sentence to Spanish:” . However, these approaches have limitations. Manual data creation is labor-intensive, while template-based transformation lacks diversity in instructions and may not align well with natural human input. LLMs like GPT-4 offer an efficient alternative for creating diverse and controlled SFT data by their ca...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**Divergence and Similarity, RL: Reinforcement Learning**\n",
      "\n",
      "- **Reinforcement Learning (RL):** \n",
      "  - *Example:* Manual data creation is labor-intensive, while template-based transformation lacks diversity in instructions and may not align well with natural human input.\n",
      "  - *Example:* LLMs like GPT-4 offer an efficient alternative for creating diverse and controlled SFT data by their ability to in-context learn.\n",
      "- **Template-based Transformation:**\n",
      "  - *Example:* Formats with templates, such as pre...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "GPT-3 to expand 15 a seed pool of 175 tasks to 52K task-agnostic instructions, ensuring a broad spectrum of general instructions. Addi- tionally, a filtering and post-processing stage is introduced to eliminate redundant or similar instructions. Notably, through training with this enriched dataset, GPT-3 acquires the ability to follow instructions, enabling it to perform comparably to InstructGPT in zero-shot instruction tasks and when provided with expert-written instructions for novel tasks. B...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "To expand a seed pool of 175 tasks to 52K task-agnostic instructions, ensuring a broad spectrum of general instructions. A filtering and post-processing stage is introduced to eliminate redundant or similar instructions, enhancing the model's adaptability. Through training with this enriched dataset, GPT-3 acquires the ability to follow instructions and perform comparably in zero-shot instruction tasks, when provided with expert-written instructions for novel tasks. The self-instruct method by T...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Complex Instructions. Some works promote students to solve more complex instructions (Xu et al., 2023a; Luo et al., 2023b,a; Guo et al., 2023c). According to Xu et al. (2023a), in- struction datasets derived from human-written seeds often exhibit low to moderate complexity. To enhance the com- plex instruction-following capabilities of smaller models, WizardLM (Xu et al., 2023a) introduces Evol-Instruct . This method gradually transforms instructions into more com- plex forms through a multi-ste...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**Complex Instructions**\n",
      "\n",
      "Some works promote students to solve more complex instructions. According to Xu et al., (2023a), instruction datasets derived from human-written seeds often exhibit low to moderate complexity. To enhance the complex instruction-following capabilities of smaller models, WizardLM (Xu et al., 2023a) introduces Evol-Instruct. This method gradually transforms instructions into more complex forms through a multi-step evolution process, focusing on both increasing difficulty l...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "preliminary studies revealing the effectiveness of increasing instruction complexity. Instruction Fusion (Guo et al., 2023c) further uses teacher LLMs to increase the complexity by fusing two distinct evolved instructions. Furthermore, this concept of “evolving” instructions has been extended to distill specific skills such as coding (Luo et al., 2023a) and mathematics (Luo et al., 2023b). Human Instructions. In contrast to works that rely on gener- ating instructions from ChatGPT, which may lac...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "In preliminary studies, it is revealed that increasing instruction complexity leads to better outcomes. Instruction Fusion (Guo et al., 2023c) enhances instruction complexity by fusing two distinct evolved instructions. This concept of \"evolving\" instructions has also been applied to distill specific skills like coding (Luo et al., 2023a) and mathematics (Luo et al., 2023b). Human instructions, in contrast to those generated by ChatGPT which may lack diversity and have gaps with real human instr...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "capture the reasoning process of the original teacher (Gudibande et al., 2023; Mukherjee et al., 2023). System Instructions. To encourage student models to learn the reasoning process, Orca and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) enhance the prompt, response data pairs by introducing a system message (e.g., ”explain like I’m five, think step-by-step”) to encourage student mod- els to grasp the reasoning process. This system messageprompts GPT-4 to provide explanation traces that ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "System Instructions: To encourage student models to learn the reasoning process, Orca and Orca 2 (Mukherjee et al., 2023; Mitra et al., 2023) enhance the prompt, response data pairs by introducing a system message (e.g., \"explain like I'm five\", \"think step-by-step\") to encourage student models to grasp the reasoning process. This system message prompts GPT-4 to provide explanation traces that elucidate the teacher's reasoning process. Orca 2 (Mitra et al., 2023) further trains the student model...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "2023b) distills large-scale data with high-quality and di- verse instructions from teacher LLMs by various meta- information. The UltraLLaMA model, fine-tuned on this data, consistently surpasses other open-source models. The Phi series models (Gunasekar et al., 2023; Li et al., 2023a; Mar, 2023) prioritize data quality and employ synthetic methods to generate data of “textbook quality” to enhance the learning experience for smaller models. Notably, Phi exhibits the ability to follow instruction...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "distilled data, generating higher-quality prompts for fine-tuning. This approach not only enhances the learning experience but also adds a layer of sophistication to the instruction data, leading to better model performance....\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "the quality of responses. ExpertLLaMA (Xu et al., 2023f) improves the quality of responses by augment- ing vanilla instructions with specialized Expert Identity descriptions. Reflection-Tuning (Li et al., 2023e) improves both the instruction and response sequentially by reflecting on specific criteria. DEITA (Liu et al., 2023h) proposes to enhance and score instructions in three directions includ- ing complexity, quality, and diversity to get high-quality distillation data. MUFFIN (Lou et al., 2...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "The quality of responses is improved by ExpertLLaMA (Xu et al., 2023f), which augments vanilla instructions with Expert Identity descriptions. Reflection-Tuning (Li et al., 2023e) sequentially refines both instructions and responses by reflecting on specific criteria. DEITA (Liu et al., 2023h) enhances instructions in three directions: complexity, quality, and diversity to generate high-quality distillation data. MUFFIN (Lou et al., 2023) scales instructions based on input facets, diversifying t...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "Cur- rent small models have made strides in enhancing var- ious aspects of instruction-following ability, like diver- sity, complexity and explanation. However, student mod- els trained on instruction data expanded by ChatGPT of- ten mimic ChatGPT’s style without replicating its factual accuracy (Gudibande et al., 2023). Achieving a more ca- pable instruction-following capability requires a stronger teacher LLM (Gudibande et al., 2023) and access to di- verse, high-quality instruction data, such...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Current small models have made strides in enhancing various aspects of instruction-following ability, such as diversity and complexity. However, student models trained on data expanded by ChatGPT often mimic its style without replicating its factual accuracy (Gudibande et al., 2023). Achieving a more capable instruction-following capability requires stronger teacher LLMs and access to diverse, high-quality instruction data. For example, the Orca model uses extensive task instructions from a coll...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "dialogue turns. Some works have been dedicated to train to small chat models by distilling multi-turn knowl- edge from teacher LLMs (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023). ShareGPT serves as a platform for users to share their conversations with ChatGPT, offering a vast repository of multi-turn conversations readily available. Some small chat models are trained using this data to acquire the capability for engaging...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Some works have been dedicated to training small chat models by distilling multi-turn knowledge from teacher LLMs (Chiang et al., 2023; Xu et al., 2023b; Ding et al., 2023b; Li et al., 2023b; Wang et al., 2023c; Tunstall et al., 2023). ShareGPT serves as a platform for users to share their conversations with ChatGPT, offering a vast repository of multi-turn conversations readily available. Some small chat models are trained using this data to acquire the capability for engaging in multi-turn dia...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "aiming to incentivize student models to produce high-quality responses. Addi- tionally, Ye et al. (2023) enhance the quality of multi-turn data from ShareGPT by generating self-feedback on model responses and iteratively refining the responses based on the received feedback. To enhance the multi-turn capabilities of student models, another line of research focuses on expanding conversa- tional datasets through self-chat and using them to train smaller models (Xu et al., 2023b; Ding et al., 2023b...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "aiming to incentivize student models to produce high-quality responses. Additional research by Ye et al. (2023) improves multi-turn data quality from ShareGPT through self-feedback on model responses, iteratively refining the responses based on received feedback. To enhance multi-turn capabilities in student models, another line of research focuses on expanding conversational datasets through self-chat. For example, Xu et al. (2023b) use questions from platforms like Quora and Stack Overflow as ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "dialogues from ChatGPT. Notably, UltraChat encom- passes a wide range of topics and instructions. Building upon the UltraChat dataset, they fine-tune a LLaMA model, resulting in the creation of a powerful chat model known as UltraLLaMA. UltraLLaMA consistently outperforms other open-source chat models, including Vicuna and Baize. Fur- thermore, UltraChat is employed in conjunction with an AI preference-aligned chat model named Zephyr (Tunstall et al., 2023). Zephyr enhances intent alignment thro...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "dialogues from ChatGPT. Notably, UltraChat encom- passes a wide range of topics and instructions. Building upon the UltraChat dataset, they fine-tune a LLaMA model, resulting in the creation of a powerful chat model known as UltraLLaMA. UltraLLaMA consistently outperforms other open-source chat models, including Vicuna and Baize. Furthermore, UltraChat is employed in conjunction with an AI preference-aligned chat model named Zephyr (Tunstall et al., 2023). Zephyr enhances intent alignment throug...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "of retrieved information is also a non- trivial skill of LLMs. Several approaches to distill RAG capabilities have been proposed (Kang et al., 2023a; Luo et al., 2023c; Asai et al., 2023). SAIL (Luo et al., 2023c) starts by retrieving search results for each training case using search APIs, creating search- augmented instructions that include both the instruction and grounding information. To encourage the language model to prioritize informative retrieval results, they input each retrieved pass...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "Of retrieved information is also a non-trivial skill of LLMs. Several approaches to distill RAG capabilities have been proposed (Kang et al., 2023a; Luo et al., 2023c; Asai et al., 2023). SAIL (Luo et al., 2023c) starts by retrieving search results for each training case using search APIs, creating search-augmented instructions that include both the instruction and grounding information. To encourage the language model to prioritize informative retrieval results, they input each retrieved passag...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "rationales are then utilized to train two models: a student LM and a Reranker. For training the student LM, the rationales serve as a means to retrieve relevant knowledge d, and the student LM is subsequently fine-tuned using the rationales along- side questions and knowledge. However, during inference, only questions are available. To address this, the Reranker is trained to mimic how the retriever scores passages with the rationale by minimizing the KL divergence between Retriever (d|r)andRera...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "rationales are then utilized to train two models: a student LM and a Reranker. For training the student LM, the rationales serve as a means to retrieve relevant knowledge \\(d\\), and the student LM is subsequently fine-tuned using the rationales along- side questions \\(q\\) and knowledge. However, during inference, only questions are available to the system.\n",
      "\n",
      "To address this issue, a Reranker is trained by mimicking how a retriever scores passages with the rationale \\(d\\), resulting in minimizing ...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "‘reflection to- kens.’ For instance, Self-Rag initiates the retrieval operation when generating the reflection token Retrieve . To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations I, the task input x, and output yto predict a reflection token ras follows: p(r|I, x, y ). 4.2 Alignment 4.2.1 Thinking Pattern Most existing methods mainly focus on directly aligning the direct responses of the student models to the responses of teacher models ...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "'reflection to- kens.' For instance, Self-Rag initiates the retrieval operation when generating the reflection token `Retrieve`. To distill this critic data, GPT-4 is prompted to assess the need for retrieval using few-shot demonstrations `I`, the task input `x`, and output `y` to predict a reflection token `ras follows:` p(r|I, x, y). 4.2 Alignment\n",
      "\n",
      "**4.2.1 Thinking Pattern**\n",
      "\n",
      "Most existing methods mainly focus on directly aligning the direct responses of student models to those from teacher mo...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "by the effectiveness of LLMs in generat- ing their own feedback without relying on external mod- els (Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022), SelFee (Ye et al., 2023) proposes to train a 17 model that has been fine-tuned to continuously revise its own answer until it provides a high-quality response in a single inference. During training, it utilizes both the final response and feedback chain as the fitting target. This pat- tern, response with the revision process, sho...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**By the effectiveness of LLMs in generating their own feedback without relying on external models (Schick et al., 2022; Madaan et al., 2023; Saunders et al., 2022), SelFee (Ye et al., 2023) proposes to train a model that has been fine-tuned to continuously revise its own answer until it provides a high-quality response in a single inference. During training, the model utilizes both the final response and feedback chain as fitting targets. This pattern shows a promising performance gain. Followi...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to equip the student models with the ability to utilize different solution strategies for different tasks, mo- tivated by the capability discrepancies between the smaller and larger models. By employing this training pattern, the student models are able to gain a better reasoning ability. Be- sides learning with the corresponding revision or reflection process,...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "experiments verify the effectiveness of distilling with this thinking pattern. The following Orca2 (Mitra et al., 2023) further presents to equip the student models with the ability to utilize different solution strategies for different tasks, motivated by the capability discrepancies between the smaller and larger models. By employing this training pattern, the student models are able to gain a better reasoning ability.\n",
      "\n",
      "Besides learning with the corresponding revision or reflection process, an...\n",
      "==========================================================================================\n",
      "\n",
      "INPUT TEXT:\n",
      "distilled into the student models. 4.2.2 Preference The previously mentioned methods primarily focus on the basic capability of student models to produce outcomes that are strictly accurate but may not align with human preferences, reaching alignment at this level enables these models to aid in various tasks without meeting higher-level demands. Early methods mainly utilize human feed...\n",
      "\n",
      "PROCESSED TEXT:\n",
      "**Student Models:**  \n",
      "The student models are designed to generate outcomes that align with human preferences, despite their accuracy. This feature enables the student models to assist in various tasks without meeting higher-level demands.\n",
      "\n",
      "**Preference:**  \n",
      "The methods discussed primarily focus on the basic accuracy of student models, which may not align with human preferences. Achieving alignment at this level allows the student models to assist in tasks without exceeding higher-level demands.\n",
      "...\n",
      "==========================================================================================\n",
      "\n"
     ]
    }
   ],
   "execution_count": 15
  },
  {
   "cell_type": "markdown",
   "id": "31cffe8d",
   "metadata": {},
   "source": [
    "Let's print out the final processed versions to make sure things look good"
   ]
  },
  {
   "cell_type": "code",
   "id": "89ef51a7-f13f-49a4-8f73-9ac8ce75319d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:20:12.659779Z",
     "start_time": "2024-10-29T16:20:12.657371Z"
    }
   },
   "source": [
    "print(f\"\\nProcessing complete!\")\n",
    "print(f\"Input file: {INPUT_FILE}\")\n",
    "print(f\"Output file: {output_file}\")\n",
    "print(f\"Total chunks processed: {num_chunks}\")\n",
    "\n",
    "# Preview the beginning and end of the complete processed text\n",
    "print(\"\\nPreview of final processed text:\")\n",
    "print(\"\\nBEGINNING:\")\n",
    "print(processed_text[:1000])\n",
    "print(\"\\n...\\n\\nEND:\")\n",
    "print(processed_text[-1000:])"
   ],
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Processing complete!\n",
      "Input file: ./resources/extracted_text.txt\n",
      "Output file: ./resources/clean_extracted_text.txt\n",
      "Total chunks processed: 101\n",
      "\n",
      "Preview of final processed text:\n",
      "\n",
      "BEGINNING:\n",
      "This is a survey on Knowledge Distillation (KD) within Large Language Models. It highlights the pivotal role of KD in transferring capabilities from leading proprietary LLMs to open-source models, such as GPT-4. Additionally, it underscores the crucial role of KD in compressing open-source LLMs and facilitating their self-improvement. The survey is conducted by Xiaohan Xu, Ming Li, Chongyang Tao from the University of Hong Kong and Microsoft respectively. It is titled \"Knowledge Distillation in Large Language Models\" by the authors at <https://shawnxxh.github.io/knowledge_distillation.html> and is published under a Creative Commons Attribution-ShareAlike 4.0 International License.\n",
      "advanced knowledge to smaller models and its utility in model compression and self-improvement. Our survey is meticulously structured around three foundational pillars: algorithm, skill, and verticalization – providing a comprehensive examination of knowledge distillation (KD) mechanisms. The survey highlight\n",
      "\n",
      "...\n",
      "\n",
      "END:\n",
      "e obtained.\n",
      "\n",
      "In summary:\n",
      "- **Distillation**: Enhances reasoning by transferring knowledge.\n",
      "- **Generating Responses and Preferences** (Zhang et al., 2023a): Combines knowledge with preferences.\n",
      "- **Controversial Topic Debate** (Li et al., 2024e): Improves LLM controllability through debates.\n",
      "\n",
      "These methods contribute to enhancing AI systems' adaptability and reasoning capabilities in diverse contexts.\n",
      "**Student Models:**  \n",
      "The student models are designed to generate outcomes that align with human preferences, despite their accuracy. This feature enables the student models to assist in various tasks without meeting higher-level demands.\n",
      "\n",
      "**Preference:**  \n",
      "The methods discussed primarily focus on the basic accuracy of student models, which may not align with human preferences. Achieving alignment at this level allows the student models to assist in tasks without exceeding higher-level demands.\n",
      "\n",
      "**Human Feed:**  \n",
      "Early methods primarily rely on human feedback to train the student models.\n",
      "\n"
     ]
    }
   ],
   "execution_count": 16
  },
  {
   "cell_type": "markdown",
   "id": "3d996ac5",
   "metadata": {},
   "source": [
    "### Next Notebook: Transcript Writer\n",
    "\n",
    "Now that we have the pre-processed text ready, we can move to converting into a transcript in the next notebook"
   ]
  },
  {
   "cell_type": "code",
   "id": "1b16ae0e-04cf-4eb9-a369-dee1728b89ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-10-29T16:20:12.667134Z",
     "start_time": "2024-10-29T16:20:12.665655Z"
    }
   },
   "source": [
    "#fin"
   ],
   "outputs": [],
   "execution_count": 17
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}