polyglot-KULLM_inference.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### https://github.com/nlpai-lab/KULLM\n",
    "merge_lora.py 참고함."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "finetune_kullm_polyglot.py을 통해 생성된 adapter_model.bin 파일을 불러와서 모델 답변 확인해보기 <br>\n",
    "\n",
    "LoRA의 방식으로 파인튜닝하면 사전 학습 모델의 파라미터는 고정되기 때문에 사전학습 모델의 파라미터와 파인튜닝해서 얻은 adapter_model.bin 파라미터를 합치는 과정이 필요하다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from peft import PeftModel\n",
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "import pandas as pd\n",
    "from peft import (\n",
    "    LoraConfig,\n",
    "    get_peft_model,\n",
    "    get_peft_model_state_dict,\n",
    "    prepare_model_for_int8_training,\n",
    "    set_peft_model_state_dict,\n",
    "    prepare_model_for_kbit_training\n",
    ")\n",
    "from transformers import GPTNeoXForCausalLM, GPTNeoXTokenizerFast\n",
    "import sys\n",
    "\n",
    "\n",
    "def answer_chat(input,model,tokenizer,device=1):\n",
    "    inputs = tokenizer(input, return_tensors=\"pt\", return_token_type_ids=False).to(device)\n",
    "\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        print(tokenizer.decode(model.generate(**inputs, max_new_tokens=300, pad_token_id=2)[0], skip_special_tokens=True)) # max_new_tokens 너무 크면 같은 단어의 반복\n",
    "        \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "데이터 컬럼 : ['instruction', 'input', 'output']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Found cached dataset json (/home/brianjang7/.cache/huggingface/datasets/json/default-a76f94fd1b103ee7/0.0.0/8bb11242116d547c741b2e8a1f18598ffdd40a1d4f2a2872c7a28b697434bc96)\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dc663b0cab1a4583baf8477cb24a58c9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    train: Dataset({\n",
       "        features: ['instruction', 'input', 'output'],\n",
       "        num_rows: 90309\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "data_path = 'data.jsonl'\n",
    "data = load_dataset(\"json\", data_files=data_path)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## merge 방법 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "사용량이 적은 GPU 자동할당"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "device: 0\n"
     ]
    }
   ],
   "source": [
    "# Set the available GPU devices\n",
    "# available_gpus = [0]  # 사용 가능한 GPU가 1개일 경우\n",
    "available_gpus = [0, 1]  # 사용 가능한 GPU가 2개일 경우\n",
    "\n",
    "# Function to find the GPU with the least memory usage\n",
    "def get_least_memory_gpu():\n",
    "    least_memory = float('inf')\n",
    "    least_memory_gpu = None\n",
    "    for gpu in available_gpus:\n",
    "        allocated_memory = torch.cuda.memory_allocated(gpu)\n",
    "        if allocated_memory < least_memory:\n",
    "            least_memory = allocated_memory\n",
    "            least_memory_gpu = gpu\n",
    "    return least_memory_gpu\n",
    "\n",
    "# Allocate the remaining GPU memory to the current process\n",
    "device = get_least_memory_gpu()\n",
    "torch.cuda.set_device(device)\n",
    "print('device:',device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lora 적용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def apply_lora(base_model_path, lora_path):\n",
    "    print(f\"Loading the base model from {base_model_path}\")\n",
    "    base = AutoModelForCausalLM.from_pretrained(base_model_path, torch_dtype=torch.float16, low_cpu_mem_usage=True,device_map=device)\n",
    "    base_tokenizer = AutoTokenizer.from_pretrained(base_model_path)\n",
    "    base_tokenizer.pad_token_id = 0\n",
    "    \n",
    "    print(f\"Loading the LoRA adapter from {lora_path}\")\n",
    "\n",
    "    lora_model = PeftModel.from_pretrained(\n",
    "        base,\n",
    "        lora_path,\n",
    "        torch_dtype=torch.float16,\n",
    "    )\n",
    "\n",
    "    print(\"Applying the LoRA\")\n",
    "    merged_model = lora_model.merge_and_unload()\n",
    "\n",
    "    return merged_model, base_tokenizer\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    print(df['output'][i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompt template\n",
    "\n",
    "\"아래는 작업을 설명하는 명령어와 추가 컨텍스트를 제공하는 입력이 짝을 이루는 예제입니다. 요청을 적절히 완료하는 응답을 작성하세요.\\n\\n### 명령어:\\n{instruction}\\n\\n### 입력:\\n{input}\\n\\n### 응답:\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = 'nlpai-lab/kullm-polyglot-5.8b-v2'\n",
    "lora_path = './polyglot-ko-5.8B_KULLM_patent_30000'\n",
    "model, tokenizer = apply_lora(model_id, lora_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    instruction = df['instruction'][i]\n",
    "    input = df['input'][i]\n",
    "    \n",
    "    query = f\"아래는 작업을 설명하는 명령어와 추가 컨텍스트를 제공하는 입력이 짝을 이루는 예제입니다. 요청을 적절히 완료하는 응답을 작성하세요.\\n\\n### 명령어:\\n{instruction}\\n\\n### 입력:\\n{input}\\n\\n### 응답:\\n\"\n",
    "    answer_chat(query,model=model,tokenizer=tokenizer,device=device)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## merge 방법 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "메모리 사용량이 적은 GPU로 device 설정"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "device: 0\n"
     ]
    }
   ],
   "source": [
    "# Set the available GPU devices\n",
    "available_gpus = [0, 1]  # Update this list with the available GPU device IDs\n",
    "\n",
    "# Function to find the GPU with the least memory usage\n",
    "def get_least_memory_gpu():\n",
    "    least_memory = float('inf')\n",
    "    least_memory_gpu = None\n",
    "    for gpu in available_gpus:\n",
    "        allocated_memory = torch.cuda.memory_allocated(gpu)\n",
    "        if allocated_memory < least_memory:\n",
    "            least_memory = allocated_memory\n",
    "            least_memory_gpu = gpu\n",
    "    return least_memory_gpu\n",
    "\n",
    "# Allocate the remaining GPU memory to the current process\n",
    "device = get_least_memory_gpu()\n",
    "torch.cuda.set_device(device)\n",
    "print('device:',device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_id = 'nlpai-lab/kullm-polyglot-5.8b-v2'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lora config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "lora_r= 16           # 153 k 데이터 8 에폭 학습할 때\n",
    "lora_alpha = 8\n",
    "lora_dropout= 0.05\n",
    "lora_target_modules = [\"query_key_value\", \"xxx\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "load pretrained model weight and lora weight"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8db27ae9a8bb462782837fb6e4289048",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "_IncompatibleKeys(missing_keys=['base_model.model.gpt_neox.embed_in.weight', 'base_model.model.gpt_neox.layers.0.input_layernorm.weight', 'base_model.model.gpt_neox.layers.0.input_layernorm.bias', 'base_model.model.gpt_neox.layers.0.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.0.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.0.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.0.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.0.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.0.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.0.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.0.attention.dense.weight', 'base_model.model.gpt_neox.layers.0.attention.dense.bias', 'base_model.model.gpt_neox.layers.0.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.0.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.0.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.0.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.1.input_layernorm.weight', 'base_model.model.gpt_neox.layers.1.input_layernorm.bias', 'base_model.model.gpt_neox.layers.1.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.1.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.1.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.1.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.1.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.1.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.1.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.1.attention.dense.weight', 'base_model.model.gpt_neox.layers.1.attention.dense.bias', 'base_model.model.gpt_neox.layers.1.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.1.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.1.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.1.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.2.input_layernorm.weight', 'base_model.model.gpt_neox.layers.2.input_layernorm.bias', 'base_model.model.gpt_neox.layers.2.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.2.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.2.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.2.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.2.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.2.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.2.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.2.attention.dense.weight', 'base_model.model.gpt_neox.layers.2.attention.dense.bias', 'base_model.model.gpt_neox.layers.2.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.2.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.2.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.2.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.3.input_layernorm.weight', 'base_model.model.gpt_neox.layers.3.input_layernorm.bias', 'base_model.model.gpt_neox.layers.3.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.3.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.3.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.3.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.3.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.3.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.3.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.3.attention.dense.weight', 'base_model.model.gpt_neox.layers.3.attention.dense.bias', 'base_model.model.gpt_neox.layers.3.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.3.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.3.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.3.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.4.input_layernorm.weight', 'base_model.model.gpt_neox.layers.4.input_layernorm.bias', 'base_model.model.gpt_neox.layers.4.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.4.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.4.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.4.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.4.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.4.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.4.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.4.attention.dense.weight', 'base_model.model.gpt_neox.layers.4.attention.dense.bias', 'base_model.model.gpt_neox.layers.4.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.4.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.4.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.4.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.5.input_layernorm.weight', 'base_model.model.gpt_neox.layers.5.input_layernorm.bias', 'base_model.model.gpt_neox.layers.5.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.5.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.5.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.5.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.5.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.5.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.5.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.5.attention.dense.weight', 'base_model.model.gpt_neox.layers.5.attention.dense.bias', 'base_model.model.gpt_neox.layers.5.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.5.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.5.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.5.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.6.input_layernorm.weight', 'base_model.model.gpt_neox.layers.6.input_layernorm.bias', 'base_model.model.gpt_neox.layers.6.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.6.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.6.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.6.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.6.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.6.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.6.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.6.attention.dense.weight', 'base_model.model.gpt_neox.layers.6.attention.dense.bias', 'base_model.model.gpt_neox.layers.6.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.6.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.6.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.6.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.7.input_layernorm.weight', 'base_model.model.gpt_neox.layers.7.input_layernorm.bias', 'base_model.model.gpt_neox.layers.7.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.7.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.7.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.7.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.7.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.7.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.7.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.7.attention.dense.weight', 'base_model.model.gpt_neox.layers.7.attention.dense.bias', 'base_model.model.gpt_neox.layers.7.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.7.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.7.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.7.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.8.input_layernorm.weight', 'base_model.model.gpt_neox.layers.8.input_layernorm.bias', 'base_model.model.gpt_neox.layers.8.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.8.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.8.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.8.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.8.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.8.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.8.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.8.attention.dense.weight', 'base_model.model.gpt_neox.layers.8.attention.dense.bias', 'base_model.model.gpt_neox.layers.8.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.8.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.8.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.8.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.9.input_layernorm.weight', 'base_model.model.gpt_neox.layers.9.input_layernorm.bias', 'base_model.model.gpt_neox.layers.9.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.9.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.9.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.9.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.9.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.9.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.9.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.9.attention.dense.weight', 'base_model.model.gpt_neox.layers.9.attention.dense.bias', 'base_model.model.gpt_neox.layers.9.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.9.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.9.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.9.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.10.input_layernorm.weight', 'base_model.model.gpt_neox.layers.10.input_layernorm.bias', 'base_model.model.gpt_neox.layers.10.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.10.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.10.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.10.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.10.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.10.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.10.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.10.attention.dense.weight', 'base_model.model.gpt_neox.layers.10.attention.dense.bias', 'base_model.model.gpt_neox.layers.10.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.10.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.10.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.10.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.11.input_layernorm.weight', 'base_model.model.gpt_neox.layers.11.input_layernorm.bias', 'base_model.model.gpt_neox.layers.11.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.11.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.11.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.11.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.11.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.11.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.11.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.11.attention.dense.weight', 'base_model.model.gpt_neox.layers.11.attention.dense.bias', 'base_model.model.gpt_neox.layers.11.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.11.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.11.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.11.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.12.input_layernorm.weight', 'base_model.model.gpt_neox.layers.12.input_layernorm.bias', 'base_model.model.gpt_neox.layers.12.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.12.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.12.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.12.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.12.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.12.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.12.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.12.attention.dense.weight', 'base_model.model.gpt_neox.layers.12.attention.dense.bias', 'base_model.model.gpt_neox.layers.12.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.12.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.12.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.12.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.13.input_layernorm.weight', 'base_model.model.gpt_neox.layers.13.input_layernorm.bias', 'base_model.model.gpt_neox.layers.13.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.13.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.13.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.13.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.13.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.13.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.13.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.13.attention.dense.weight', 'base_model.model.gpt_neox.layers.13.attention.dense.bias', 'base_model.model.gpt_neox.layers.13.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.13.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.13.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.13.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.14.input_layernorm.weight', 'base_model.model.gpt_neox.layers.14.input_layernorm.bias', 'base_model.model.gpt_neox.layers.14.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.14.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.14.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.14.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.14.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.14.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.14.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.14.attention.dense.weight', 'base_model.model.gpt_neox.layers.14.attention.dense.bias', 'base_model.model.gpt_neox.layers.14.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.14.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.14.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.14.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.15.input_layernorm.weight', 'base_model.model.gpt_neox.layers.15.input_layernorm.bias', 'base_model.model.gpt_neox.layers.15.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.15.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.15.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.15.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.15.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.15.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.15.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.15.attention.dense.weight', 'base_model.model.gpt_neox.layers.15.attention.dense.bias', 'base_model.model.gpt_neox.layers.15.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.15.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.15.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.15.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.16.input_layernorm.weight', 'base_model.model.gpt_neox.layers.16.input_layernorm.bias', 'base_model.model.gpt_neox.layers.16.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.16.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.16.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.16.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.16.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.16.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.16.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.16.attention.dense.weight', 'base_model.model.gpt_neox.layers.16.attention.dense.bias', 'base_model.model.gpt_neox.layers.16.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.16.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.16.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.16.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.17.input_layernorm.weight', 'base_model.model.gpt_neox.layers.17.input_layernorm.bias', 'base_model.model.gpt_neox.layers.17.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.17.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.17.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.17.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.17.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.17.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.17.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.17.attention.dense.weight', 'base_model.model.gpt_neox.layers.17.attention.dense.bias', 'base_model.model.gpt_neox.layers.17.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.17.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.17.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.17.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.18.input_layernorm.weight', 'base_model.model.gpt_neox.layers.18.input_layernorm.bias', 'base_model.model.gpt_neox.layers.18.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.18.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.18.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.18.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.18.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.18.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.18.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.18.attention.dense.weight', 'base_model.model.gpt_neox.layers.18.attention.dense.bias', 'base_model.model.gpt_neox.layers.18.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.18.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.18.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.18.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.19.input_layernorm.weight', 'base_model.model.gpt_neox.layers.19.input_layernorm.bias', 'base_model.model.gpt_neox.layers.19.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.19.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.19.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.19.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.19.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.19.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.19.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.19.attention.dense.weight', 'base_model.model.gpt_neox.layers.19.attention.dense.bias', 'base_model.model.gpt_neox.layers.19.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.19.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.19.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.19.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.20.input_layernorm.weight', 'base_model.model.gpt_neox.layers.20.input_layernorm.bias', 'base_model.model.gpt_neox.layers.20.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.20.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.20.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.20.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.20.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.20.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.20.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.20.attention.dense.weight', 'base_model.model.gpt_neox.layers.20.attention.dense.bias', 'base_model.model.gpt_neox.layers.20.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.20.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.20.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.20.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.21.input_layernorm.weight', 'base_model.model.gpt_neox.layers.21.input_layernorm.bias', 'base_model.model.gpt_neox.layers.21.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.21.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.21.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.21.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.21.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.21.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.21.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.21.attention.dense.weight', 'base_model.model.gpt_neox.layers.21.attention.dense.bias', 'base_model.model.gpt_neox.layers.21.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.21.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.21.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.21.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.22.input_layernorm.weight', 'base_model.model.gpt_neox.layers.22.input_layernorm.bias', 'base_model.model.gpt_neox.layers.22.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.22.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.22.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.22.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.22.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.22.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.22.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.22.attention.dense.weight', 'base_model.model.gpt_neox.layers.22.attention.dense.bias', 'base_model.model.gpt_neox.layers.22.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.22.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.22.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.22.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.23.input_layernorm.weight', 'base_model.model.gpt_neox.layers.23.input_layernorm.bias', 'base_model.model.gpt_neox.layers.23.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.23.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.23.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.23.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.23.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.23.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.23.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.23.attention.dense.weight', 'base_model.model.gpt_neox.layers.23.attention.dense.bias', 'base_model.model.gpt_neox.layers.23.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.23.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.23.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.23.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.24.input_layernorm.weight', 'base_model.model.gpt_neox.layers.24.input_layernorm.bias', 'base_model.model.gpt_neox.layers.24.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.24.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.24.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.24.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.24.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.24.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.24.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.24.attention.dense.weight', 'base_model.model.gpt_neox.layers.24.attention.dense.bias', 'base_model.model.gpt_neox.layers.24.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.24.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.24.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.24.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.25.input_layernorm.weight', 'base_model.model.gpt_neox.layers.25.input_layernorm.bias', 'base_model.model.gpt_neox.layers.25.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.25.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.25.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.25.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.25.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.25.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.25.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.25.attention.dense.weight', 'base_model.model.gpt_neox.layers.25.attention.dense.bias', 'base_model.model.gpt_neox.layers.25.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.25.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.25.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.25.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.26.input_layernorm.weight', 'base_model.model.gpt_neox.layers.26.input_layernorm.bias', 'base_model.model.gpt_neox.layers.26.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.26.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.26.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.26.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.26.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.26.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.26.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.26.attention.dense.weight', 'base_model.model.gpt_neox.layers.26.attention.dense.bias', 'base_model.model.gpt_neox.layers.26.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.26.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.26.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.26.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.layers.27.input_layernorm.weight', 'base_model.model.gpt_neox.layers.27.input_layernorm.bias', 'base_model.model.gpt_neox.layers.27.post_attention_layernorm.weight', 'base_model.model.gpt_neox.layers.27.post_attention_layernorm.bias', 'base_model.model.gpt_neox.layers.27.attention.rotary_emb.inv_freq', 'base_model.model.gpt_neox.layers.27.attention.query_key_value.weight', 'base_model.model.gpt_neox.layers.27.attention.query_key_value.bias', 'base_model.model.gpt_neox.layers.27.attention.query_key_value.lora_A.default.weight', 'base_model.model.gpt_neox.layers.27.attention.query_key_value.lora_B.default.weight', 'base_model.model.gpt_neox.layers.27.attention.dense.weight', 'base_model.model.gpt_neox.layers.27.attention.dense.bias', 'base_model.model.gpt_neox.layers.27.mlp.dense_h_to_4h.weight', 'base_model.model.gpt_neox.layers.27.mlp.dense_h_to_4h.bias', 'base_model.model.gpt_neox.layers.27.mlp.dense_4h_to_h.weight', 'base_model.model.gpt_neox.layers.27.mlp.dense_4h_to_h.bias', 'base_model.model.gpt_neox.final_layer_norm.weight', 'base_model.model.gpt_neox.final_layer_norm.bias', 'base_model.model.embed_out.weight'], unexpected_keys=[])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_id,\n",
    "    load_in_8bit=True,\n",
    "    torch_dtype=torch.float16,\n",
    "    device_map=device,\n",
    ")\n",
    "\n",
    "old_state_dict = model.state_dict\n",
    "model.state_dict = (lambda self, *_, **__: get_peft_model_state_dict(self, old_state_dict())).__get__(\n",
    "    model, type(model)\n",
    ")\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_id)\n",
    "    \n",
    "model = prepare_model_for_kbit_training(model)\n",
    "\n",
    "config = LoraConfig(\n",
    "    r=lora_r,\n",
    "    lora_alpha=lora_alpha,\n",
    "    target_modules=lora_target_modules,\n",
    "    lora_dropout=lora_dropout,\n",
    "    bias=\"none\",\n",
    "    task_type=\"CAUSAL_LM\",\n",
    ")\n",
    "model = get_peft_model(model, config)\n",
    "\n",
    "\n",
    "checkpoint_name =  \"output/checkpoint-50000/adapter_model.bin\"\n",
    "adapters_weights = torch.load(checkpoint_name)\n",
    "set_peft_model_state_dict(model, adapters_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "학습 데이터 inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(3):\n",
    "    instruction = df['instruction'][i]\n",
    "    input = df['input'][i]\n",
    "    \n",
    "    query = f\"아래는 작업을 설명하는 명령어와 추가 컨텍스트를 제공하는 입력이 짝을 이루는 예제입니다. 요청을 적절히 완료하는 응답을 작성하세요.\\n\\n### 명령어:\\n{instruction}\\n\\n### 입력:\\n{input}\\n\\n### 응답:\\n\"\n",
    "    answer_chat(query,model=model,tokenizer=tokenizer,device=device)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "instruction에 질문 날리기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/brianjang7/home1/.venv/lib/python3.11/site-packages/bitsandbytes/autograd/_functions.py:321: UserWarning: MatMul8bitLt: inputs will be cast from torch.float32 to float16 during quantization\n",
      "  warnings.warn(f\"MatMul8bitLt: inputs will be cast from {A.dtype} to float16 during quantization\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "아래는 작업을 설명하는 명령어와 추가 컨텍스트를 제공하는 입력이 짝을 이루는 예제입니다. 요청을 적절히 완료하는 응답을 작성하세요.\n",
      "\n",
      "### 명령어:\n",
      "가방이 무거워? 책이 무거워? 이유를 설명해줘.\n",
      "\n",
      "### 입력:\n",
      "\n",
      "\n",
      "### 응답:\n",
      "죄송하지만 가방과 책의 무게를 비교할 수 있는 구체적인 데이터나 정보가 없습니다. 가방과 책의 무게는 개인의 체격, 체격, 가방의 크기와 모양에 따라 달라질 수 있습니다.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "instruction = '가방이 무거워? 책이 무거워? 이유를 설명해줘.'\n",
    "input = ''      # 예시가 없으면 input은 공란\n",
    "\n",
    "query = f\"아래는 작업을 설명하는 명령어와 추가 컨텍스트를 제공하는 입력이 짝을 이루는 예제입니다. 요청을 적절히 완료하는 응답을 작성하세요.\\n\\n### 명령어:\\n{instruction}\\n\\n### 입력:\\n{input}\\n\\n### 응답:\\n\"\n",
    "answer_chat(query,model=model,tokenizer=tokenizer,device=device)\n",
    "print()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}