20221111(2)_YES24API.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/heejininku/class2022Fall/blob/main/20221111(2)_YES24API.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "교보문고보다 복잡함.\n",
        "\n",
        "\n",
        "여기는 예쁘게 정리 안되어있음"
      ],
      "metadata": {
        "id": "u_ETBqZbJcx5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import json\n",
        "import requests\n",
        "from bs4 import BeautifulSoup #크롤링할 때 체계적이지 않을 때에 사용하는 BeautifulSoup #soup:잡탕 - 아름답게 정리하는 것\n",
        "import re\n",
        "import numpy as np\n"
      ],
      "metadata": {
        "id": "nhZj-Bi8Atg2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "bookID = '176787' #사이트 위에 북 아이디 있음."
      ],
      "metadata": {
        "id": "j4Cj1FNgAyDU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "url = f'http://www.yes24.com/Product/communityModules/GoodsReviewList/{bookID}?Sort=1&PageNumber=1&Type=ALL'\n",
        "#여기도 요청임. \n",
        "response = requests.get(url) \n",
        "#그냥 response하면 (response.text하면 내부 보여주는데 너무 지저분) 지저분해서\n",
        "#아래처럼 soup로 어느정도 정리\n",
        "soup = BeautifulSoup(response.content, 'html.parser')\n",
        "tmp = soup.find(string=re.compile('reviewCountText'))\n",
        "tmp = str(tmp)\n",
        "cnt = re.findall('(?<= reviewCountText = \\')\\d*(?=\\')', tmp)\n",
        "cnt = int(cnt[0])\n",
        "\n",
        "nPage = int(np.ceil(cnt/5))"
      ],
      "metadata": {
        "id": "q9xZlnJzSP5i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "data = {'title': [],\n",
        "        'ID': [],\n",
        "        'date': [],\n",
        "        'contRating': [],\n",
        "        'designRating': [],\n",
        "        'text': []}\n",
        "df = pd.DataFrame(data)"
      ],
      "metadata": {
        "id": "MkNNj561QaZh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "아래 복잡해보여도 구조로는 모르는 게 없어야 함."
      ],
      "metadata": {
        "id": "t5ukKMUwKHQ0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "i = 0\n",
        "for n in range(nPage):\n",
        "  PageNumber = n + 1\n",
        "  url = f'http://www.yes24.com/Product/communityModules/GoodsReviewList/{bookID}?Sort=1&PageNumber={PageNumber}&Type=ALL'\n",
        "  response = requests.get(url) #입력이 url\n",
        "  \n",
        "  soup = BeautifulSoup(response.content, 'html.parser')\n",
        "  title = soup.select('span.txt')\n",
        "  ID = soup.select('a.lnk_id')\n",
        "  date = soup.select('em.txt_date')\n",
        "  rating = soup.select('span.review_rating > span.rating');\n",
        "  contRating = rating[0:9:2]; designRating = rating[1:10:2]\n",
        "  text = soup.select('div.reviewInfoBot.origin > div.review_cont')\n",
        "\n",
        "  for r in zip(title, ID, date, contRating, designRating, text):\n",
        "    title, ID, date, contRating, designRating, text = [r[0].get_text(), r[1].get_text(), r[2].get_text(), r[3].get_text(), r[4].get_text(), r[5].get_text()]    \n",
        "    row = [title, ID, date, contRating, designRating, text]\n",
        "    df.loc[i, :] = row\n",
        "    i +=1"
      ],
      "metadata": {
        "id": "YJ6D1OVA4MLk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import xlwt\n",
        "df.to_excel('review.xls')"
      ],
      "metadata": {
        "id": "Mgr-AdFGYVIj",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8af2b823-8a51-4844-fb59-c03acc4e0c3f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:2: FutureWarning: As the xlwt package is no longer maintained, the xlwt engine will be removed in a future version of pandas. This is the only engine in pandas that supports writing in the xls format. Install openpyxl and write to an xlsx file instead. You can set the option io.excel.xls.writer to 'xlwt' to silence this warning. While this option is deprecated and will also raise a warning, it can be globally set and the warning suppressed.\n",
            "  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "엑셀 안에 제목, 아이디, 날짜, 리뷰 등"
      ],
      "metadata": {
        "id": "_ecGvrDZKSr7"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "나중에 텍스트 가지고 분석해보기\n",
        "nlp 자연어처리 활용\n",
        "\n",
        "구글에 자연어처리 검색도 하심.\n",
        "기본적으로 텍스트 처리하는 것.\n",
        "nltk 자연어처리_테이터 사이언스 스쿨\n",
        "교수님 유튜브 강의도 있음."
      ],
      "metadata": {
        "id": "bI3e3nh4KYFm"
      }
    }
  ]
}