Googlenews_test.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# GoogleNews 套件語法測試"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> GoogleNews Version: 1.6.15\n",
    "\n",
    "> Python version : 3.12.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 匯入模組"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "jupyter": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from GoogleNews import GoogleNews\n",
    "googlenews = GoogleNews()\n",
    "\n",
    "# 透過調用 GoogleNews 建構函數 產生一個 ”googlenews” 物件 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "beautifulsoup4  網頁解析和數據提取\n",
    "dateparser      日期字符串解析\n",
    "GoogleNews      Google新聞搜索API\n",
    "pip             Python包管理工具\n",
    "python-dateutil 日期和時間操作擴展\n",
    "pytz            時區定義和轉換\n",
    "regex           增強的正則表達式功能\n",
    "six             Python 2和3兼容性庫\n",
    "soupsieve       CSS選擇器引擎，用於beautifulsoup4\n",
    "tzlocal         本地時區檢測\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查看套件版本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-08T09:17:23.876018Z",
     "start_time": "2024-09-08T09:17:23.744317Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6.15\n"
     ]
    }
   ],
   "source": [
    "print(googlenews.getVersion())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> or using pip show\n",
    "```python=\n",
    "(.venv) ➜  Python-WordCloud git:(master) ✗ pip show googlenews \n",
    "Name: GoogleNews\n",
    "Version: 1.6.15\n",
    "Summary: Google News search for Python\n",
    "Home-page: https://github.com/Iceloof/GoogleNews\n",
    "Author: Hurin Hu\n",
    "Author-email: hurin@live.ca\n",
    "License: \n",
    "Location: /.venv/lib/python3.12/site-packages\n",
    "Requires: beautifulsoup4, dateparser, python-dateutil\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 查看套件可使用參數"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<GoogleNews.GoogleNews object at 0x104294d30>\n",
      "<class 'GoogleNews.GoogleNews'>\n",
      "可以使用的屬性與方法 : \n",
      "['_GoogleNews__encode', '_GoogleNews__end', '_GoogleNews__exception', '_GoogleNews__lang', '_GoogleNews__links', '_GoogleNews__period', '_GoogleNews__results', '_GoogleNews__section', '_GoogleNews__start', '_GoogleNews__texts', '_GoogleNews__topic', '_GoogleNews__totalcount', '_GoogleNews__version', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'build_response', 'clear', 'enableException', 'getVersion', 'get_links', 'get_news', 'get_page', 'get_texts', 'getpage', 'gettext', 'headers', 'page_at', 'remove_after_last_fullstop', 'result', 'results', 'search', 'setTimeRange', 'set_encode', 'set_lang', 'set_period', 'set_section', 'set_time_range', 'set_topic', 'setencode', 'setlang', 'setperiod', 'total_count', 'user_agent']\n"
     ]
    }
   ],
   "source": [
    "print(googlenews)\n",
    "print(type(googlenews))\n",
    "print(\"可以使用的屬性與方法 : \")\n",
    "print(dir(googlenews))\n",
    "\n",
    "# 觀察 class 類別 擁有的屬性與方法\n",
    "\n",
    "# class 類別 說明文章：\n",
    "# https://utrustcorp.com/python-class/\n",
    "# https://steam.oxxostudio.tw/category/python/basic/class.html#google_vignette"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(移除底線屬性後)\n",
    "\n",
    "['build_response', 'clear', 'content', 'enableException', 'getVersion', 'get_links', 'get_news', 'get_page', 'get_texts', 'getpage', 'gettext', 'headers', 'page', 'page_at', 'remove_after_last_fullstop', 'req', 'response', 'result', 'results', 'search', 'setTimeRange', 'set_encode', 'set_lang', 'set_period', 'set_section', 'set_time_range', 'set_topic', 'setencode', 'setlang', 'setperiod', 'total_count', 'url', 'user_agent']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 建構物件 設定參數"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 設定搜尋新聞日期 特定語言的新聞 zh-TW\n",
    "# 使用屬性設定參數\n",
    "googlenews = GoogleNews(lang='zh-tw', region='TW', encode='utf-8', period='1d')\n",
    "# googlenews = GoogleNews(encode='utf-8')\n",
    "# googlenews = GoogleNews(period='7d')\n",
    "# googlenews = GoogleNews(start='02/01/2020',end='02/28/2020')\n",
    "# 或是使用方法來設定參數\n",
    "# googlenews.set_encode('utf-8')\n",
    "# googlenews.set_period('7d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 開啟除錯模式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "googlenews.enableException(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "延伸學習：Try...except\n",
    "\n",
    "https://hackmd.io/@DCT/python-function-object-exception\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Google 搜尋網址參數設定"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "查看該套件的部分程式碼 觀察參數如何對應 google 搜尋網址\n",
    "\n",
    "```python=\n",
    "    def page_at(self, page=1):\n",
    "        \"\"\"\n",
    "        Retrieves a specific page from google.com in the news sections into __results.\n",
    "        Parameter:\n",
    "        page = number of the page to be retrieved\n",
    "        \"\"\"\n",
    "        results = []\n",
    "        try:\n",
    "            if self.__start != \"\" and self.__end != \"\":\n",
    "                self.url = \"https://www.google.com/search?q={}&lr=lang_{}&biw=1920&bih=976&source=lnt&&tbs=lr:lang_1{},cdr:1,cd_min:{},cd_max:{},sbd:1&tbm=nws&start={}\".format(self.__key,self.__lang,self.__lang,self.__start,self.__end,(10 * (page - 1)))\n",
    "            elif self.__period != \"\":\n",
    "                self.url = \"https://www.google.com/search?q={}&lr=lang_{}&biw=1920&bih=976&source=lnt&&tbs=lr:lang_1{},qdr:{},,sbd:1&tbm=nws&start={}\".format(self.__key,self.__lang,self.__lang,self.__period,(10 * (page - 1))) \n",
    "            else:\n",
    "                self.url = \"https://www.google.com/search?q={}&lr=lang_{}&biw=1920&bih=976&source=lnt&&tbs=lr:lang_1{},sbd:1&tbm=nws&start={}\".format(self.__key,self.__lang,self.__lang,(10 * (page - 1))) \n",
    "\n",
    "    ----\n",
    "\n",
    "\n",
    "            if region:\n",
    "            self.accept_language= lang + '-' + region + ',' + lang + ';q=0.9'\n",
    "            self.headers = {'User-Agent': self.user_agent, 'Accept-Language': self.accept_language}\n",
    "        else:\n",
    "            self.headers = {'User-Agent': self.user_agent}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "延伸學習：format 字串格式化\n",
    "\n",
    "https://hackmd.io/@DCT/python-function-object-exception"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 觀察網址結構，了解 Google 網址參數設定代表意義\n",
    "\n",
    "self.url = \"https://www.google.com/search?q={}&lr=lang_{}&tbs=lr:lang_1{},cdr:1,cd_min:{},cd_max:{}&tbm=nws&start={}\".format(self.__key,self.__lang,self.__lang,self.__start,self.__end,(10 * (page - 1)))\n",
    "\n",
    "\n",
    "> Google 官方搜尋網址參數說明：\n",
    "https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list?hl=zh-tw\n",
    "\n",
    "hl 即 host language（設定使用者介面語言）\n",
    "\n",
    "gl 從使用者的地理位置搜尋相關新聞\n",
    "\n",
    "lr 僅回傳搜尋特定語言為主的網頁結果\n",
    "\n",
    "> hl 介面語言代號：\n",
    "https://developers.google.com/custom-search/docs/json_api_reference?hl=zh-tw#interfaceLanguages\n",
    "\n",
    "> gl 國家/地區代碼：\n",
    "https://developers.google.com/custom-search/docs/json_api_reference?hl=zh-tw#countryCodes\n",
    "\n",
    "> lr 篩選特定語言：\n",
    "https://developers.google.com/custom-search/v1/reference/rest/v1/cse/list?hl=zh-tw\n",
    "\n",
    "\n",
    "hl : \"zh-TW\" 中文 (繁體)\n",
    "\n",
    "gl : \"tw\" 台灣\n",
    "\n",
    "lr : \"lang_zh-TW\" 中文 (繁體)\n",
    "\n",
    "(可是我發現上述 google 提供的網址 hl 都是小寫 \"zh-tw\" 經測試都可以運作 0.0)\n",
    "\n",
    "https://www.google.com/search?q=搜尋&hl=zh-tw&lr=lang_zh-TW\n",
    "\n",
    "https://www.google.com/search?q=搜尋&hl=zh-tw&lr=lang_zh-CN\n",
    "\n",
    "https://www.google.com/search?q=搜尋&hl=zh-TW&lr=lang_zh-TW\n",
    "\n",
    "\n",
    ">中文說明講解：\n",
    "https://www.biaodianfu.com/google.html\n",
    "\n",
    ">英文對照表(使用同樣參數)：\n",
    "Supported Google Languages\n",
    "https://serpapi.com/google-languages\n",
    "\n",
    "發現到它是一家提供 google 服務 api 的服務\n",
    "https://serpapi.com/\n",
    "\n",
    "免費版本，有使用限制。\n",
    "https://serpapi.com/search.json?q=貪污&location=Taiwan&hl=zh-tw&gl=tw&google_domain=google.com.tw\n",
    "\n",
    "蠻方便的，直接回傳 json 檔案，給的資訊更多。\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Region 搜尋設定問題及討論：\n",
    "\n",
    "Region setting in Chinese\n",
    "\n",
    "https://github.com/Iceloof/GoogleNews/issues/76"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 搜尋特定關鍵字"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 用 get_news() or search() 搜尋\n",
    "\n",
    "> 搜尋結果透過 result() 讀取出來"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## get_news()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "googlenews.get_news('taiwan typhoon')\n",
    "gn_get = googlenews.results()\n",
    "# print ( gn_get )\n",
    "# print ( type(gn_get) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## search() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'title': '全台續放颱風假 粉專：山陀兒非首例 上1個追溯23年前', 'media': '聯合新聞網', 'date': '7 分鐘前', 'datetime': nan, 'desc': '山陀兒颱風緩慢接近台灣，雖強度減弱，但仍對全台各地造成影響，全台連放2天颱風假，氣象粉專「用事實說颱風」指出，山陀兒並非...', 'link': 'https://udn.com/news/story/7266/8267047&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIABAC&usg=AOvVaw3dLK4aZR746K6lchD2agMV', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '救謝國樑罷免案？北北基桃放颱風假「基隆偷跑」 市長們回應了', 'media': '鏡週刊Mirror Media', 'date': '2 小時前', 'datetime': nan, 'desc': '因應颱風山陀兒，全台今（2日）皆放颱風假，然而颱風預計今晚才會從南部登陸，再加上北台灣今日上午風雨不大，北北基桃放假的決策引發質疑，尤其昨晚基隆又率先其他3市搶先...', 'link': 'https://www.mirrormedia.mg/story/20241002edi030&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQICRAC&usg=AOvVaw0EsVwmAfNJVBMAxRXSxOoH', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '換位思考 放錯假也隨遇而安', 'media': '聯合新聞網', 'date': '2 小時前', 'datetime': nan, 'desc': '全台昨天放颱風假。一早起來，無雨微風，太陽還露臉，下午偶有陣風，整體來說，是天涼好個秋的舒爽天氣。網路上有人欣喜，有人抱...', 'link': 'https://udn.com/news/story/7339/8266932&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIARAC&usg=AOvVaw3nHdkvD1EUoZcMq7gTiHsd', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '放颱風假 已成全民基因', 'media': '聯合新聞網', 'date': '2 小時前', 'datetime': nan, 'desc': '一年至少有四分之一的時間面對颱風逼近的我們，放颱風假與否長期引發社會討論，從過往防災措施避禍演變成一種「社會現象」。颱風假的議題，可從民粹、團體盲動和休閒逃避...', 'link': 'https://udn.com/news/story/7339/8266946&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIBxAC&usg=AOvVaw3UOTOmSUmEeJWQv1WVfxGt', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '北北基桃再放颱風假！全台22縣市明天停班課（總整理）', 'media': 'Yahoo新聞', 'date': '3 小時前', 'datetime': nan, 'desc': '【即時中心／綜合報導】根據中央氣象署最新資料顯示，中度颱風山陀兒目前在高雄西南方約110公里處，以每小時8公里速度，向東北轉北北東進行，近中心最大風速每秒43公尺（時速...', 'link': 'https://tw.news.yahoo.com/%25E5%258C%2597%25E5%258C%2597%25E5%259F%25BA%25E6%25A1%2583%25E5%2586%258D%25E6%2594%25BE%25E9%25A2%25B1%25E9%25A2%25A8%25E5%2581%2587-%25E5%2585%25A8%25E5%258F%25B022%25E7%25B8%25A3%25E5%25B8%2582%25E6%2598%258E%25E5%25A4%25A9%25E5%2581%259C%25E7%258F%25AD%25E8%25AA%25B2-%25E7%25B8%25BD%25E6%2595%25B4%25E7%2590%2586-170400268.html&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIAhAC&usg=AOvVaw39-QrjQoKrHQiM4L8XAAiT', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '新北週四再放一天颱風假 釣蝦場客人挑燈「夜釣」', 'media': 'Yahoo新聞', 'date': '3 小時前', 'datetime': nan, 'desc': '社會中心／黃國誠宮仲毅新北市報導受颱風影響，新北市週三放颱風假，但一直到晚上才開始飄雨，風雨並不大，不少民眾，前往KTV歡唱，甚至跑到釣蝦場打發時間，而新北市宣布，週四...', 'link': 'https://tw.news.yahoo.com/%25E6%2596%25B0%25E5%258C%2597%25E9%2580%25B1%25E5%259B%259B%25E5%2586%258D%25E6%2594%25BE-%25E5%25A4%25A9%25E9%25A2%25B1%25E9%25A2%25A8%25E5%2581%2587-%25E9%2587%25A3%25E8%259D%25A6%25E5%25A0%25B4%25E5%25AE%25A2%25E4%25BA%25BA%25E6%258C%2591%25E7%2587%2588-%25E5%25A4%259C%25E9%2587%25A3-170328025.html&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQICBAC&usg=AOvVaw21g6xh_4gicRf-7ttQMrcT', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '聯合報黑白集／一個颱風三天假', 'media': '聯合新聞網', 'date': '3 小時前', 'datetime': nan, 'desc': '全台火速宣布放颱風假，「山陀兒」卻龜速前進，甚至原地打轉。南部縣市已連放兩天假，高雄市長陳其邁三叮嚀四吩咐，叫大家別去逛...', 'link': 'https://udn.com/news/story/7338/8266936&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIBhAC&usg=AOvVaw0vM6GpWlHN_gd4nEtfvIVz', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '鄭運鵬竟稱北北基桃放「罷免颱風假」 鍾沛君反嗆：怎不提「賴神勸童」？', 'media': 'Yahoo新聞', 'date': '3 小時前', 'datetime': nan, 'desc': '颱風山陀兒來襲，全台各地方政府2日均宣布停班停課，但民進黨前立委鄭運鵬卻酸北北基桃的「罷免颱風假」名留青史。對此，台北市議員鍾沛君痛批鄭運鵬，到底多看不起基隆...', 'link': 'https://tw.news.yahoo.com/%25E9%2584%25AD%25E9%2581%258B%25E9%25B5%25AC%25E7%25AB%259F%25E7%25A8%25B1%25E5%258C%2597%25E5%258C%2597%25E5%259F%25BA%25E6%25A1%2583%25E6%2594%25BE-%25E7%25BD%25B7%25E5%2585%258D%25E9%25A2%25B1%25E9%25A2%25A8%25E5%2581%2587-%25E9%258D%25BE%25E6%25B2%259B%25E5%2590%259B%25E5%258F%258D%25E5%2597%2586-%25E6%2580%258E%25E4%25B8%258D%25E6%258F%2590-%25E8%25B3%25B4%25E7%25A5%259E%25E5%258B%25B8%25E7%25AB%25A5-162539665.html&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIAxAC&usg=AOvVaw2EkabSXSyZs82iQvBBQa4y', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '再放颱風假! 山陀兒龜速前進 全台22縣市週四停班停課', 'media': 'Yahoo新聞', 'date': '3 小時前', 'datetime': nan, 'desc': '生活中心／綜合報導颱風山陀兒來襲，全台22縣市，週三都放颱風假，只是颱風龜速前進，中心直到週三晚間都還沒登陸，這讓全台22縣市首長都果斷決定，週四再放一天颱風假，...', 'link': 'https://tw.news.yahoo.com/%25E5%2586%258D%25E6%2594%25BE%25E9%25A2%25B1%25E9%25A2%25A8%25E5%2581%2587-%25E5%25B1%25B1%25E9%2599%2580%25E5%2585%2592%25E9%25BE%259C%25E9%2580%259F%25E5%2589%258D%25E9%2580%25B2-%25E5%2585%25A8%25E5%258F%25B022%25E7%25B8%25A3%25E5%25B8%2582%25E9%2580%25B1%25E5%259B%259B%25E5%2581%259C%25E7%258F%25AD%25E5%2581%259C%25E8%25AA%25B2-162307652.html&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIBRAC&usg=AOvVaw3XyH7vEQ-ogsyGhHgFjQWy', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}, {'title': '宜蘭入夜後風雨逐漸增強 民眾放颱風假開心逛夜市', 'media': '民視新聞網', 'date': '3 小時前', 'datetime': nan, 'desc': '地方中心／黃富溢宜蘭報導山陀兒颱風就要來了，宜蘭地區，相比上午無風無雨，入夜後雨勢不斷增強，到東門夜市可以看到，還是有不少民眾不畏風雨逛夜市，不過，就有攤商表示，...', 'link': 'https://www.ftvnews.com.tw/news/detail/2024A02N08M1&ved=2ahUKEwjJj8WAufCIAxXbm68BHSkxHaYQxfQBegQIBBAC&usg=AOvVaw1ndepfIlPAxYMPE813QqII', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}]\n"
     ]
    }
   ],
   "source": [
    "googlenews.clear()\n",
    "googlenews.search('颱風假')\n",
    "result_search_news = googlenews.result()\n",
    "print(result_search_news)\n",
    "with open(\"./check-datatype/search-result.txt\", \"a+\", encoding=\"utf-8\") as file:\n",
    "    file.write(\"Results from search test:\\n\")\n",
    "    file.write(str(result_search_news))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## get_texts() 純文字資料 標題"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['全台續放颱風假 粉專：山陀兒非首例 上1個追溯23年前', '救謝國樑罷免案？北北基桃放颱風假「基隆偷跑」 市長們回應了', '換位思考 放錯假也隨遇而安', '放颱風假 已成全民基因', '北北基桃再放颱風假！全台22縣市明天停班課（總整理）', '新北週四再放一天颱風假 釣蝦場客人挑燈「夜釣」', '聯合報黑白集／一個颱風三天假', '鄭運鵬竟稱北北基桃放「罷免颱風假」 鍾沛君反嗆：怎不提「賴神勸童」？', '再放颱風假! 山陀兒龜速前進 全台22縣市週四停班停課', '宜蘭入夜後風雨逐漸增強 民眾放颱風假開心逛夜市']\n"
     ]
    }
   ],
   "source": [
    "gn_txt = googlenews.get_texts()\n",
    "print(gn_txt)\n",
    "with open(\"./check-datatype/gn_text-result.txt\", \"a+\", encoding=\"utf-8\") as file:\n",
    "    file.write(\"Results from gn_text test:\\n\")\n",
    "    file.write(str(gn_txt))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 搜尋兩則新聞結果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.6.15\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "from GoogleNews import GoogleNews\n",
    "\n",
    "# 創建一個 GoogleNews 物件，這個物件可以用來檢索 Google News 中的新聞\n",
    "googlenews = GoogleNews()\n",
    "\n",
    "# 當前使用的 GoogleNews 模組版本\n",
    "print(googlenews.getVersion())\n",
    "\n",
    "# 啟用異常處理，當發生錯誤時會拋出異常，方便調試\n",
    "googlenews.enableException(True)\n",
    "\n",
    "googlenews = GoogleNews(lang='zh-TW', region='TW', encode='utf-8')\n",
    "# 設置 GoogleNews 只搜尋語言為繁體中文的網站\n",
    "\n",
    "# 設定編碼為 'utf-8'，避免出現編碼問題\n",
    "# googlenews = GoogleNews(encode='utf-8')\n",
    "\n",
    "# (兩種搜尋特定時間方式不能混用)\n",
    "# 設定新聞搜尋時間範圍為過去 n 天\n",
    "# googlenews = GoogleNews(period='7d')\n",
    "\n",
    "# 指定開始和結束日期，搜尋特定日期範圍內的新聞\n",
    "# googlenews = GoogleNews(start='02/01/2020',end='02/28/2020')\n",
    "\n",
    "# 使用方法設定，避免覆蓋原有物件設定\n",
    "# googlenews.set_period('7d')\n",
    "# googlenews.set_time_range('02/01/2020','02/28/2020')\n",
    "\n",
    "\n",
    "# 清空物件暫存搜尋結果\n",
    "googlenews.clear()\n",
    "googlenews.get_news('台灣')\n",
    "result_get_news = googlenews.result()\n",
    "# print(result_get_news)\n",
    "\n",
    "time.sleep(5)\n",
    "\n",
    "googlenews.clear()\n",
    "googlenews.search('柯文哲')\n",
    "result_search_news = googlenews.result()\n",
    "# print(result_search_news)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 可使用 pretty code 線上工具方便查看資料結構\n",
    "\n",
    "> https://www.cleancss.com/python-beautify/\n",
    "\n",
    "```\n",
    "\n",
    "result_get_news\n",
    "\n",
    "[{\n",
    "    'title': '「台灣嚴選（Taiwan Select）」食品進駐馬來西亞AEON超市',\n",
    "    'desc': None,\n",
    "    'date': '昨天',\n",
    "    'datetime': nan,\n",
    "    'link': 'https://news.google.com/read/CBMijAFBVV95cUxPZGNtenpSb2swNDJhcFZ1ZjVjcE1RZ0VuMGlkNXhZUURtR1lfRERWeFprbWcyaWYxeVhwNkVwcXJFRGZXQnNLamEwendQbENlR0ZUSnN4QjB2MVZtUVNhaTNTQ00zcUF2ZW9RT2RNdUFSdVQyN3M0Q3g2SWE0RUR0YW9nNEF4dTV6akhzVg?hl=zh-TW&gl=TW&ceid=TW%3Azh-Hant',\n",
    "    'img': 'https://news.google.com/api/attachments/CC8iK0NnNDJVbDl4YWsxa2RYSmxRbTlJVFJDM0FSaVRBaWdCTWdZSk1JcUdGZ28=-w200-h112-p-df',\n",
    "    'media': '中華民國經濟部',\n",
    "    'site': None,\n",
    "    'reporter': None\n",
    "}, {\n",
    "    'title': '《臺灣米其林指南 2024》完整名單',\n",
    "    'desc': None,\n",
    "    'date': '8月27日',\n",
    "    'datetime': nan,\n",
    "    'link': 'https://news.google.com/read/CBMijAFBVV95cUxOazZlR2tuNFhqT01HVkx4Rk5JX2xxWVFsTnpmRUJzbGRlVF9CWlpxTlJBU2lZb3ViX2huSUVLeS1nYXJmRS1wUW1XaDYwT0t5X3RNVExEQ3BjWVA2S2hreVAzSU5icnRiRW5qM0F3LWE2d1Z3aWRPcVJXSzBXU3V6dkYtLTNDQi1KYTkzVQ?hl=zh-TW&gl=TW&ceid=TW%3Azh-Hant',\n",
    "    'img': 'https://news.google.com/api/attachments/CC8iK0NnNTFaSE5GU1Vzd1dtOUlTM2hyVFJDOUFSaUxBaWdCTWdZQmNJUXV3UU0=-w200-h112-p-df',\n",
    "    'media': 'Michelin.com',\n",
    "    'site': None,\n",
    "    'reporter': None\n",
    "},....]\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 將資料儲存至文字檔案 txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "# 清空之前的結果\n",
    "googlenews.clear()\n",
    "\n",
    "# 搜尋關鍵字\n",
    "googlenews.search('颱風假')\n",
    "\n",
    "# 取得 0 - 2 頁的結果\n",
    "googlenews.get_page(2)\n",
    "\n",
    "result = googlenews.result()\n",
    "\n",
    "\n",
    "# 測試每個結果的索引和值\n",
    "# 將結果寫入文字檔案\n",
    "with open(\"./check-datatype/page/results-page0to2.txt\", \"w+\", encoding=\"utf-8\") as file:\n",
    "    file.write(\"Results from get_page(2):\\n\")\n",
    "    for n in range(len(result)):\n",
    "        file.write(f\"Result {n}:\\n\")\n",
    "        for key, value in result[n].items():\n",
    "            file.write(f\"{key}: {value}\\n\")\n",
    "        file.write(\"\\n\")\n",
    "        \n",
    "# item() 語法可以從字典中提取出來key value\n",
    "\n",
    "    \n",
    "# 清空之前的結果\n",
    "googlenews.clear()\n",
    "\n",
    "time.sleep(2)\n",
    "\n",
    "googlenews.search('柯文哲')\n",
    "\n",
    "# 取得第 2 頁的結果\n",
    "googlenews.page_at(2)\n",
    "\n",
    "result = googlenews.result()\n",
    "\n",
    "with open(\"./check-datatype/page/results-pageonly-2.txt\", \"w+\", encoding=\"utf-8\") as file:\n",
    "    file.write(\"Results from page_at(2):\\n\")\n",
    "    for n in range(len(result)):\n",
    "        file.write(f\"Result {n}:\\n\")\n",
    "        for key, value in result[n].items():\n",
    "            file.write(f\"{key}: {value}\\n\")\n",
    "        file.write(\"\\n\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 將資料分開儲存至串列"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['柯文哲被問到火大拒應訊？沈政男：覺得浪費時間「摔病歷」', '柯文哲拒提訊綠諷「自知大勢已去」 蔡正元揣測「聰明人」深意', '幕後／朱亞虎認送錢後柯文哲突拒應訊 使出緘默險棋恐增延押機率', '柯文哲2個月後若又延押 郭正亮驚人預言1事', '柯文哲遭羈押黨內同志、小草們齊聲援 傳鄭文燦吐苦水怨嘆：沒人聲援我', '柯文哲盯監視器「總統一笑置之」 黃國昌怒嗆：怎麼笑得出來', '【完整版下集】李文宗、朱亞虎被收押 突破柯文哲疑不法金流全貌？ 少康戰情室 20241002', '【#原音重現】柯文哲拒絕北檢傳喚！苗博雅曝訴訟策略成分高？因朱亞虎坦承行賄供詞不能貿然更改：為自己爭取時間｜三立新聞網SETN.com', '對柯文哲沉痛控訴「一笑置之」 黃國昌怒嗆賴清德：怎麼還笑得出來？', '柯文哲被羈押小草聲援鄭文燦傳大吐苦水：沒人聲援他- 政治']\n",
      "10\n",
      "10\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "title_tokens = []\n",
    "desc_tokens = []\n",
    "link_tokens = []\n",
    "media_tokens = []\n",
    "datetime_tokens = []\n",
    "\n",
    "for n in range(len(result)):\n",
    "\n",
    "    title = result[n]['title']\n",
    "    desc = result[n]['desc']  \n",
    "    link = result[n]['link'] \n",
    "    media = result[n]['media']\n",
    "    datetime = result[n]['datetime']\n",
    "    # print(f\"Result {n}:\")\n",
    "    # print(title)\n",
    "    # print(desc)\n",
    "    # print(link)\n",
    "    # print(media)\n",
    "\n",
    "    # 資料清洗預處理\n",
    "\n",
    "    if title:\n",
    "        title_tokens.append(title.strip())\n",
    "    else:\n",
    "        print(f\"There is no title at {n} news list\")\n",
    "    if desc:\n",
    "        desc_tokens.append(desc.strip())\n",
    "    else:\n",
    "        print(f\"There is no description at {n} news list\")\n",
    "    if link:\n",
    "        link_tokens.append(link.strip())\n",
    "    else:\n",
    "        print(f\"There is no link at {n} news list\")\n",
    "    if media:\n",
    "        media_tokens.append(media.strip())\n",
    "    else:\n",
    "        print(f\"There is no media at {n} news list\")\n",
    "    if datetime:\n",
    "        datetime_tokens.append(datetime)\n",
    "    else:\n",
    "        print(f\"There is no datetime at {n} news list\")\n",
    "        \n",
    "    \n",
    "print(title_tokens)\n",
    "print(len(title_tokens))\n",
    "print(len(desc_tokens))\n",
    "print(len(media_tokens))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 將資料儲存至 CSV 檔案格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-08T09:51:03.511313Z",
     "start_time": "2024-09-08T09:50:58.674234Z"
    }
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from GoogleNews import GoogleNews\n",
    "googlenews = GoogleNews()\n",
    "googlenews.clear()\n",
    "time.sleep(2)\n",
    "googlenews.search('颱風假')\n",
    "googlenews.page_at(2)\n",
    "result_search_news = googlenews.result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-09-08T11:04:29.533447Z",
     "start_time": "2024-09-08T11:04:29.529446Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10\n",
      "{'title': 'Typhoon Krathon expected to weaken after landfall｜Taiwan News', 'media': '民視新聞網', 'date': '3 hours ago', 'datetime': datetime.datetime(2024, 10, 3, 0, 37, 19, 481100), 'desc': \"Typhoon Krathon continues to approach Taiwan after losing some strength in the early hours of Wednesday. Meteorologists say the storm's predicted path has...\", 'link': 'https://www.ftvnews.com.tw/video/detail/njjzN05zbGI&ved=2ahUKEwiRx6KHufCIAxXGma8BHZXwAEgQxfQBegQICBAC&usg=AOvVaw2bGdaT8yyWHomjAWQs4Fkm', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}\n",
      "2024-10-03 00:37:19.481100\n",
      "<class 'datetime.datetime'>\n",
      "{'title': '《TAIPEI TIMES》NSTC planning southern Taiwan technology center', 'media': '自由時報', 'date': '1 month ago', 'datetime': datetime.datetime(2024, 9, 3, 3, 37, 19, 524859), 'desc': \"TECHNOLOGICAL HUB:Tainan's Shalun Township would serve as the nation's development center for the artificial intelligence industry, an NSTC vice minister...\", 'link': 'https://news.ltn.com.tw/news/focus/breakingnews/4777510&ved=2ahUKEwiRx6KHufCIAxXGma8BHZXwAEgQxfQBegQIBRAC&usg=AOvVaw3tXkBlDMSm8t_q7ubtJOqJ', 'img': 'data:image/gif;base64,R0lGODlhAQABAIAAAP///////yH5BAEKAAEALAAAAAABAAEAAAICTAEAOw=='}\n"
     ]
    }
   ],
   "source": [
    "# result_search_news\n",
    "print(len(result_search_news))\n",
    "print(result_search_news[0])\n",
    "print(result_search_news[0]['datetime'])\n",
    "print(type(result_search_news[0]['datetime']))\n",
    "print(result_search_news[9])\n",
    "# 中文搜尋無法取得 datetime 資料"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has been saved to ./csv_output/news_data_20241003_033719.csv\n"
     ]
    }
   ],
   "source": [
    "import csv\n",
    "import os\n",
    "from datetime import datetime\n",
    "\n",
    "# 確保輸出目錄存在\n",
    "output_dir = \"./csv_output\"\n",
    "os.makedirs(output_dir, exist_ok=True)\n",
    "\n",
    "# 生成檔案名稱，包含當前日期和時間\n",
    "current_time = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n",
    "filename = f\"news_data_{current_time}.csv\"\n",
    "filepath = os.path.join(output_dir, filename)\n",
    "\n",
    "# 將結果寫入 CSV 文件\n",
    "with open(filepath, 'w', newline='', encoding='utf-8') as file:\n",
    "    writer = csv.DictWriter(file, fieldnames=['title', 'desc', 'date', 'datetime', 'link', 'img', 'media', 'site'])\n",
    "    writer.writeheader()\n",
    "    for item in result_search_news:\n",
    "        writer.writerow(item)\n",
    "\n",
    "print(f\"Data has been saved to {filepath}\")\n",
    "\n",
    "# 如果之後需要讀取數據，可以使用以下代碼：\n",
    "# import pandas as pd\n",
    "# df = pd.read_csv(filepath)\n",
    "# 然後可以使用 df['column_name'] 來訪問特定列的數據"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 資料清洗優化"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 增加停用詞\n",
    "\n",
    "優點：可自行定義停用詞\n",
    "缺點：無法涵蓋所有情況\n",
    "\n",
    "## 去除英文數字\n",
    "\n",
    "AIT關切柯文哲案 ====> 標題本身內有英文\n",
    "2024-09-15Ep.204@TheStormMedia ====> 媒體名稱日期\n",
    "\n",
    "優點：過濾掉日期，媒體英文名稱\n",
    "缺點：當事件簡稱常用英文來代表，會被過濾掉\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 資料呈現方式\n",
    "\n",
    "思路：\n",
    "\n",
    "0. 媒體來源\n",
    "   - 圓餅圖呈現媒體來源類型 (左派右派)\n",
    "   - 條狀圖呈現媒體來源數量 (數量多寡)\n",
    "  \n",
    "1. 字詞頻率分析\n",
    "   - 對中文內容進行斷詞\n",
    "   - 停用詞過濾\n",
    "   - 按頻率高低排序\n",
    "   - 使用 jieba 的算法權重顯示關鍵字\n",
    "\n",
    "2. 討論熱度推論\n",
    "   - 使用 pandas 處理時間數據\n",
    "   - 分析 N 天內每天的新聞數量\n",
    "   - 基於關鍵字搜索結果,可能不具代表性\n",
    "\n",
    "3. 文字雲視覺化\n",
    "   - 自定義遮罩圖\n",
    "   - 根據詞頻調整字體大小\n",
    "   - 使用 wordcloud 生成文字雲\n",
    "\n",
    "4. 部署至平台\n",
    "   - 選項 A: 使用 IFTTT 發送到 Line (單次執行)\n",
    "   - 選項 B: 使用 discord.py 創建 Discord 機器人互動呈現\n",
    "   - 選項 C: 使用 Streamlit 建立網頁前端\n",
    "   - 選項 D: 串接 Siri 捷徑 朗讀新聞 (單次執行)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自己練習的區域：\n",
    "\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "每日新聞數量：\n",
      "date\n",
      "2024-09-03    1\n",
      "2024-09-04    0\n",
      "2024-09-05    0\n",
      "2024-09-06    0\n",
      "2024-09-07    0\n",
      "2024-09-08    0\n",
      "2024-09-09    0\n",
      "2024-09-10    0\n",
      "2024-09-11    0\n",
      "2024-09-12    0\n",
      "2024-09-13    0\n",
      "2024-09-14    0\n",
      "2024-09-15    0\n",
      "2024-09-16    0\n",
      "2024-09-17    0\n",
      "2024-09-18    0\n",
      "2024-09-19    1\n",
      "2024-09-20    0\n",
      "2024-09-21    0\n",
      "2024-09-22    0\n",
      "2024-09-23    0\n",
      "2024-09-24    0\n",
      "2024-09-25    0\n",
      "2024-09-26    1\n",
      "2024-09-27    0\n",
      "2024-09-28    3\n",
      "2024-09-29    0\n",
      "2024-09-30    0\n",
      "2024-10-01    0\n",
      "2024-10-02    0\n",
      "2024-10-03    4\n",
      "Freq: D, dtype: int64\n",
      "\n",
      "新聞最多的日期是 2024-10-03，共有 4 條新聞。\n",
      "\n",
      "DataFrame 預覽：\n",
      "                                                                        title  \\\n",
      "date                                                                            \n",
      "2024-09-03 03:37:19.524859  《TAIPEI TIMES》NSTC planning southern Taiwan te...   \n",
      "2024-09-19 03:37:19.517296          Taipei a haven for super rich｜Taiwan News   \n",
      "2024-09-26 03:37:19.510964  NTU alumni card offers various discounts｜Taiwa...   \n",
      "2024-09-28 03:37:19.492327  Taiwan and Palau presidents witness signing of...   \n",
      "2024-09-28 03:37:19.498756  National Palace Museum exhibition showcases fi...   \n",
      "\n",
      "                            media  \n",
      "date                               \n",
      "2024-09-03 03:37:19.524859   自由時報  \n",
      "2024-09-19 03:37:19.517296  民視新聞網  \n",
      "2024-09-26 03:37:19.510964  民視新聞網  \n",
      "2024-09-28 03:37:19.492327  民視新聞網  \n",
      "2024-09-28 03:37:19.498756  民視新聞網  \n"
     ]
    }
   ],
   "source": [
    "# AI 提供的解法 只有英文搜尋會回傳 datetime \n",
    "\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "\n",
    "# 假設 result 是您之前獲得的新聞結果列表\n",
    "\n",
    "# 創建一個空的列表來存儲處理後的數據\n",
    "data = []\n",
    "\n",
    "for item in result_search_news:\n",
    "    # 檢查 datetime 是否存在且不為空\n",
    "    if item['datetime'] and not pd.isna(item['datetime']):\n",
    "        # 如果 datetime 已經是 datetime 對象，直接使用\n",
    "        if isinstance(item['datetime'], datetime):\n",
    "            date = item['datetime']\n",
    "        else:\n",
    "            # 否則，嘗試將字符串轉換為 datetime 對象\n",
    "            try:\n",
    "                date = pd.to_datetime(item['datetime'])\n",
    "            except:\n",
    "                # 如果轉換失敗，跳過這條記錄\n",
    "                continue\n",
    "        \n",
    "        # 將需要的數據添加到列表中\n",
    "        data.append({\n",
    "            'date': date,\n",
    "            'title': item['title'],\n",
    "            'media': item['media']\n",
    "        })\n",
    "\n",
    "# 創建 DataFrame\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "# 將 date 列設置為索引\n",
    "df.set_index('date', inplace=True)\n",
    "\n",
    "# 按日期排序\n",
    "df.sort_index(inplace=True)\n",
    "\n",
    "# 統計每天的新聞數量\n",
    "daily_news_count = df.resample('D').size()\n",
    "\n",
    "# 顯示結果\n",
    "print(\"每日新聞數量：\")\n",
    "print(daily_news_count)\n",
    "\n",
    "# 找出新聞最多的日期\n",
    "max_news_date = daily_news_count.idxmax()\n",
    "max_news_count = daily_news_count.max()\n",
    "\n",
    "print(f\"\\n新聞最多的日期是 {max_news_date.date()}，共有 {max_news_count} 條新聞。\")\n",
    "\n",
    "# 顯示DataFrame的前幾行，以檢查數據\n",
    "print(\"\\nDataFrame 預覽：\")\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---> 短時間內統計哪天最多似乎沒有太大意義 \n",
    "\n",
    "---> 新聞網址再拿來讀取內文文字內容 p or h1 ~ h5 \n",
    "\n",
    "----> 串接 Siri 捷徑 朗讀新聞 (單次執行) 要怎麼傳遞到手機上？"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}