![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Company Info with `llama-index-tools-scrapegraphai`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "source": [
+ ""
+ ],
+ "metadata": {
+ "id": "RaHnNF9bYX8N"
+ }
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index-tools-scrapegraphai"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "sffqFG2EJ8bI"
+ },
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | company_name | \n", + "description | \n", + "logo | \n", + "contact_emails | \n", + "privacy_policy | \n", + "terms_of_service | \n", + "api_status | \n", + "github | \n", + "||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "ScrapeGraphAI | \n", + "ScrapeGraphAI is a powerful AI scraping API de... | \n", + "https://scrapegraphai.com/images/scrapegraphai... | \n", + "contact@scrapegraphai.com | \n", + "https://scrapegraphai.com/privacy | \n", + "https://scrapegraphai.com/terms | \n", + "https://scrapegraphapi.openstatus.dev | \n", + "https://www.linkedin.com/company/101881123 | \n", + "https://x.com/scrapegraphai | \n", + "https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n", + "
| \n", + " | name | \n", + "role | \n", + "|
|---|---|---|---|
| 0 | \n", + "Marco Perini | \n", + "Founder & Technical Lead | \n", + "https://www.linkedin.com/in/perinim/ | \n", + "
| 1 | \n", + "Marco Vinciguerra | \n", + "Founder & Software Engineer | \n", + "https://www.linkedin.com/in/marco-vinciguerra-... | \n", + "
| 2 | \n", + "Lorenzo Padoan | \n", + "Founder & Product Engineer | \n", + "https://www.linkedin.com/in/lorenzo-padoan-452... | \n", + "
| \n", + " | tier | \n", + "price | \n", + "credits | \n", + "
|---|---|---|---|
| 0 | \n", + "Free | \n", + "$0 | \n", + "100 | \n", + "
| 1 | \n", + "Starter | \n", + "$20/month | \n", + "5000 | \n", + "
| 2 | \n", + "Growth | \n", + "$100/month | \n", + "40000 | \n", + "
| 3 | \n", + "Pro | \n", + "$500/month | \n", + "250000 | \n", + "
| \n", + " | partner | \n", + "
|---|---|
| 0 | \n", + "PostHog | \n", + "
| 1 | \n", + "AWS | \n", + "
| 2 | \n", + "NVIDIA | \n", + "
| 3 | \n", + "JinaAI | \n", + "
| 4 | \n", + "DagWorks | \n", + "
| 5 | \n", + "Browserbase | \n", + "
| 6 | \n", + "ScrapeDo | \n", + "
| 7 | \n", + "HackerNews | \n", + "
| 8 | \n", + "Medium | \n", + "
| 9 | \n", + "HackADay | \n", + "
\n",
+ " ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",)
\n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Company Info with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"3Q5VM3SsRlxO"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734532300517,"user_tz":-60,"elapsed":6877,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"f6b837cd-0f00-49cc-cb6f-f2bca57544f5"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | company_name | \n","description | \n","logo | \n","contact_emails | \n","privacy_policy | \n","terms_of_service | \n","api_status | \n","github | \n","||
|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","ScrapeGraphAI | \n","ScrapeGraphAI is a powerful AI scraping API de... | \n","https://scrapegraphai.com/images/scrapegraphai... | \n","contact@scrapegraphai.com | \n","https://scrapegraphai.com/privacy | \n","https://scrapegraphai.com/terms | \n","https://scrapegraphapi.openstatus.dev | \n","https://www.linkedin.com/company/101881123 | \n","https://x.com/scrapegraphai | \n","https://github.com/ScrapeGraphAI/Scrapegraph-ai | \n","
| \n"," | name | \n","role | \n","|
|---|---|---|---|
| 0 | \n","Marco Perini | \n","Founder & Technical Lead | \n","https://www.linkedin.com/in/perinim/ | \n","
| 1 | \n","Marco Vinciguerra | \n","Founder & Software Engineer | \n","https://www.linkedin.com/in/marco-vinciguerra-... | \n","
| 2 | \n","Lorenzo Padoan | \n","Founder & Product Engineer | \n","https://www.linkedin.com/in/lorenzo-padoan-452... | \n","
| \n"," | tier | \n","price | \n","credits | \n","
|---|---|---|---|
| 0 | \n","Free | \n","$0 | \n","100 | \n","
| 1 | \n","Starter | \n","$20/month | \n","5000 | \n","
| 2 | \n","Growth | \n","$100/month | \n","40000 | \n","
| 3 | \n","Pro | \n","$500/month | \n","250000 | \n","
| \n"," | partner | \n","
|---|---|
| 0 | \n","PostHog | \n","
| 1 | \n","AWS | \n","
| 2 | \n","NVIDIA | \n","
| 3 | \n","JinaAI | \n","
| 4 | \n","DagWorks | \n","
| 5 | \n","Browserbase | \n","
| 6 | \n","ScrapeDo | \n","
| 7 | \n","HackerNews | \n","
| 8 | \n","Medium | \n","
| 9 | \n","HackADay | \n","
\n"," ![\"ScrapeGraph](\"https://raw.githubusercontent.com/VinciGit00/Scrapegraph-ai/main/docs/assets/api-banner.png\"\n",")
\n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Github Trending Repositories with llama index and scrapegraphai's APIs\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IhozYNwsgJzt"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index\n",
+ "!pip install llama-index-tools-scrapegraphai\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "07af4bbe-c226-4fb3-8f68-7ccd429f59cc"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "SGAI_API_KEY not found in environment.\n",
+ "SGAI_API_KEY has been set in the environment.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "# Check if the API key is already set in the environment\n",
+ "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
+ "\n",
+ "if sgai_api_key:\n",
+ " print(\"SGAI_API_KEY found in environment.\")\n",
+ "else:\n",
+ " print(\"SGAI_API_KEY not found in environment.\")\n",
+ " # Prompt the user to input the API key securely (hidden input)\n",
+ " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
+ " if sgai_api_key:\n",
+ " # Set the API key in the environment\n",
+ " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
+ " print(\"SGAI_API_KEY has been set in the environment.\")\n",
+ " else:\n",
+ " print(\"No API key entered. Please set the API key to continue.\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | name | \n", + "description | \n", + "stars | \n", + "forks | \n", + "today_stars | \n", + "language | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "XiaoMi/ha_xiaomi_home | \n", + "Xiaomi Home Integration for Home Assistant | \n", + "11097 | \n", + "472 | \n", + "3023 | \n", + "Python | \n", + "
| 1 | \n", + "comet-ml/opik | \n", + "Open-source end-to-end LLM Development Platform | \n", + "2741 | \n", + "169 | \n", + "91 | \n", + "Java | \n", + "
| 2 | \n", + "EbookFoundation/free-programming-books | \n", + "📚 Freely available programming books | \n", + "341919 | \n", + "62038 | \n", + "225 | \n", + "HTML | \n", + "
| 3 | \n", + "konfig-dev/konfig | \n", + "Sunset as of December 2024 | \n", + "689 | \n", + "192 | \n", + "224 | \n", + "TypeScript | \n", + "
| 4 | \n", + "anoma/anoma | \n", + "Reference implementation of Anoma | \n", + "9451 | \n", + "452 | \n", + "4129 | \n", + "Elixir | \n", + "
| 5 | \n", + "stripe/stripe-ios | \n", + "Stripe iOS SDK | \n", + "2292 | \n", + "1004 | \n", + "49 | \n", + "Swift | \n", + "
| 6 | \n", + "Guovin/iptv-api | \n", + "IPTV live TV source update tool | \n", + "9385 | \n", + "2010 | \n", + "91 | \n", + "Python | \n", + "
| 7 | \n", + "facebookresearch/AnimatedDrawings | \n", + "Code to accompany \"A Method for Animating Chil... | \n", + "11473 | \n", + "988 | \n", + "398 | \n", + "Python | \n", + "
| 8 | \n", + "apache/airflow | \n", + "Apache Airflow - A platform to programmaticall... | \n", + "37690 | \n", + "14411 | \n", + "25 | \n", + "Python | \n", + "
| 9 | \n", + "seleniumbase/SeleniumBase | \n", + "Python APIs for web automation, testing, and b... | \n", + "6646 | \n", + "1028 | \n", + "624 | \n", + "Python | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Github Trending Repositories with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"d7Zro0xiuo-l"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734439787062,"user_tz":-60,"elapsed":5826,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"ab74193e-e746-4de6-d65d-33a2a26b5d86"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | name | \n","description | \n","stars | \n","forks | \n","today_stars | \n","language | \n","
|---|---|---|---|---|---|---|
| 0 | \n","Byaidu/PDFMathTranslate | \n","PDF scientific paper translation with preserve... | \n","8902 | \n","633 | \n","816 | \n","Python | \n","
| 1 | \n","bigskysoftware/htmx | \n","htmx - high power tools for HTML | \n","39143 | \n","1324 | \n","186 | \n","JavaScript | \n","
| 2 | \n","commaai/openpilot | \n","openpilot is an operating system for robotics.... | \n","50945 | \n","9206 | \n","132 | \n","Python | \n","
| 3 | \n","google-gemini/cookbook | \n","Examples and guides for using the Gemini API | \n","8108 | \n","1011 | \n","1221 | \n","Jupyter Notebook | \n","
| 4 | \n","stripe/stripe-ios | \n","Stripe iOS SDK | \n","2179 | \n","994 | \n","19 | \n","Swift | \n","
| 5 | \n","RIOT-OS/RIOT | \n","RIOT - The friendly OS for IoT | \n","5234 | \n","2017 | \n","168 | \n","C | \n","
| 6 | \n","zju3dv/EasyVolcap | \n","EasyVolcap: Accelerating Neural Volumetric Vid... | \n","802 | \n","52 | \n","30 | \n","Python | \n","
| 7 | \n","TEN-framework/TEN-Agent | \n","TEN Agent is a conversational AI powered by TE... | \n","3245 | \n","313 | \n","296 | \n","Python | \n","
| 8 | \n","DS4SD/docling | \n","Get your documents ready for gen AI | \n","15201 | \n","774 | \n","281 | \n","Python | \n","
| 9 | \n","Guovin/iptv-api | \n","📺IPTV电视直播源更新工具：✨央视频、📱卫视、☕各省份地方台、🌏港·澳·台、🎥电影、🎮游戏... | \n","9046 | \n","1938 | \n","101 | \n","Python | \n","
| 10 | \n","fatedier/frp | \n","A fast reverse proxy to help you expose a loca... | \n","87828 | \n","13502 | \n","64 | \n","Go | \n","
| 11 | \n","facebookresearch/AnimatedDrawings | \n","Code to accompany \"A Method for Animating Chil... | \n","10766 | \n","955 | \n","38 | \n","Python | \n","
| 12 | \n","gorilla/websocket | \n","Package gorilla/websocket is a fast, well-test... | \n","22633 | \n","3495 | \n","13 | \n","Go | \n","
| 13 | \n","DefiLlama/chainlist | \n","NA | \n","2368 | \n","2476 | \n","5 | \n","JavaScript | \n","
| 14 | \n","open-telemetry/opentelemetry-collector | \n","OpenTelemetry Collector | \n","4570 | \n","1497 | \n","4 | \n","Go | \n","
| 15 | \n","RocketChat/Rocket.Chat | \n","The communications platform that puts data pro... | \n","41169 | \n","10877 | \n","73 | \n","TypeScript | \n","
| 16 | \n","langgenius/dify | \n","Dify is an open-source LLM app development pla... | \n","54976 | \n","8083 | \n","127 | \n","TypeScript | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Houses Listing on Zillow with llama-index and ScrapegraphAI APIs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5cVkde_LpVkF"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index\n",
+ "!pip install llama-index-tools-scrapegraphai"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "sffqFG2EJ8bI",
+ "outputId": "c588274d-64ce-4d13-f12d-0458d9c4839d"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "SGAI_API_KEY not found in environment.\n",
+ "SGAI_API_KEY has been set in the environment.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from getpass import getpass\n",
+ "\n",
+ "# Check if the API key is already set in the environment\n",
+ "sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n",
+ "\n",
+ "if sgai_api_key:\n",
+ " print(\"SGAI_API_KEY found in environment.\")\n",
+ "else:\n",
+ " print(\"SGAI_API_KEY not found in environment.\")\n",
+ " # Prompt the user to input the API key securely (hidden input)\n",
+ " sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n",
+ " if sgai_api_key:\n",
+ " # Set the API key in the environment\n",
+ " os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n",
+ " print(\"SGAI_API_KEY has been set in the environment.\")\n",
+ " else:\n",
+ " print(\"No API key entered. Please set the API key to continue.\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | price | \n", + "bedrooms | \n", + "bathrooms | \n", + "square_feet | \n", + "address | \n", + "city | \n", + "state | \n", + "zip_code | \n", + "tags | \n", + "agent_name | \n", + "agency | \n", + "
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", + "449000 | \n", + "3 | \n", + "3 | \n", + "2166 | \n", + "3229 Jennings St | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[] | \n", + "Michelle K. Pender | \n", + "ENGEL & VOELKERS SAN FRANCISCO | \n", + "
| 1 | \n", + "950000 | \n", + "2 | \n", + "2 | \n", + "1686 | \n", + "401 Huron Ave | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Cozy fireplace] | \n", + "Allison Chapleau | \n", + "COMPASS | \n", + "
| 2 | \n", + "207555 | \n", + "1 | \n", + "1 | \n", + "1593 | \n", + "2040 Fell St APT 10 | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[] | \n", + "Trista Elizabeth Bernasconi | \n", + "COMPASS | \n", + "
| 3 | \n", + "795000 | \n", + "4 | \n", + "2 | \n", + "2000 | \n", + "515 Athens St | \n", + "San Francisco | \n", + "CA | \n", + "94112 | \n", + "[Level fenced rear yard] | \n", + "Darin J. Holwitz | \n", + "COMPASS | \n", + "
| 4 | \n", + "599000 | \n", + "1 | \n", + "1 | \n", + "0 | \n", + "380 Dolores St #6 | \n", + "San Francisco | \n", + "CA | \n", + "94114 | \n", + "[] | \n", + "Melody A. Hultgren | \n", + "NOVA REAL ESTATE | \n", + "
| 5 | \n", + "875000 | \n", + "2 | \n", + "2 | \n", + "907 | \n", + "426 Fillmore St #A | \n", + "San Francisco | \n", + "CA | \n", + "94117 | \n", + "[Sleek finishes] | \n", + "NA | \n", + "NA | \n", + "
| 6 | \n", + "335512 | \n", + "2 | \n", + "2 | \n", + "886 | \n", + "1688 Pine St #101 | \n", + "San Francisco | \n", + "CA | \n", + "94109 | \n", + "[] | \n", + "Trista Elizabeth Bernasconi | \n", + "COMPASS | \n", + "
| 7 | \n", + "899000 | \n", + "4 | \n", + "2 | \n", + "1680 | \n", + "351 Chenery St | \n", + "San Francisco | \n", + "CA | \n", + "94131 | \n", + "[South-facing panoramic views] | \n", + "Easton S. Thodos | \n", + "THESEUS REAL ESTATE | \n", + "
| 8 | \n", + "155659 | \n", + "0 | \n", + "1 | \n", + "514 | \n", + "52 Kirkwood Ave #203 | \n", + "San Francisco | \n", + "CA | \n", + "94124 | \n", + "[Modern cabinetry] | \n", + "Lynn Anne Bell | \n", + "CHRISTIE'S INT'L R.E. SF | \n", + "
\n",
+ " \n",
+ "
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Houses Listing with Official Scrapegraph SDK"]},{"cell_type":"markdown","metadata":{"id":"8vZBkAWLq9C1"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` API key"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":null,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","outputId":"18dfce64-db37-4825-d316-fabd064100d0"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY found in environment.\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | price | \n","bedrooms | \n","bathrooms | \n","square_feet | \n","address | \n","city | \n","state | \n","zip_code | \n","tags | \n","agent_name | \n","agency | \n","
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | \n","549000 | \n","1 | \n","1 | \n","477 | \n","380 14th St Unit 405 | \n","San Francisco | \n","CA | \n","94103 | \n","[New construction] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 1 | \n","1799000 | \n","4 | \n","2 | \n","2735 | \n","123 Grattan St | \n","San Francisco | \n","CA | \n","94117 | \n","[] | \n","Sean Engmann | \n","eXp Realty of Northern CA Inc. | \n","
| 2 | \n","1995000 | \n","7 | \n","3 | \n","3330 | \n","1590 Washington St | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 3 | \n","549000 | \n","0 | \n","1 | \n","477 | \n","240 Lombard St Unit 835 | \n","San Francisco | \n","CA | \n","94111 | \n","[] | \n","Tim Gullicksen | \n","Corcoran Icon Properties | \n","
| 4 | \n","5495000 | \n","10 | \n","7 | \n","6505 | \n","1057 Steiner St | \n","San Francisco | \n","CA | \n","94115 | \n","[] | \n","Bonnie Spindler | \n","Corcoran Icon Properties | \n","
| 5 | \n","925000 | \n","2 | \n","1 | \n","779 | \n","2 Fallon Place Unit 57 | \n","San Francisco | \n","CA | \n","94133 | \n","[] | \n","Eddie O'Sullivan | \n","Elevation Real Estate | \n","
| 6 | \n","898000 | \n","2 | \n","2 | \n","1175 | \n","5160 Diamond Heights Blvd Unit 208C | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Joe Polyak | \n","Rise Homes | \n","
| 7 | \n","1700000 | \n","4 | \n","2 | \n","1950 | \n","1351 26th Ave | \n","San Francisco | \n","CA | \n","94122 | \n","[] | \n","Glenda Queensbury | \n","Referral Realty-BV | \n","
| 8 | \n","1899000 | \n","3 | \n","2 | \n","1560 | \n","340 Yerba Buena Ave | \n","San Francisco | \n","CA | \n","94127 | \n","[] | \n","Jeannie Anderson | \n","Coldwell Banker Realty | \n","
| 9 | \n","850000 | \n","2 | \n","2 | \n","1055 | \n","588 Minna Unit 604 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Mohamed Lakdawala | \n","Remax Prestigious Properties | \n","
| 10 | \n","1990000 | \n","3 | \n","1 | \n","1280 | \n","1450 Diamond St | \n","San Francisco | \n","CA | \n","94131 | \n","[] | \n","Mary Anne Villamil | \n","Kinetic Real Estate | \n","
| 11 | \n","849000 | \n","1 | \n","1 | \n","855 | \n","81 Lansing St Unit 401 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Kristen Haenggi | \n","Compass | \n","
| 12 | \n","1080000 | \n","2 | \n","2 | \n","936 | \n","451 Kansas St Unit 466 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Maureen DeBoer | \n","LKJ Realty | \n","
| 13 | \n","1499000 | \n","4 | \n","2 | \n","2145 | \n","486 Yale St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Alicia Atienza | \n","Statewide Realty | \n","
| 14 | \n","1140000 | \n","2 | \n","2 | \n","998 | \n","588 Minna Unit 801 | \n","San Francisco | \n","CA | \n","94103 | \n","[] | \n","Milan Jezdimirovic | \n","Compass | \n","
| 15 | \n","1988000 | \n","2 | \n","1 | \n","3800 | \n","183 19th Ave | \n","San Francisco | \n","CA | \n","94121 | \n","[Amazing Property, Marina Style, Needs TLC] | \n","Leo Cheung | \n","eXp Realty of California, Inc | \n","
| 16 | \n","1218000 | \n","2 | \n","2 | \n","1275 | \n","1998 Pacific Ave Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[Light-filled, Freshly painted, Walker's parad... | \n","Grace Sun | \n","Compass | \n","
| 17 | \n","895000 | \n","1 | \n","1 | \n","837 | \n","425 1st St Unit 2501 | \n","San Francisco | \n","CA | \n","94105 | \n","[Unobstructed bay bridge views, Open layout] | \n","Matt Fuller | \n","Jackson Fuller Real Estate | \n","
| 18 | \n","1499000 | \n","3 | \n","1 | \n","1500 | \n","Unlisted Address | \n","San Francisco | \n","CA | \n","NA | \n","[Contractor's Special, Fixer-upper] | \n","Jaymee Faith Sagisi | \n","IMPACT | \n","
| 19 | \n","900000 | \n","1 | \n","1 | \n","930 | \n","1101 Green St Unit 302 | \n","San Francisco | \n","CA | \n","94109 | \n","[Historic Art Deco, Iconic views] | \n","NA | \n","NA | \n","
| 20 | \n","858000 | \n","1 | \n","1 | \n","1104 | \n","260 King St Unit 557 | \n","San Francisco | \n","CA | \n","94107 | \n","[] | \n","Miyuki Takami | \n","eXp Realty of California, Inc | \n","
| 21 | \n","945000 | \n","2 | \n","1 | \n","767 | \n","307 Page St Unit 1 | \n","San Francisco | \n","CA | \n","94102 | \n","[] | \n","NA | \n","NA | \n","
| 22 | \n","1099000 | \n","2 | \n","2 | \n","1330 | \n","1080 Sutter St Unit 202 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Annette Liberty | \n","Coldwell Banker Realty | \n","
| 23 | \n","950000 | \n","4 | \n","3 | \n","2090 | \n","3328 26th St Unit 3330 | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Isaac Munene | \n","Coldwell Banker Realty | \n","
| 24 | \n","1088000 | \n","2 | \n","2 | \n","1065 | \n","1776 Sacramento St Unit 503 | \n","San Francisco | \n","CA | \n","94109 | \n","[] | \n","Marilyn Becklehimer | \n","Dio Real Estate | \n","
| 25 | \n","1788888 | \n","4 | \n","3 | \n","1856 | \n","2317 15th St | \n","San Francisco | \n","CA | \n","94114 | \n","[] | \n","Joel Gile | \n","Sequoia Real Estate | \n","
| 26 | \n","1650000 | \n","3 | \n","2 | \n","1547 | \n","2475 47th Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Lucy Goldenshteyn | \n","Redfin | \n","
| 27 | \n","998000 | \n","2 | \n","2 | \n","1202 | \n","50 Lansing St Unit 201 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Tracey Broadman | \n","Vanguard Properties | \n","
| 28 | \n","1595000 | \n","3 | \n","5 | \n","1995 | \n","15 Joy St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Mike Stack | \n","Vanguard Properties | \n","
| 29 | \n","1028000 | \n","2 | \n","2 | \n","1065 | \n","50 Lansing St Unit 403 | \n","San Francisco | \n","CA | \n","94105 | \n","[] | \n","Robyn Kaufman | \n","Vivre Real Estate | \n","
| 30 | \n","999000 | \n","1 | \n","1 | \n","1021 | \n","338 Spear St Unit 6J | \n","San Francisco | \n","CA | \n","94105 | \n","[Spacious, Balcony, Bright courtyard views] | \n","Paul Hwang | \n","Skybox Realty | \n","
| 31 | \n","799800 | \n","2 | \n","2 | \n","1109 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 32 | \n","529880 | \n","1 | \n","1 | \n","740 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 33 | \n","489000 | \n","1 | \n","1 | \n","741 | \n","10 Innes Ct | \n","San Francisco | \n","CA | \n","94124 | \n","[New Construction] | \n","Lennar | \n","Lennar | \n","
| 34 | \n","1359000 | \n","4 | \n","2 | \n","1845 | \n","170 Thrift St | \n","San Francisco | \n","CA | \n","94112 | \n","[Updated, Single-family home] | \n","Cristal Wright | \n","Vanguard Properties | \n","
| 35 | \n","1295000 | \n","3 | \n","1 | \n","1214 | \n","1922 43rd Ave | \n","San Francisco | \n","CA | \n","94116 | \n","[] | \n","Mila Romprey | \n","Premier Realty Associates | \n","
| 36 | \n","1098000 | \n","3 | \n","1 | \n","1006 | \n","150 Putnam St | \n","San Francisco | \n","CA | \n","94110 | \n","[] | \n","Genie Mantzoros | \n","Epic Real Estate & Asso. Inc. | \n","
| 37 | \n","1189870 | \n","3 | \n","2 | \n","1436 | \n","327 Ordway St | \n","San Francisco | \n","CA | \n","94134 | \n","[] | \n","Shawn Zahraie | \n","Affinity Enterprises, Inc | \n","
| 38 | \n","899000 | \n","2 | \n","1 | \n","1118 | \n","272 Farallones St | \n","San Francisco | \n","CA | \n","94112 | \n","[] | \n","Janice Lee | \n","Coldwell Banker Realty | \n","
| 39 | \n","30000 | \n","0 | \n","0 | \n","0 | \n","0 Evans Ave | \n","San Francisco | \n","CA | \n","94124 | \n","[Land, 0.12 Acre, $251,467 per Acre] | \n","Heidy Carrera | \n","Berkshire Hathaway HomeService | \n","
\n"," \n","
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Wired Science News with `langchain-scrapegraph` and `langgraph`"]},{"cell_type":"markdown","source":[""],"metadata":{"id":"FxtXj1Qtx3zH"}},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"os_vm0MkIxr9","executionInfo":{"status":"ok","timestamp":1734786986883,"user_tz":-60,"elapsed":13752,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}}},"outputs":[],"source":["%%capture\n","!pip install langgraph langchain-scrapegraph langchain-openai"]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":2,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":67210,"status":"ok","timestamp":1734787067908,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"},"user_tz":-60},"id":"sffqFG2EJ8bI","outputId":"8c3dd34f-22d5-4557-c562-97728738726b"},"outputs":[{"name":"stdout","output_type":"stream","text":["Scrapegraph API key:\n","··········\n","OpenAI API key:\n","··········\n"]}],"source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")\n","\n","if not os.environ.get(\"OPENAI_API_KEY\"):\n"," os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API key:\\n\")"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | title | \n","link | \n","author | \n","
|---|---|---|---|
| 0 | \n","December Wildfires Are Now a Thing | \n","https://www.wired.com/story/december-wildfires... | \n","Kylie Mohr | \n","
| 1 | \n","How to Manage Food Anxiety Over the Holidays | \n","https://www.wired.com/story/how-to-cope-with-f... | \n","Alison Fixsen | \n","
| 2 | \n","A Spacecraft Is About to Fly Into the Sun’s At... | \n","https://www.wired.com/story/parker-solar-probe... | \n","Eric Berger, Ars Technica | \n","
| 3 | \n","To Improve Your Gut Microbiome, Spend More Tim... | \n","https://www.wired.com/story/to-improve-your-gu... | \n","Kathy Willis | \n","
| 4 | \n","This Tropical Virus Is Spreading Out of the Am... | \n","https://www.wired.com/story/this-tropical-viru... | \n","Geraldine Castro | \n","
| 5 | \n","CDC Confirms First US Case of Severe Bird Flu | \n","https://www.wired.com/story/cdc-confirms-first... | \n","Emily Mullin | \n","
| 6 | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 7 | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
| 10 | \n","Environmental Sensing Is Here, Tracking Everyt... | \n","https://www.wired.com/story/environmental-sens... | \n","Sabrina Weiss | \n","
| 11 | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 12 | \n","Climate Change Is Destroying Monarch Butterfli... | \n","https://www.wired.com/story/global-warming-thr... | \n","Andrea J. Arratibel | \n","
| 13 | \n","More Humanitarian Organizations Will Harness A... | \n","https://www.wired.com/story/humanitarian-organ... | \n","David Miliband | \n","
| 14 | \n","Chocolate Has a Sustainability Problem. Scienc... | \n","https://www.wired.com/story/chocolate-has-a-su... | \n","Eve Thomas | \n","
| 15 | \n","We’ve Never Been Closer to Finding Life Outsid... | \n","https://www.wired.com/story/james-webb-space-t... | \n","Lisa Kaltenegger | \n","
| 16 | \n","The End Is Near for NASA’s Voyager Probes | \n","https://www.wired.com/story/the-end-is-near-fo... | \n","Luca Nardi | \n","
| 17 | \n","Why Can’t You Switch Seats in an Empty Airplane? | \n","https://www.wired.com/story/why-cant-you-switc... | \n","Rhett Allain | \n","
| 18 | \n","The Simple Math Behind Public Key Cryptography | \n","https://www.wired.com/story/how-public-key-cry... | \n","John Pavlus | \n","
| 19 | \n","Everyone Is Capable of Mathematical Thinking--... | \n","https://www.wired.com/story/everyone-is-capabl... | \n","Kelsey Houston-Edwards | \n","
| 20 | \n","The Physics of the Macy’s Thanksgiving Day Par... | \n","https://www.wired.com/story/the-physics-of-the... | \n","Rhett Allain | \n","
| 21 | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 22 | \n","Muscle Implants Could Allow Mind-Controlled Pr... | \n","https://www.wired.com/story/amputees-could-con... | \n","Emily Mullin | \n","
| 23 | \n","Combining AI and Crispr Will Be Transformational | \n","https://www.wired.com/story/combining-ai-and-c... | \n","Jennifer Doudna | \n","
| 24 | \n","Neuralink Plans to Test Whether Its Brain Impl... | \n","https://www.wired.com/story/neuralink-robotic-... | \n","Emily Mullin | \n","
| 25 | \n","Eight Scientists, a Billion Dollars, and the M... | \n","https://www.wired.com/story/aria-moonshot-darp... | \n","Matt Reynolds | \n","
| 26 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","NA | \n","
| 27 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","Carlton Reid | \n","
| 28 | \n","Meet the Next Generation of Doctors--and Their... | \n","https://www.wired.com/story/next-generation-do... | \n","Neha Mukherjee | \n","
| 29 | \n","AI Is Building Highly Effective Antibodies Tha... | \n","https://www.wired.com/story/labgenius-antibody... | \n","Amit Katwala | \n","
| 30 | \n","An Uncertain Future Requires Uncertain Predict... | \n","https://www.wired.com/story/embrace-uncertaint... | \n","David Spiegelhalter | \n","
| 31 | \n","These Rats Learned to Drive--and They Love It | \n","https://www.wired.com/story/these-rats-learned... | \n","Kelly Lambert | \n","
| 32 | \n","Scientists Are Unlocking the Secrets of Your ‗... | \n","https://www.wired.com/story/cerebellum-brain-m... | \n","R Douglas Fields | \n","
| 33 | \n","Meet the Designer Behind Neuralink’s Surgical ... | \n","https://www.wired.com/story/designer-behind-ne... | \n","Emily Mullin | \n","
| 34 | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 35 | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 36 | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 37 | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 38 | \n","Designer Babies Are Teenagers Now--and Some of... | \n","https://www.wired.com/story/your-next-job-desi... | \n","Emi Nietfeld | \n","
| 39 | \n","US Meat, Milk Prices Should Spike if Donald Tr... | \n","https://www.wired.com/story/us-meat-milk-price... | \n","Matt Reynolds | \n","
| 40 | \n","An Augmented Reality Program Can Help Patients... | \n","https://www.wired.com/story/lining-up-tech-to-... | \n","Grace Browne | \n","
| 41 | \n","Meet the Plant Hacker Creating Flowers Never S... | \n","https://www.wired.com/story/meet-the-plant-hac... | \n","Matt Reynolds | \n","
| 42 | \n","A Mysterious Respiratory Disease Has the Democ... | \n","https://www.wired.com/story/drc-mysterious-res... | \n","Marta Musso | \n","
| 43 | \n","Skip the Sea Kelp Supplements | \n","https://www.wired.com/story/pass-on-sea-kelp-s... | \n","Boutayna Chokrane | \n","
| 44 | \n","Why Soccer Players Are Training in the Dark | \n","https://www.wired.com/story/why-soccer-players... | \n","RM Clark | \n","
| 45 | \n","A Parasite That Eats Cattle Alive Is Creeping ... | \n","https://www.wired.com/story/a-parasite-that-ea... | \n","Geraldine Castro | \n","
| 46 | \n","Lasers Are Making It Easier to Find Buried Lan... | \n","https://www.wired.com/story/this-laser-system-... | \n","Ritsuko Kawai | \n","
| 47 | \n","Mark Cuban’s War on Drug Prices: ‖How Much Fuc... | \n","https://www.wired.com/story/big-interview-mark... | \n","Marah Eakin | \n","
| 48 | \n","Can Artificial Rain, Drones, or Satellites Cle... | \n","https://www.wired.com/story/artificial-rain-dr... | \n","Arunima Kar | \n","
| 49 | \n","These Stem Cell Treatments Are Worth Millions.... | \n","https://www.wired.com/story/stem-cells-cost-ri... | \n","Matt Reynolds | \n","
| 50 | \n","The Mystery of How Supermassive Black Holes Merge | \n","https://www.wired.com/story/how-do-merging-sup... | \n","Jonathan O’Callaghan | \n","
| 51 | \n","The $60 Billion Potential Hiding in Your Disca... | \n","https://www.wired.com/story/a-dollar60-billion... | \n","Vince Beiser | \n","
| 52 | \n","Tune In to the Healing Powers of a Decent Play... | \n","https://www.wired.com/story/music-therapy-heal... | \n","Daniel Levitin | \n","
| 53 | \n","Returning the Amazon Rainforest to Its True Ca... | \n","https://www.wired.com/story/amazon-rainforest-... | \n","Nemonte Nenquimo and Mitch Anderson | \n","
\n"," \n","
\n",""]},{"cell_type":"markdown","metadata":{"id":"jEkuKbcRrPcK"},"source":["## 🕷️ Extract Wired Science News with `langchain-scrapegraph` and `langgraph`"]},{"cell_type":"markdown","metadata":{"id":"FxtXj1Qtx3zH"},"source":[""]},{"cell_type":"markdown","metadata":{"id":"IzsyDXEWwPVt"},"source":["### 🔧 Install `dependencies`"]},{"cell_type":"code","execution_count":1,"metadata":{"executionInfo":{"elapsed":13752,"status":"ok","timestamp":1734786986883,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"},"user_tz":-60},"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install llama-index\n","!pip install llama-index-tools-scrapegraphai "]},{"cell_type":"markdown","metadata":{"id":"apBsL-L2KzM7"},"source":["### 🔑 Import `ScrapeGraph` and `OpenAI` API keys"]},{"cell_type":"markdown","metadata":{"id":"ol9gQbAFkh9b"},"source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"]},{"cell_type":"code","execution_count":1,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":67210,"status":"ok","timestamp":1734787067908,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"},"user_tz":-60},"id":"sffqFG2EJ8bI","outputId":"8c3dd34f-22d5-4557-c562-97728738726b"},"outputs":[{"name":"stdout","output_type":"stream","text":["SGAI_API_KEY not found in environment.\n","SGAI_API_KEY has been set in the environment.\n"]}],"source":["import os\n","from getpass import getpass\n","\n","# Check if the API key is already set in the environment\n","sgai_api_key = os.getenv(\"SGAI_API_KEY\")\n","\n","if sgai_api_key:\n"," print(\"SGAI_API_KEY found in environment.\")\n","else:\n"," print(\"SGAI_API_KEY not found in environment.\")\n"," # Prompt the user to input the API key securely (hidden input)\n"," sgai_api_key = getpass(\"Please enter your SGAI_API_KEY: \").strip()\n"," if sgai_api_key:\n"," # Set the API key in the environment\n"," os.environ[\"SGAI_API_KEY\"] = sgai_api_key\n"," print(\"SGAI_API_KEY has been set in the environment.\")\n"," else:\n"," print(\"No API key entered. Please set the API key to continue.\")\n"]},{"cell_type":"markdown","metadata":{"id":"jnqMB2-xVYQ7"},"source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"]},{"cell_type":"markdown","metadata":{"id":"VZvxbjfXvbgd"},"source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | title | \n","link | \n","author | \n","
|---|---|---|---|
| 0 | \n","December Wildfires Are Now a Thing | \n","https://www.wired.com/story/december-wildfires... | \n","Kylie Mohr | \n","
| 1 | \n","How to Manage Food Anxiety Over the Holidays | \n","https://www.wired.com/story/how-to-cope-with-f... | \n","Alison Fixsen | \n","
| 2 | \n","A Spacecraft Is About to Fly Into the Sun’s At... | \n","https://www.wired.com/story/parker-solar-probe... | \n","Eric Berger, Ars Technica | \n","
| 3 | \n","To Improve Your Gut Microbiome, Spend More Tim... | \n","https://www.wired.com/story/to-improve-your-gu... | \n","Kathy Willis | \n","
| 4 | \n","This Tropical Virus Is Spreading Out of the Am... | \n","https://www.wired.com/story/this-tropical-viru... | \n","Geraldine Castro | \n","
| 5 | \n","CDC Confirms First US Case of Severe Bird Flu | \n","https://www.wired.com/story/cdc-confirms-first... | \n","Emily Mullin | \n","
| 6 | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 7 | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
| 10 | \n","Environmental Sensing Is Here, Tracking Everyt... | \n","https://www.wired.com/story/environmental-sens... | \n","Sabrina Weiss | \n","
| 11 | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 12 | \n","Climate Change Is Destroying Monarch Butterfli... | \n","https://www.wired.com/story/global-warming-thr... | \n","Andrea J. Arratibel | \n","
| 13 | \n","More Humanitarian Organizations Will Harness A... | \n","https://www.wired.com/story/humanitarian-organ... | \n","David Miliband | \n","
| 14 | \n","Chocolate Has a Sustainability Problem. Scienc... | \n","https://www.wired.com/story/chocolate-has-a-su... | \n","Eve Thomas | \n","
| 15 | \n","We’ve Never Been Closer to Finding Life Outsid... | \n","https://www.wired.com/story/james-webb-space-t... | \n","Lisa Kaltenegger | \n","
| 16 | \n","The End Is Near for NASA’s Voyager Probes | \n","https://www.wired.com/story/the-end-is-near-fo... | \n","Luca Nardi | \n","
| 17 | \n","Why Can’t You Switch Seats in an Empty Airplane? | \n","https://www.wired.com/story/why-cant-you-switc... | \n","Rhett Allain | \n","
| 18 | \n","The Simple Math Behind Public Key Cryptography | \n","https://www.wired.com/story/how-public-key-cry... | \n","John Pavlus | \n","
| 19 | \n","Everyone Is Capable of Mathematical Thinking--... | \n","https://www.wired.com/story/everyone-is-capabl... | \n","Kelsey Houston-Edwards | \n","
| 20 | \n","The Physics of the Macy’s Thanksgiving Day Par... | \n","https://www.wired.com/story/the-physics-of-the... | \n","Rhett Allain | \n","
| 21 | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 22 | \n","Muscle Implants Could Allow Mind-Controlled Pr... | \n","https://www.wired.com/story/amputees-could-con... | \n","Emily Mullin | \n","
| 23 | \n","Combining AI and Crispr Will Be Transformational | \n","https://www.wired.com/story/combining-ai-and-c... | \n","Jennifer Doudna | \n","
| 24 | \n","Neuralink Plans to Test Whether Its Brain Impl... | \n","https://www.wired.com/story/neuralink-robotic-... | \n","Emily Mullin | \n","
| 25 | \n","Eight Scientists, a Billion Dollars, and the M... | \n","https://www.wired.com/story/aria-moonshot-darp... | \n","Matt Reynolds | \n","
| 26 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","NA | \n","
| 27 | \n","The Atlas Robot Is Dead. Long Live the Atlas R... | \n","https://www.wired.com/story/the-atlas-robot-is... | \n","Carlton Reid | \n","
| 28 | \n","Meet the Next Generation of Doctors--and Their... | \n","https://www.wired.com/story/next-generation-do... | \n","Neha Mukherjee | \n","
| 29 | \n","AI Is Building Highly Effective Antibodies Tha... | \n","https://www.wired.com/story/labgenius-antibody... | \n","Amit Katwala | \n","
| 30 | \n","An Uncertain Future Requires Uncertain Predict... | \n","https://www.wired.com/story/embrace-uncertaint... | \n","David Spiegelhalter | \n","
| 31 | \n","These Rats Learned to Drive--and They Love It | \n","https://www.wired.com/story/these-rats-learned... | \n","Kelly Lambert | \n","
| 32 | \n","Scientists Are Unlocking the Secrets of Your ‗... | \n","https://www.wired.com/story/cerebellum-brain-m... | \n","R Douglas Fields | \n","
| 33 | \n","Meet the Designer Behind Neuralink’s Surgical ... | \n","https://www.wired.com/story/designer-behind-ne... | \n","Emily Mullin | \n","
| 34 | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 35 | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 36 | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 37 | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 38 | \n","Designer Babies Are Teenagers Now--and Some of... | \n","https://www.wired.com/story/your-next-job-desi... | \n","Emi Nietfeld | \n","
| 39 | \n","US Meat, Milk Prices Should Spike if Donald Tr... | \n","https://www.wired.com/story/us-meat-milk-price... | \n","Matt Reynolds | \n","
| 40 | \n","An Augmented Reality Program Can Help Patients... | \n","https://www.wired.com/story/lining-up-tech-to-... | \n","Grace Browne | \n","
| 41 | \n","Meet the Plant Hacker Creating Flowers Never S... | \n","https://www.wired.com/story/meet-the-plant-hac... | \n","Matt Reynolds | \n","
| 42 | \n","A Mysterious Respiratory Disease Has the Democ... | \n","https://www.wired.com/story/drc-mysterious-res... | \n","Marta Musso | \n","
| 43 | \n","Skip the Sea Kelp Supplements | \n","https://www.wired.com/story/pass-on-sea-kelp-s... | \n","Boutayna Chokrane | \n","
| 44 | \n","Why Soccer Players Are Training in the Dark | \n","https://www.wired.com/story/why-soccer-players... | \n","RM Clark | \n","
| 45 | \n","A Parasite That Eats Cattle Alive Is Creeping ... | \n","https://www.wired.com/story/a-parasite-that-ea... | \n","Geraldine Castro | \n","
| 46 | \n","Lasers Are Making It Easier to Find Buried Lan... | \n","https://www.wired.com/story/this-laser-system-... | \n","Ritsuko Kawai | \n","
| 47 | \n","Mark Cuban’s War on Drug Prices: ‖How Much Fuc... | \n","https://www.wired.com/story/big-interview-mark... | \n","Marah Eakin | \n","
| 48 | \n","Can Artificial Rain, Drones, or Satellites Cle... | \n","https://www.wired.com/story/artificial-rain-dr... | \n","Arunima Kar | \n","
| 49 | \n","These Stem Cell Treatments Are Worth Millions.... | \n","https://www.wired.com/story/stem-cells-cost-ri... | \n","Matt Reynolds | \n","
| 50 | \n","The Mystery of How Supermassive Black Holes Merge | \n","https://www.wired.com/story/how-do-merging-sup... | \n","Jonathan O’Callaghan | \n","
| 51 | \n","The $60 Billion Potential Hiding in Your Disca... | \n","https://www.wired.com/story/a-dollar60-billion... | \n","Vince Beiser | \n","
| 52 | \n","Tune In to the Healing Powers of a Decent Play... | \n","https://www.wired.com/story/music-therapy-heal... | \n","Daniel Levitin | \n","
| 53 | \n","Returning the Amazon Rainforest to Its True Ca... | \n","https://www.wired.com/story/amazon-rainforest-... | \n","Nemonte Nenquimo and Mitch Anderson | \n","
\n"," \n","
\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jEkuKbcRrPcK"
+ },
+ "source": [
+ "## 🕷️ Extract Wired Science News Info with llama-index and ScrapegraphAI's APIs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "oQa0HZo7nuB_"
+ },
+ "source": [
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "IzsyDXEWwPVt"
+ },
+ "source": [
+ "### 🔧 Install `dependencies`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "os_vm0MkIxr9"
+ },
+ "outputs": [],
+ "source": [
+ "%%capture\n",
+ "!pip install llama-index-tools-scrapegraphai\n",
+ "!pip install llama-index"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "apBsL-L2KzM7"
+ },
+ "source": [
+ "### 🔑 Import `ScrapeGraph` API key"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "ol9gQbAFkh9b"
+ },
+ "source": [
+ "You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "id": "sffqFG2EJ8bI"
+ },
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "if not os.environ.get(\"SGAI_API_KEY\"):\n",
+ " os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "jnqMB2-xVYQ7"
+ },
+ "source": [
+ "### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "VZvxbjfXvbgd"
+ },
+ "source": [
+ "If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n",
+ "\n",
+ "| \n", + " | category | \n", + "title | \n", + "link | \n", + "author | \n", + "
|---|---|---|---|---|
| 0 | \n", + "Science | \n", + "NASA Postpones Return of Stranded Starliner As... | \n", + "https://www.wired.com/story/boeing-starliner-a... | \n", + "Fernanda González | \n", + "
| 1 | \n", + "Public Health | \n", + "CDC Confirms First US Case of Severe Bird Flu | \n", + "https://www.wired.com/story/cdc-confirms-first... | \n", + "Emily Mullin | \n", + "
| 2 | \n", + "Science | \n", + "The Study That Called Out Black Plastic Utensi... | \n", + "https://www.wired.com/story/black-plastic-uten... | \n", + "Beth Mole, Ars Technica | \n", + "
| 3 | \n", + "Health | \n", + "A Third Person Has Received a Transplant of a ... | \n", + "https://www.wired.com/story/a-third-person-has... | \n", + "Emily Mullin | \n", + "
| 4 | \n", + "Health | \n", + "Antibodies Could Soon Help Slow the Aging Process | \n", + "https://www.wired.com/story/antibodies-could-s... | \n", + "Andrew Steele | \n", + "
| 5 | \n", + "Science | \n", + "Good at Reading? Your Brain May Be Structured ... | \n", + "https://www.wired.com/story/good-at-reading-yo... | \n", + "Mikael Roll | \n", + "
| 6 | \n", + "Environment | \n", + "Mega-Farms Are Driving the Threat of Bird Flu | \n", + "https://www.wired.com/story/mega-farms-are-dri... | \n", + "Georgina Gustin | \n", + "
| 7 | \n", + "Environment | \n", + "How Christmas Trees Could Become a Source of L... | \n", + "https://www.wired.com/story/how-christmas-tree... | \n", + "Alexa Phillips | \n", + "
| 8 | \n", + "Environment | \n", + "Creating a Global Package to Solve the Problem... | \n", + "https://www.wired.com/story/global-plastics-tr... | \n", + "Susan Solomon | \n", + "
| 9 | \n", + "Environment | \n", + "These 3 Things Are Standing in the Way of a Gl... | \n", + "https://www.wired.com/story/these-3-things-are... | \n", + "Steve Fletcher and Samuel Winton | \n", + "
| 10 | \n", + "Environment | \n", + "Environmental Sensing Is Here, Tracking Everyt... | \n", + "https://www.wired.com/story/environmental-sens... | \n", + "Sabrina Weiss | \n", + "
| 11 | \n", + "Climate | \n", + "Generative AI and Climate Change Are on a Coll... | \n", + "https://www.wired.com/story/true-cost-generati... | \n", + "Sasha Luccioni | \n", + "
| 12 | \n", + "Climate | \n", + "Climate Change Is Destroying Monarch Butterfli... | \n", + "https://www.wired.com/story/global-warming-thr... | \n", + "Andrea J. Arratibel | \n", + "
| 13 | \n", + "Politics | \n", + "More Humanitarian Organizations Will Harness A... | \n", + "https://www.wired.com/story/humanitarian-organ... | \n", + "David Miliband | \n", + "
| 14 | \n", + "Environment | \n", + "Chocolate Has a Sustainability Problem. Scienc... | \n", + "https://www.wired.com/story/chocolate-has-a-su... | \n", + "Eve Thomas | \n", + "
| 15 | \n", + "Energy | \n", + "Big Tech Will Scour the Globe in Its Search fo... | \n", + "https://www.wired.com/story/big-tech-data-cent... | \n", + "Azeem Azhar | \n", + "
| 16 | \n", + "Environment | \n", + "Humans Will Continue to Live in an Age of Incr... | \n", + "https://www.wired.com/story/food-production-en... | \n", + "Vaclav Smil | \n", + "
| 17 | \n", + "Energy | \n", + "A Uranium-Mining Boom Is Sweeping Through Texas | \n", + "https://www.wired.com/story/a-uranium-mining-b... | \n", + "Dylan Baddour | \n", + "
| 18 | \n", + "Space | \n", + "The End Is Near for NASA’s Voyager Probes | \n", + "https://www.wired.com/story/the-end-is-near-fo... | \n", + "Luca Nardi | \n", + "
| 19 | \n", + "Space | \n", + "The Mystery of How Supermassive Black Holes Merge | \n", + "https://www.wired.com/story/how-do-merging-sup... | \n", + "Jonathan O’Callaghan | \n", + "
| 20 | \n", + "Space | \n", + "Starship’s Next Launch Could Be Just Two Weeks... | \n", + "https://www.wired.com/story/starships-next-lau... | \n", + "Eric Berger, Ars Technica | \n", + "
| 21 | \n", + "Math | \n", + "The Simple Math Behind Public Key Cryptography | \n", + "https://www.wired.com/story/how-public-key-cry... | \n", + "John Pavlus | \n", + "
| 22 | \n", + "Math | \n", + "Everyone Is Capable of Mathematical Thinking--... | \n", + "https://www.wired.com/story/everyone-is-capabl... | \n", + "Kelsey Houston-Edwards | \n", + "
| 23 | \n", + "Math | \n", + "The Physics of the Macy’s Thanksgiving Day Par... | \n", + "https://www.wired.com/story/the-physics-of-the... | \n", + "Rhett Allain | \n", + "
| 24 | \n", + "Math | \n", + "Mathematicians Just Debunked the ‘Bunkbed Conj... | \n", + "https://www.wired.com/story/maths-bunkbed-conj... | \n", + "Joseph Howlett | \n", + "
| 25 | \n", + "Biotech | \n", + "Muscle Implants Could Allow Mind-Controlled Pr... | \n", + "https://www.wired.com/story/amputees-could-con... | \n", + "Emily Mullin | \n", + "
| 26 | \n", + "Biotech | \n", + "Combining AI and Crispr Will Be Transformational | \n", + "https://www.wired.com/story/combining-ai-and-c... | \n", + "Jennifer Doudna | \n", + "
| 27 | \n", + "Biotech | \n", + "Neuralink Plans to Test Whether Its Brain Impl... | \n", + "https://www.wired.com/story/neuralink-robotic-... | \n", + "Emily Mullin | \n", + "
| 28 | \n", + "Public Health | \n", + "A Mysterious Respiratory Disease Has the Democ... | \n", + "https://www.wired.com/story/drc-mysterious-res... | \n", + "Marta Musso | \n", + "
| 29 | \n", + "Health | \n", + "Skip the Sea Kelp Supplements | \n", + "https://www.wired.com/story/pass-on-sea-kelp-s... | \n", + "Boutayna Chokrane | \n", + "
| 30 | \n", + "Sports | \n", + "Why Soccer Players Are Training in the Dark | \n", + "https://www.wired.com/story/why-soccer-players... | \n", + "RM Clark | \n", + "
| 31 | \n", + "Environment | \n", + "A Parasite That Eats Cattle Alive Is Creeping ... | \n", + "https://www.wired.com/story/a-parasite-that-ea... | \n", + "Geraldine Castro | \n", + "
| 32 | \n", + "Technology | \n", + "Lasers Are Making It Easier to Find Buried Lan... | \n", + "https://www.wired.com/story/this-laser-system-... | \n", + "Ritsuko Kawai | \n", + "
| 33 | \n", + "Health | \n", + "Mark Cuban’s War on Drug Prices: ‘How Much Fuc... | \n", + "https://www.wired.com/story/big-interview-mark... | \n", + "Marah Eakin | \n", + "
| 34 | \n", + "Environment | \n", + "Can Artificial Rain, Drones, or Satellites Cle... | \n", + "https://www.wired.com/story/artificial-rain-dr... | \n", + "Arunima Kar | \n", + "
| 35 | \n", + "Health | \n", + "These Stem Cell Treatments Are Worth Millions.... | \n", + "https://www.wired.com/story/stem-cells-cost-ri... | \n", + "Matt Reynolds | \n", + "
| 36 | \n", + "Environment | \n", + "The $60 Billion Potential Hiding in Your Disca... | \n", + "https://www.wired.com/story/a-dollar60-billion... | \n", + "Vince Beiser | \n", + "
| 37 | \n", + "Health | \n", + "Tune In to the Healing Powers of a Decent Play... | \n", + "https://www.wired.com/story/music-therapy-heal... | \n", + "Daniel Levitin | \n", + "
| 38 | \n", + "Human History | \n", + "The Whole Story of How Humans Evolved From Gre... | \n", + "https://www.wired.com/story/the-whole-story-of... | \n", + "John Gowlett | \n", + "
| 39 | \n", + "Health | \n", + "Why an Offline Nuclear Reactor Led to Thousand... | \n", + "https://www.wired.com/story/why-an-offline-nuc... | \n", + "Chris Baraniuk | \n", + "
\n",
+ " \n",
+ "
\n",""],"metadata":{"id":"ReBHQ5_834pZ"}},{"cell_type":"markdown","source":["## 🕷️ Extract Wired Science News with Official Scrapegraph SDK"],"metadata":{"id":"jEkuKbcRrPcK"}},{"cell_type":"markdown","source":[""],"metadata":{"id":"6Rgz5T4eHBVz"}},{"cell_type":"markdown","source":["### 🔧 Install `dependencies`"],"metadata":{"id":"IzsyDXEWwPVt"}},{"cell_type":"code","execution_count":null,"metadata":{"id":"os_vm0MkIxr9"},"outputs":[],"source":["%%capture\n","!pip install scrapegraph-py"]},{"cell_type":"markdown","source":["### 🔑 Import `ScrapeGraph` API key"],"metadata":{"id":"apBsL-L2KzM7"}},{"cell_type":"markdown","source":["You can find the Scrapegraph API key [here](https://dashboard.scrapegraphai.com/)"],"metadata":{"id":"ol9gQbAFkh9b"}},{"cell_type":"code","source":["import getpass\n","import os\n","\n","if not os.environ.get(\"SGAI_API_KEY\"):\n"," os.environ[\"SGAI_API_KEY\"] = getpass.getpass(\"Scrapegraph API key:\\n\")"],"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"sffqFG2EJ8bI","executionInfo":{"status":"ok","timestamp":1734531168564,"user_tz":-60,"elapsed":17535,"user":{"displayName":"ScrapeGraphAI","userId":"10474323355016263615"}},"outputId":"bcdd9ae1-151e-41d6-df36-8835c660460f"},"execution_count":null,"outputs":[{"output_type":"stream","name":"stdout","text":["SGAI_API_KEY not found in environment.\n","Please enter your SGAI_API_KEY: ··········\n","SGAI_API_KEY has been set in the environment.\n"]}]},{"cell_type":"markdown","source":["### 📝 Defining an `Output Schema` for Webpage Content Extraction\n"],"metadata":{"id":"jnqMB2-xVYQ7"}},{"cell_type":"markdown","source":["If you already know what you want to extract from a webpage, you can **define an output schema** using **Pydantic**. This schema acts as a \"blueprint\" that tells the AI how to structure the response.\n","\n","| \n"," | category | \n","title | \n","link | \n","author | \n","
|---|---|---|---|---|
| 0 | \n","Science | \n","The Study That Called Out Black Plastic Utensi... | \n","https://www.wired.com/story/black-plastic-uten... | \n","Beth Mole, Ars Technica | \n","
| 1 | \n","Environment | \n","Generative AI and Climate Change Are on a Coll... | \n","https://www.wired.com/story/true-cost-generati... | \n","Sasha Luccioni | \n","
| 2 | \n","Xenotransplantation | \n","A Third Person Has Received a Transplant of a ... | \n","https://www.wired.com/story/a-third-person-has... | \n","Emily Mullin | \n","
| 3 | \n","Health | \n","Antibodies Could Soon Help Slow the Aging Process | \n","https://www.wired.com/story/antibodies-could-s... | \n","Andrew Steele | \n","
| 4 | \n","Science | \n","Good at Reading? Your Brain May Be Structured ... | \n","https://www.wired.com/story/good-at-reading-yo... | \n","Mikael Roll | \n","
| 5 | \n","Health | \n","Mega-Farms Are Driving the Threat of Bird Flu | \n","https://www.wired.com/story/mega-farms-are-dri... | \n","Georgina Gustin | \n","
| 6 | \n","Health | \n","RFK Plans to Take on Big Pharma. It’s Easier S... | \n","https://www.wired.com/story/rfks-plan-to-take-... | \n","Emily Mullin | \n","
| 7 | \n","Environment | \n","How Christmas Trees Could Become a Source of L... | \n","https://www.wired.com/story/how-christmas-tree... | \n","Alexa Phillips | \n","
| 8 | \n","Environment | \n","Creating a Global Package to Solve the Problem... | \n","https://www.wired.com/story/global-plastics-tr... | \n","Susan Solomon | \n","
| 9 | \n","Environment | \n","These 3 Things Are Standing in the Way of a Gl... | \n","https://www.wired.com/story/these-3-things-are... | \n","Steve Fletcher and Samuel Winton | \n","
\n"," \n","