diff --git a/README.md b/README.md index bc86438..2519524 100644 --- a/README.md +++ b/README.md @@ -79,10 +79,10 @@ You can host containers within either the older Google Container Registry, or el ## **Serverless Functionality** -Serverless services are those that allow you to run things, an analysis, an app, a website etc., and not have to deal with servers (VMs). There are still servers running somewhere, you just don't have to manage them. All you have to do is call a command that runs your analysis in the background and copies the output files to a storage bucket. The most relevant serverless feature on GCP to Cloud Lab users (especially 'omics' analyses) is the [Google Cloud Life Sciences API](https://cloud.google.com/life-sciences/docs/reference/rest). You can walk through a tutorial of this service using this [notebook](/tutorials/notebooks/LifeSciencesAPI) Those doing health informatics should look into the [Google Cloud Healthcare Data Engine](https://cloud.google.com/healthcare). You can find a variety of other [tutorials](https://cloud.google.com/life-sciences/docs/tutorials) that leverage the Life Sciences API for life sciences applications, but we will point out that most of the examples are related to genomics. If you are doing other biomedical research related to imaging, NLP, or other fields, look at the [tutorials](/tutorials/) section of this repo for inspiration. Some of these also leverage the API from within the notebooks. Besides the Google-specific examples, you can use the Life Sciences API to run workflows using other workflow managers like [Snakemake](https://snakemake.readthedocs.io/en/stable/executing/cloud.html) or [Nextflow](https://cloud.google.com/life-sciences/docs/tutorials/nextflow). Google also just released a new service in preview called [Batch](https://cloud.google.com/blog/products/compute/new-batch-service-processes-batch-jobs-on-google-cloud) which is a compute scheduler that should feel similar to submitting jobs to a cluster. It will likely replace the Life Sciences API in the near future, but for now, you should experiment with both for submitting serverless jobs. +Serverless services are those that allow you to run things, an analysis, an app, a website etc., and not have to deal with servers (VMs). There are still servers running somewhere, you just don't have to manage them. All you have to do is call a command that runs your analysis in the background and copies the output files to a storage bucket. The most relevant serverless feature on GCP to Cloud Lab users (especially 'omics' analyses) is [Google Batch](https://cloud.google.com/batch/docs/get-started). You can walk through a tutorial of this service using this [notebook](/tutorials/notebooks/GoogleBatch/nextflow) Those doing health informatics should look into the [Google Cloud Healthcare Data Engine](https://cloud.google.com/healthcare). You can find a variety of other tutorials from the [NIGMS Sandbox](https://github.com/NIGMS/NIGMS-Sandbox) as well as this [Google tutorial](https://cloud.google.com/batch/docs/nextflow). ## **Clusters** -One great thing about the cloud is its ability to scale with demand. When you submit a job to a traditional computing cluster (a set of computers that work together to execute a function), you have to specify up front how many CPUs and how much memory you want to allocate to your job, and you may over- or under-utilize these resources. Alternatively, on the cloud you can leverage a feature called autoscaling, where the compute resources will scale up or down with the demand. This is more efficient and keeps costs down when demand is low, but prevents latency when demand is high (e.g., a whole hackathon submitting jobs to a cluster). For most users of Cloud Lab, the best way to benefit from autoscaling is to use an API like the Life Sciences API or Google BATCH, but in some cases, maybe for a whole lab group or a large project, it may make sense to spin up a [Kubernetes cluster](https://cloud.google.com/kubernetes-engine/docs/quickstart) and submit jobs to the cluster using a workflow manager like [Snakemake](https://snakemake.readthedocs.io/en/stable/executing/cloud.html). [One of our tutorials](/tutorials/notebooks/DL-gwas-gcp-example) uses a marketplace solution to deploy a small Kubernetes cluster and then run AI models using Kubeflow. Finally, you can spin up SLURM clusters on GCP, which is easiest to do through the Cloud Shell or a VM, but can be accessed via SSH from your terminal. Instructions are [here](https://cloud.google.com/architecture/deploying-slurm-cluster-compute-engine). +One great thing about the cloud is its ability to scale with demand. When you submit a job to a traditional computing cluster (a set of computers that work together to execute a function), you have to specify up front how many CPUs and how much memory you want to allocate to your job, and you may over- or under-utilize these resources. Alternatively, on the cloud you can leverage a feature called autoscaling, where the compute resources will scale up or down with the demand. This is more efficient and keeps costs down when demand is low, but prevents latency when demand is high (e.g., a whole hackathon submitting jobs to a cluster). For most users of Cloud Lab, the best way to benefit from autoscaling is to use an API like the Life Sciences API or Google BATCH, but in some cases, maybe for a whole lab group or a large project, it may make sense to spin up a [Kubernetes cluster](https://cloud.google.com/kubernetes-engine/docs/quickstart) and submit jobs to the cluster using a workflow manager like [Snakemake](https://snakemake.readthedocs.io/en/stable/getting_started/migration.html#command-line-interface). [One of our tutorials](/tutorials/notebooks/DL-gwas-gcp-example) uses a marketplace solution to deploy a small Kubernetes cluster and then run AI models using Kubeflow. Finally, you can spin up SLURM clusters on GCP, which is easiest to do through the Cloud Shell or a VM, but can be accessed via SSH from your terminal. Instructions are [here](https://cloud.google.com/architecture/deploying-slurm-cluster-compute-engine). ## **Billing and Benchmarking** Many Cloud Lab users are interested in understanding how to estimate the price of a large-scale project using a reduced sample size. Generally, you should be able to benchmark with a few representative samples to get an idea of the time and cost required for a larger scale project. In terms of cost, a great way to estimate costs is to use the [GCP cost calculator](https://cloud.google.com/products/calculator#id=b64e9c4f-c637-432f-8e2c-4a7238dde0f2) for an initial figure. The calculator is a tool that estimates costs based on location, VM type/size, and runtime. Then, you can run some benchmarks and double check that everything is acting as you expect. For example, if you know that your analysis on your on-premises cluster takes 4 hours to run for a single sample with 12 CPUs, and that each sample needs about 30 GB of storage to run a workflow, then you can extrapolate to determine total costs using the calculator (e.g., compute engine + cloud storage). diff --git a/images/Gemini_1.png b/images/Gemini_1.png new file mode 100644 index 0000000..b42b482 Binary files /dev/null and b/images/Gemini_1.png differ diff --git a/images/Gemini_2.png b/images/Gemini_2.png new file mode 100644 index 0000000..3fcaf6a Binary files /dev/null and b/images/Gemini_2.png differ diff --git a/images/Gemini_3.png b/images/Gemini_3.png new file mode 100644 index 0000000..62fe5ec Binary files /dev/null and b/images/Gemini_3.png differ diff --git a/images/Gemini_4.png b/images/Gemini_4.png new file mode 100644 index 0000000..ad96e82 Binary files /dev/null and b/images/Gemini_4.png differ diff --git a/tutorials/README.md b/tutorials/README.md index b2f966e..e9cdb3f 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -31,9 +31,9 @@ There are a lot of ways to run workflows on GCP. Here we list a few possibilitie - The simplest method is probably to spin up a Compute Engine instance, and run your command interactively, or using `screen` or, as a [startup script](https://cloud.google.com/compute/docs/instances/startup-scripts/linux) attached as metadata. - You could also run your pipeline via a Vertex AI notebook, either by splitting out each command as a different block, or by running a workflow manager (Nextflow etc.). [Schedule notebooks](https://codelabs.developers.google.com/vertex_notebook_executor#0) to let them run longer. You can find a nice tutorial for using managed notebooks [here](https://codelabs.developers.google.com/vertex_notebook_executor#0). Note that there is now a difference between `managed notebooks` and `user managed notebooks`. The `managed notebooks` have more features and can be scheduled, but give you less control about conda environments/install. -- You can interact with [Google Batch](https://cloud.google.com/batch/docs/get-started), or the [Google Life Sciences API](https://cloud.google.com/life-sciences/docs/reference/rest) using a workflow manager like [Nextflow](https://cloud.google.com/life-sciences/docs/tutorials/nextflow), [Snakemake](https://snakemake.readthedocs.io/en/stable/executing/cloud.html), or [Cromwell](https://github.com/GoogleCloudPlatform/rad-lab/tree/main/modules/genomics_cromwell). We currently have example notebooks for both [Nextflow and Snakemake that use the Life Sciences API](/tutorials/notebooks/LifeSciencesAPI/), as well as [Google Batch with Nextflow](/tutorials/notebooks/GoogleBatch/nextflow) as well as a [local version of Snakemake run via Pangolin](/tutorials/notebooks/pangolin). +- You can interact with [Google Batch](https://cloud.google.com/batch/docs/get-started), or the [Google Life Sciences API](https://cloud.google.com/life-sciences/docs/reference/rest) using a workflow manager like [Nextflow](https://cloud.google.com/life-sciences/docs/tutorials/nextflow), [Snakemake](https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/googlebatch.html), or [Cromwell](https://github.com/GoogleCloudPlatform/rad-lab/tree/main/modules/genomics_cromwell). We currently have example notebooks for both [Nextflow and Snakemake that use the Life Sciences API](/tutorials/notebooks/LifeSciencesAPI/), as well as [Google Batch with Nextflow](/tutorials/notebooks/GoogleBatch/nextflow) as well as a [local version of Snakemake run via Pangolin](/tutorials/notebooks/pangolin). - You may find other APIs better suite your needs such as the [Google Cloud Healthcare Data Engine](https://cloud.google.com/healthcare). -- Most of the notebooks below require just a few CPUs. Start small (maybe 4 CPUs), then scale up as needed. Likewise, when you need a GPU, start with a smaller or older generation GPU (e.g. T4) for testing, then switch to a newer GPU (A100/V100) once you know things will work or you need more horsepower. +- Most of the notebooks below require just a few CPUs. Start small (maybe 4 CPUs), then scale up as needed. Likewise, when you need a GPU, start with a smaller or older generation GPU (e.g. T4) for testing, then switch to a newer GPU (A100/V100) once you know things will work or you need more compute power. ## **Artificial Intelligence and Machine Learning** Machine learning is a subfield of artificial intelligence that focuses on the development of algorithms and models that enable computers to learn from and make predictions or decisions based on data, without being explicitly programmed. Machine learning on GCP generally occurs within VertexAI. You can learn more about machine learning on GCP at this [Google Crash Course](https://developers.google.com/machine-learning/crash-course). For hands-on examples, try out [this module](https://github.com/NIGMS/COVIDMachineLearningSFSU) developed by San Francisco State University or [this one from the University of Arkasas](https://github.com/NIGMS/MachineLearningUA) developed for the NIGMS Sandbox Project. @@ -122,7 +122,7 @@ These notebooks were created to run in Google Colab, so if you run them in Googl You can interact with Google Batch directly to submit commands, or more commonly you can interact with it through orchestration engines like [Nextflow](https://www.nextflow.io/docs/latest/google.html) and [Cromwell](https://cromwell.readthedocs.io/en/latest/backends/GCPBatch/), etc. We have tutorials that utilize Google Batch using [Nextflow](/tutorials/notebooks/GoogleBatch/nextflow) where we run the nf-core Methylseq pipeline, as well as several from the NIGMS Sandbox including [transcriptome assembly](https://github.com/NIGMS/rnaAssemblyMDI), [multiomics](https://github.com/NIGMS/MultiomicsUND), [methylseq](https://github.com/NIGMS/MethylSeqUH), and [metagenomics](https://github.com/NIGMS/MetagenomicsUSD). ## **Using the Life Sciences API (depreciated)** -__Life Science API is depreciated on GCP and will no longer be available by July 8, 2025 on the platform,__ we recommend using Google Batch instead. For now you can still interact with the Life Sciences API directly to submit commands, or more commonly you can interact with it through orchestration engines like [Snakemake](https://snakemake.readthedocs.io/en/stable/executing/cloud.html), as of now this workflow manager only supports Life Sciences API. +__Life Science API is depreciated on GCP and will no longer be available by July 8, 2025 on the platform,__ we recommend using Google Batch instead. For now you can still interact with the Life Sciences API directly to submit commands, or more commonly you can interact with it through orchestration engines like [Snakemake](https://snakemake.readthedocs.io/en/v7.0.0/executor_tutorial/google_lifesciences.html), as of now this workflow manager only supports Life Sciences API. ## **Public Data Sets** Google has a lot of public datasets available that you can use for your testing. These can be viewed [here](https://cloud.google.com/life-sciences/docs/resources/public-datasets) and can be accessed via [BigQuery](https://cloud.google.com/bigquery/public-data) or directly from the cloud bucket. For example, to view Phase 3 1000 Genomes at the command line type `gsutil ls gs://genomics-public-data/1000-genomes-phase-3`. diff --git a/tutorials/notebooks/GenAI/Gemini_Intro.ipynb b/tutorials/notebooks/GenAI/Gemini_Intro.ipynb new file mode 100644 index 0000000..94b8d7a --- /dev/null +++ b/tutorials/notebooks/GenAI/Gemini_Intro.ipynb @@ -0,0 +1,853 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c51ed24c-53dd-4239-8be0-0e0422596ba3", + "metadata": {}, + "source": [ + "# Intro to GCP's Gemini " + ] + }, + { + "cell_type": "markdown", + "id": "22cc412f-5338-45ba-8b29-9039c19208d9", + "metadata": {}, + "source": [ + "**Gemini** is a Google multimodal model that has the capability to **summarize, chat, and generate text from images or videos**. Gemini comes in two model versions **Gemini Pro** and **Gemini Pro Vision**, for this tutorial we will be looking into utilizing both models via python packages and GCPs model playground, **Vertex AI Studio**." + ] + }, + { + "cell_type": "markdown", + "id": "28289191-0813-47d1-be6d-8feb0ae708bd", + "metadata": {}, + "source": [ + "## Gemini in Python" + ] + }, + { + "cell_type": "markdown", + "id": "b8ec6d40-b5b3-434f-adc4-2838b7f49d1d", + "metadata": { + "tags": [] + }, + "source": [ + "The first step inorder to stat using Gemini is to update the google-cloud-aiplatform python package if you havent already." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "ed9781dd-9764-4e9c-88ba-fcd7bb95842a", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: google-cloud-aiplatform in /opt/conda/lib/python3.10/site-packages (1.37.0)\n", + "Collecting google-cloud-aiplatform\n", + " Downloading google_cloud_aiplatform-1.39.0-py2.py3-none-any.whl.metadata (28 kB)\n", + "Requirement already satisfied: google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.34.0)\n", + "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (1.23.0)\n", + "Requirement already satisfied: protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (3.20.3)\n", + "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (23.2)\n", + "Requirement already satisfied: google-cloud-storage<3.0.0dev,>=1.32.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (2.13.0)\n", + "Requirement already satisfied: google-cloud-bigquery<4.0.0dev,>=1.15.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (3.13.0)\n", + "Requirement already satisfied: google-cloud-resource-manager<3.0.0dev,>=1.3.3 in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (1.11.0)\n", + "Requirement already satisfied: shapely<3.0.0dev in /opt/conda/lib/python3.10/site-packages (from google-cloud-aiplatform) (2.0.2)\n", + "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.62.0)\n", + "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.25.2)\n", + "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.10/site-packages (from google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2.31.0)\n", + "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.60.0)\n", + "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.10/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.48.2)\n", + "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.6.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.4.1)\n", + "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.6.0)\n", + "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.10/site-packages (from google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (2.8.2)\n", + "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /opt/conda/lib/python3.10/site-packages (from google-cloud-resource-manager<3.0.0dev,>=1.3.3->google-cloud-aiplatform) (0.13.0)\n", + "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.10/site-packages (from google-cloud-storage<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.5.0)\n", + "Requirement already satisfied: numpy>=1.14 in /opt/conda/lib/python3.10/site-packages (from shapely<3.0.0dev->google-cloud-aiplatform) (1.25.2)\n", + "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (5.3.2)\n", + "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.3.0)\n", + "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.10/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (4.9)\n", + "Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.10/site-packages (from python-dateutil<3.0dev,>=2.7.2->google-cloud-bigquery<4.0.0dev,>=1.15.0->google-cloud-aiplatform) (1.16.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (3.6)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (1.26.18)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (2023.11.17)\n", + "Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /opt/conda/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0->google-cloud-aiplatform) (0.5.1)\n", + "Downloading google_cloud_aiplatform-1.39.0-py2.py3-none-any.whl (3.4 MB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.4/3.4 MB\u001b[0m \u001b[31m13.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hInstalling collected packages: google-cloud-aiplatform\n", + " Attempting uninstall: google-cloud-aiplatform\n", + " Found existing installation: google-cloud-aiplatform 1.37.0\n", + " Uninstalling google-cloud-aiplatform-1.37.0:\n", + " Successfully uninstalled google-cloud-aiplatform-1.37.0\n", + "Successfully installed google-cloud-aiplatform-1.39.0\n" + ] + } + ], + "source": [ + "!pip install --upgrade google-cloud-aiplatform" + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "id": "54499594-1d7b-41cc-858b-ffe6f6c2770b", + "metadata": { + "scrolled": true, + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting langchain\n", + " Downloading langchain-0.1.0-py3-none-any.whl.metadata (13 kB)\n", + "Requirement already satisfied: PyYAML>=5.3 in /opt/conda/lib/python3.10/site-packages (from langchain) (6.0.1)\n", + "Requirement already satisfied: SQLAlchemy<3,>=1.4 in /opt/conda/lib/python3.10/site-packages (from langchain) (2.0.23)\n", + "Requirement already satisfied: aiohttp<4.0.0,>=3.8.3 in /opt/conda/lib/python3.10/site-packages (from langchain) (3.9.1)\n", + "Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /opt/conda/lib/python3.10/site-packages (from langchain) (4.0.3)\n", + "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain)\n", + " Downloading dataclasses_json-0.6.3-py3-none-any.whl.metadata (25 kB)\n", + "Requirement already satisfied: jsonpatch<2.0,>=1.33 in /opt/conda/lib/python3.10/site-packages (from langchain) (1.33)\n", + "Collecting langchain-community<0.1,>=0.0.9 (from langchain)\n", + " Downloading langchain_community-0.0.11-py3-none-any.whl.metadata (7.3 kB)\n", + "Collecting langchain-core<0.2,>=0.1.7 (from langchain)\n", + " Downloading langchain_core-0.1.9-py3-none-any.whl.metadata (4.0 kB)\n", + "Requirement already satisfied: langsmith<0.1.0,>=0.0.77 in /opt/conda/lib/python3.10/site-packages (from langchain) (0.0.77)\n", + "Requirement already satisfied: numpy<2,>=1 in /opt/conda/lib/python3.10/site-packages (from langchain) (1.25.2)\n", + "Requirement already satisfied: pydantic<3,>=1 in /opt/conda/lib/python3.10/site-packages (from langchain) (1.10.13)\n", + "Requirement already satisfied: requests<3,>=2 in /opt/conda/lib/python3.10/site-packages (from langchain) (2.31.0)\n", + "Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /opt/conda/lib/python3.10/site-packages (from langchain) (8.2.3)\n", + "Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (23.1.0)\n", + "Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (6.0.4)\n", + "Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.9.4)\n", + "Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.4.0)\n", + "Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp<4.0.0,>=3.8.3->langchain) (1.3.1)\n", + "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n", + " Downloading marshmallow-3.20.2-py3-none-any.whl.metadata (7.5 kB)\n", + "Collecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain)\n", + " Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB)\n", + "Requirement already satisfied: jsonpointer>=1.9 in /opt/conda/lib/python3.10/site-packages (from jsonpatch<2.0,>=1.33->langchain) (2.4)\n", + "Requirement already satisfied: anyio<5,>=3 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.2,>=0.1.7->langchain) (3.7.1)\n", + "Requirement already satisfied: packaging<24.0,>=23.2 in /opt/conda/lib/python3.10/site-packages (from langchain-core<0.2,>=0.1.7->langchain) (23.2)\n", + "Requirement already satisfied: typing-extensions>=4.2.0 in /opt/conda/lib/python3.10/site-packages (from pydantic<3,>=1->langchain) (4.8.0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain) (3.6)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain) (1.26.18)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests<3,>=2->langchain) (2023.11.17)\n", + "Requirement already satisfied: greenlet!=0.4.17 in /opt/conda/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain) (3.0.2)\n", + "Requirement already satisfied: sniffio>=1.1 in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.7->langchain) (1.3.0)\n", + "Requirement already satisfied: exceptiongroup in /opt/conda/lib/python3.10/site-packages (from anyio<5,>=3->langchain-core<0.2,>=0.1.7->langchain) (1.2.0)\n", + "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain)\n", + " Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n", + "Downloading langchain-0.1.0-py3-none-any.whl (797 kB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m798.0/798.0 kB\u001b[0m \u001b[31m5.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n", + "\u001b[?25hDownloading dataclasses_json-0.6.3-py3-none-any.whl (28 kB)\n", + "Downloading langchain_community-0.0.11-py3-none-any.whl (1.5 MB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.5/1.5 MB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n", + "\u001b[?25hDownloading langchain_core-0.1.9-py3-none-any.whl (216 kB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m216.5/216.5 kB\u001b[0m \u001b[31m29.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading marshmallow-3.20.2-py3-none-any.whl (49 kB)\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.4/49.4 kB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25hDownloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n", + "Installing collected packages: mypy-extensions, marshmallow, typing-inspect, langchain-core, dataclasses-json, langchain-community, langchain\n", + " Attempting uninstall: langchain-core\n", + " Found existing installation: langchain-core 0.1.6\n", + " Uninstalling langchain-core-0.1.6:\n", + " Successfully uninstalled langchain-core-0.1.6\n", + "Successfully installed dataclasses-json-0.6.3 langchain-0.1.0 langchain-community-0.0.11 langchain-core-0.1.9 marshmallow-3.20.2 mypy-extensions-1.0.0 typing-inspect-0.9.0\n" + ] + } + ], + "source": [ + "!pip install langchain" + ] + }, + { + "cell_type": "markdown", + "id": "f7bbbed7-becf-48d0-98c3-6dd9942fd377", + "metadata": {}, + "source": [ + "Next we initialize the Gemini model by setting out project id and location. We are also pulling in the packages:\n", + "- **GenerativeModel:** Allows us to specify and launch the Gemini model we need (e.g. Gemini Pro, Gemini Pro Vision).\n", + "- **ChatSession:** Set Gemini Pro in chatbot mode.\n", + "- **Part:** Loads in files from buckets.\n", + "- **Image:** Loads in image files locally.\n", + "- **GenerationConfig:** Allows us to configure the models temperature, top p, top k, and max tokens." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "47dc9232-383f-405b-b1a8-fab64a80492d", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "from google.cloud import aiplatform\n", + "import vertexai.preview\n", + "from vertexai.preview.generative_models import GenerativeModel, ChatSession, Part, Image, GenerationConfig\n", + "\n", + "# TODO(developer): Update and un-comment below lines\n", + "project_id = \n", + "location = \n", + "vertexai.init(project=project_id, location=location)" + ] + }, + { + "cell_type": "markdown", + "id": "022f24ba-4034-4424-91d8-1229682755ab", + "metadata": {}, + "source": [ + "### Gemini as a Chatbot" + ] + }, + { + "cell_type": "markdown", + "id": "b7dec028-6d30-4078-ac54-62b849ae9ced", + "metadata": {}, + "source": [ + "For dealing with text, code generation, natural language tasks we can use the **gemini-pro** model and to set our model in **chatbot mode** we need to use the `start_chat()` function. You will see below we also created a function named **get_chat_response** which will send the prompt or message we have for our model using the `send_message()` function and returns only the text of the chats response." + ] + }, + { + "cell_type": "code", + "execution_count": 105, + "id": "70bc5b25-c796-4015-82dc-6bc861bb525f", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "model = GenerativeModel(\"gemini-pro\")\n", + "chat = model.start_chat()\n", + "\n", + "def get_chat_response(chat: ChatSession, prompt: str):\n", + " response = chat.send_message(prompt)\n", + " return response.text" + ] + }, + { + "cell_type": "markdown", + "id": "ae40e87a-76ac-469b-a504-fb903319cbaf", + "metadata": {}, + "source": [ + "Now that we have our functions lets ask our Gemini chatbot some questions!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "9d3277de-ab85-417f-b1d8-b21985a7a21b", + "metadata": {}, + "outputs": [], + "source": [ + "prompt = \"Hello.\"\n", + "print(get_chat_response(chat, prompt))" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "342a0e3d-fbcb-4562-bb5f-b439a92e80e2", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "**Generative AI use cases that are Life Science or Health Care related:**\n", + "\n", + "* **Drug discovery and development:** Generative AI can be used to generate new molecules with desired properties, design new drugs, and predict how drugs will interact with biological systems. This can help to accelerate the drug discovery and development process and make it more efficient.\n", + "* **Personalized medicine:** Generative AI can be used to generate personalized treatment plans for patients based on their individual genetic and health data. This can help to improve the effectiveness of treatment and reduce side effects.\n", + "* **Disease diagnosis and prognosis:** Generative AI can be used to develop new diagnostic tools and methods, and to predict the course of a disease. This can help to improve patient outcomes and reduce healthcare costs.\n", + "* **Medical imaging:** Generative AI can be used to generate synthetic medical images, which can be used to train medical students and residents, develop new imaging technologies, and improve the accuracy of diagnosis.\n", + "* **Electronic health records (EHRs):** Generative AI can be used to generate synthetic EHRs, which can be used to train machine learning algorithms, develop new clinical decision support tools, and improve the efficiency of healthcare operations.\n", + "* **Healthcare chatbots:** Generative AI can be used to develop healthcare chatbots that can provide patients with information and support, answer questions, and schedule appointments. This can help to improve patient access to care and reduce the burden on healthcare providers.\n", + "* **Drug repurposing:** Generative AI can be used to identify new uses for existing drugs, which can help to expand treatment options for patients and reduce the cost of drug development.\n", + "* **Clinical trial design:** Generative AI can be used to design more efficient and effective clinical trials, which can help to accelerate the development of new treatments and improve patient outcomes.\n", + "* **Healthcare fraud detection:** Generative AI can be used to detect fraudulent healthcare claims, which can help to reduce costs and improve the efficiency of healthcare operations.\n", + "\n", + "These are just a few examples of the many potential use cases for generative AI in the life science and healthcare industries. As generative AI technology continues to develop, we can expect to see even more innovative and groundbreaking applications in the years to come.\n" + ] + } + ], + "source": [ + "prompt = \"List gen ai use cases that are Life Science or Health Care related. \"\n", + "print(get_chat_response(chat, prompt))" + ] + }, + { + "cell_type": "markdown", + "id": "fcfda59a-c440-4489-8f00-a4316b827292", + "metadata": {}, + "source": [ + "We can even ask it to **generate code or debug code**!" + ] + }, + { + "cell_type": "code", + "execution_count": 106, + "id": "f0b917b2-22b5-4011-a9c4-d8a667cf6b1d", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Sure, here's a Python script that will replace all null values (empty cells) with zeros within a CSV file:\n", + "\n", + "\n", + "```python\n", + "import csv\n", + "\n", + "# Open the CSV file for reading and writing.\n", + "with open('input.csv', 'r+', newline='') as csvfile:\n", + " # Create a CSV reader and writer.\n", + " reader = csv.reader(csvfile)\n", + " writer = csv.writer(csvfile)\n", + "\n", + " # Read the header row.\n", + " header = next(reader)\n", + "\n", + " # Replace null values with zeros in the remaining rows.\n", + " for row in reader:\n", + " for i, cell in enumerate(row):\n", + " if cell == '':\n", + " row[i] = '0'\n", + "\n", + " # Write the updated row to the CSV file.\n", + " writer.writerow(row)\n", + "```\n", + "\n", + "\n", + "To use this script, save it as a file (e.g. `replace_nulls.py`) and run it from the command line:\n", + "\n", + "\n", + "```\n", + "python replace_nulls.py\n", + "```\n", + "\n", + "\n", + "This will replace all null values in the 'input.csv' file with zeros and create a new CSV file called 'output.csv'.\n", + "\n", + "\n", + "**Note:** Make sure to replace `input.csv` with the actual name of your input CSV file. You can also change the output file name by modifying the `output.csv` part of the script.\n" + ] + } + ], + "source": [ + "prompt = \"create a python code that will replace all null values to zero within a csv file\"\n", + "print(get_chat_response(chat, prompt))" + ] + }, + { + "cell_type": "markdown", + "id": "3a9d57d2-dbe1-434a-9345-fe5ae3315a21", + "metadata": {}, + "source": [ + "### Gemini as a Summarizer" + ] + }, + { + "cell_type": "markdown", + "id": "4856bd61-3e24-40ea-8da4-1a6edc5f3e1d", + "metadata": {}, + "source": [ + "We can generate text like asking Gemini Pro to summarize articles we provide locally (using langchain). As of now Gemini does not support loading in documents that are not videos and images directly. " + ] + }, + { + "cell_type": "markdown", + "id": "b5d7df19-a625-40dc-b4e5-faff5e7ba241", + "metadata": {}, + "source": [ + "First we will load in a file using langchains text loader. You can also use langchain to load in files from your bucket following the instructions [here](https://python.langchain.com/docs/integrations/document_loaders/google_cloud_storage_file)." + ] + }, + { + "cell_type": "code", + "execution_count": 71, + "id": "3becd6c2-daf0-4287-80e5-06cf419287bd", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "from langchain_community.document_loaders import TextLoader\n", + "\n", + "loader = TextLoader(\"./PMC10000003.txt\")\n", + "ex_file=loader.load()" + ] + }, + { + "cell_type": "markdown", + "id": "4e3d6c19-7d68-49e1-85e9-91bcd6bd1775", + "metadata": {}, + "source": [ + "We can configure our model to give us the best optimal output by setting the parameters below:\n", + "- **Max_Output_Token**: Max number of words to generate.\n", + "- **Temperature:** Controls randomness, higher values increase diversity meaning a more unique response make the model to think harder. Must be a number from 0 to 1.\n", + "- **Top_p (nucleus):** The cumulative probability cutoff for token selection. Lower values mean sampling from a smaller, more top-weighted nucleus. Must be a number from 0 to 1.\n", + "- **Top_k:** Sample from the k most likely next tokens at each step. Lower k focuses on higher probability tokens. This means the model choses the most probable words. Lower values eliminate fewer coherent words.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "id": "c4228e44-9639-40da-8f69-343be93b65b0", + "metadata": {}, + "outputs": [], + "source": [ + "generation_config = GenerationConfig(\n", + " temperature=0.9,\n", + " top_p=1.0,\n", + " top_k=32,\n", + " candidate_count=1,\n", + " max_output_tokens=8192,\n", + ")\n", + "\n", + "def summarizer(file: str) -> str:\n", + " \n", + " # Query the model\n", + " response = model.generate_content(\n", + " [\n", + " # Add an example query\n", + " \"summarize this file.\",\n", + " file\n", + " ],\n", + " generation_config=generation_config,\n", + " )\n", + " #print(response)\n", + " return response.text" + ] + }, + { + "cell_type": "markdown", + "id": "0bb722d6-ef79-4a0b-9327-04811e7f8ffc", + "metadata": {}, + "source": [ + "Here we are inputting only the page content from our document loader." + ] + }, + { + "cell_type": "code", + "execution_count": 92, + "id": "97e7ea82-c58e-42ee-b01e-6aa51e324b05", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "The article \"Mechanical Treatment of Inflammation of the Knee Joint\" published in the Chicago Medical Examiner on January 1867, discusses the use of mechanical treatments for inflammation of the knee joint. The author emphasizes the importance of overcoming the reflex contraction of muscles surrounding the joint to prevent or correct deformities. Tenotomy of the flexor tendons may be necessary to achieve this. Additionally, the relief of pressure on the inflamed joint surfaces is crucial for recovery. This can be achieved through various methods such as adhesive strap dressings, application of an air cushion, or evacuation of pus from the joint. The author also introduces a new apparatus for making extension, which allows for optimal counter-extension and can be used in both acute and chronic cases. The advantages of this apparatus include its large counter-extending surface, security, and patient comfort. By utilizing this principle, various instruments can be crafted to address knee deformities.\n" + ] + } + ], + "source": [ + "print(summarizer(ex_file[0].page_content))" + ] + }, + { + "cell_type": "markdown", + "id": "77c10bd8-1570-4dfa-9b2a-999e3f149faf", + "metadata": {}, + "source": [ + "### Gemini as a Image to Text Generator" + ] + }, + { + "cell_type": "markdown", + "id": "476052b3-ff64-4e5a-819c-7a18daf7f413", + "metadata": {}, + "source": [ + "Gemini Pro Vision can generate text from images and videos. These text can be descriptions or questions about the image or video. You can download an image or retrieve an image from your bucket or locally." + ] + }, + { + "cell_type": "markdown", + "id": "214ee84f-f058-411a-944d-4e149cd0e9bc", + "metadata": {}, + "source": [ + "Images can only be in the following formats: \n", + "- PNG - image/png\n", + "- JPEG - image/jpeg\n", + "\n", + "Our function below takes in a prompt and the image, we have also included a if statement to recognize if the function should use `Image` to load in a image locally or `Part` to load it from a bucket." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "501efefe-d52f-43b3-b4eb-3d3fe81f4a3e", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def img2text(image_path: str, img_prompt: str) -> str:\n", + " multimodal_model = GenerativeModel(\"gemini-pro-vision\")\n", + " if \"gs://\" in image_path:\n", + " image1=Part.from_uri(image_path, mime_type=\"image/jpeg\")\n", + " else: \n", + " image1=Image.load_from_file(image_path)\n", + " #image1=Image.load_from_file(image_path)\n", + " responses = multimodal_model.generate_content(\n", + " [image1, img_prompt],\n", + " generation_config={\n", + " \"max_output_tokens\": 2048,\n", + " \"temperature\": 0.4,\n", + " \"top_p\": 1,\n", + " \"top_k\": 32\n", + " },\n", + " stream=True,\n", + " )\n", + " for response in responses:\n", + " print(response.text, end=\"\")" + ] + }, + { + "cell_type": "markdown", + "id": "4cafaaa8-f3c7-472a-8f0b-6595d2112636", + "metadata": {}, + "source": [ + "Lets look at an image locally, by loading a image first, this a image of a Covid virus from the [CDC Public Health Image Library](https://phil.cdc.gov/details.aspx?pid=23312)." + ] + }, + { + "cell_type": "code", + "execution_count": 146, + "id": "b939f105-89c2-4c38-80f8-2cddf8dcb0ca", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2024-01-11 05:24:55-- https://phil.cdc.gov//PHIL_Images/23312/23312_lores.jpg\n", + "Resolving phil.cdc.gov (phil.cdc.gov)... 198.246.102.26\n", + "Connecting to phil.cdc.gov (phil.cdc.gov)|198.246.102.26|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 31823 (31K) [image/jpeg]\n", + "Saving to: ‘example_image_covid.jpg’\n", + "\n", + "example_image_covid 100%[===================>] 31.08K --.-KB/s in 0.07s \n", + "\n", + "2024-01-11 05:24:55 (455 KB/s) - ‘example_image_covid.jpg’ saved [31823/31823]\n", + "\n" + ] + } + ], + "source": [ + "! wget -O example_image_covid.jpg \"https://phil.cdc.gov//PHIL_Images/23312/23312_lores.jpg\" " + ] + }, + { + "cell_type": "markdown", + "id": "696d9fc7-0ca9-44a7-b431-a9698b1a636c", + "metadata": {}, + "source": [ + "Now run our function!" + ] + }, + { + "cell_type": "code", + "execution_count": 180, + "id": "34e81656-4943-439d-9fbe-df439e0e30df", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " The image is a 3D rendering of a coronavirus. The virus is round and has a spiky outer coat. The spikes are made of proteins that help the virus attach to and infect cells. The virus is colored gray and red.None\n" + ] + } + ], + "source": [ + "print(img2text(\"example_image_covid.jpg\", \"describe this image.\"))" + ] + }, + { + "cell_type": "markdown", + "id": "dc152659-7bc4-46bc-9ead-4736b4ad2706", + "metadata": {}, + "source": [ + "Next we'll look at an image from a bucket." + ] + }, + { + "cell_type": "code", + "execution_count": 181, + "id": "81197d53-dd3d-4358-9835-ef513ec11d33", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " The image shows a table with a pink peony bouquet, two cups of coffee, a bowl of blueberries, and a silver spoon with the words \"Let's Jam\" on it. There are also five scones with blueberries on them. The table is covered with a white tablecloth with purple stains.None\n" + ] + } + ], + "source": [ + "print(img2text(\"gs://generativeai-downloads/images/scones.jpg\", \"describe this image.\"))" + ] + }, + { + "cell_type": "markdown", + "id": "80e3c48f-66e9-40e1-8623-04f73b672507", + "metadata": {}, + "source": [ + "We can even ask for more details related to the items in our image!" + ] + }, + { + "cell_type": "code", + "execution_count": 182, + "id": "d8395784-ea68-4a95-a0bb-b3d618f68054", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " Preheat oven to 375 degrees F (190 degrees C). Grease a baking sheet.\n", + "\n", + "In a large bowl, combine the flour, sugar, baking powder, and salt. Cut in butter until mixture resembles coarse crumbs. Stir in blueberries.\n", + "\n", + "Turn out onto a lightly floured surface; knead for 10 to 12 times. Pat into a 1/2-in.-thick circle. Cut with a 3-in. floured biscuit cutter. Place 2 in. apart on the prepared baking sheet.\n", + "\n", + "Bake for 12-15 minutes or until golden brown. Cool for 2 minutes before removing to a wire rack to cool completely.None\n" + ] + } + ], + "source": [ + "img_prompt=\"How do you make whats in this image?\"\n", + "image=\"gs://generativeai-downloads/images/scones.jpg\"\n", + "print(img2text(image, img_prompt))" + ] + }, + { + "cell_type": "markdown", + "id": "960f7ba6-8236-437a-ba99-ea1d887efd64", + "metadata": {}, + "source": [ + "### Gemini as a Video to Text Generator" + ] + }, + { + "cell_type": "markdown", + "id": "fbdbbd35-f36f-4467-8a1d-48ccd942d7cc", + "metadata": {}, + "source": [ + "Just like images we will be using the same model Gemini Pro Vision. We can load videos locally and from a bucket just like images. Video files can only be in the following formats:\n", + "- MOV - video/mov\n", + "- MPEG - video/mpeg\n", + "- MP4 - video/mp4\n", + "- MPG - video/mpg\n", + "- AVI - video/avi\n", + "- WMV - video/wmv\n", + "- MPEGPS - video/mpegps\n", + "- FLS - video/flv" + ] + }, + { + "cell_type": "markdown", + "id": "a7f94141-b5ba-49bf-a3d7-d0b68ccdd39a", + "metadata": {}, + "source": [ + "Our function below takes a video from a public bucket and asks for a prompt and the location of the video file." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c18561c6-8f8f-46e4-b3ee-0d5fc96f2d31", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def video2text(video_path: str, video_prompt: str) -> str:\n", + " # Query the model\n", + " response = multimodal_model.generate_content(\n", + " [\n", + " # Add an example image\n", + " Part.from_uri(\n", + " video_path, mime_type=\"video/mp4\"\n", + " ),\n", + " # Add an example query\n", + " video_prompt,\n", + " ],\n", + " stream=True\n", + " )\n", + " for chunk in response :\n", + " return print(chunk.text)\n" + ] + }, + { + "cell_type": "markdown", + "id": "492f9100-b5a4-446c-a03d-dd89a1b4bbde", + "metadata": {}, + "source": [ + "Run the function!" + ] + }, + { + "cell_type": "code", + "execution_count": 191, + "id": "55990074-0365-45f5-9fa6-bedbe93c9932", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " This video is about a messy world. It shows a bunch of different things that are messy, like a messy room, a messy desk, and a messy\n", + "None\n" + ] + } + ], + "source": [ + "video_prompt=\"What is this video about in detail?\"\n", + "video=\"gs://cloud-samples-data/video/Machine Learning Solving Problems Big, Small, and Prickly.mp4\"\n", + "print(video2text(video, video_prompt))" + ] + }, + { + "cell_type": "markdown", + "id": "c32253b4-d892-4c0c-896e-320c8df479c7", + "metadata": {}, + "source": [ + "## Gemini on Vertex AI Studio" + ] + }, + { + "cell_type": "markdown", + "id": "614883f2-e53a-4ba5-855c-209d755a6e6f", + "metadata": {}, + "source": [ + "You can also use Gemini Pro and Pro Vision in Vertex AI's playground called **Vertex AI Studio**. To locate Vertex AI Studio search Vertex AI and on the left hand side locate Vertex AI Studio as the image below shows. To utilize Gemini Pro Vision locate and click **Multimodal** you will have the option to use your own prompt or explore some of the other set prompts such as Extract text from images, image question answering , etc." + ] + }, + { + "cell_type": "markdown", + "id": "8a634e11-3bd5-4048-b6e4-46d5aac6ce34", + "metadata": {}, + "source": [ + "![Gemini1](../../../images/Gemini_1.png)" + ] + }, + { + "cell_type": "markdown", + "id": "c3716d9d-49af-40c9-acce-757c53be9a12", + "metadata": {}, + "source": [ + "For this tutorial we will select Open on the **Prompt Design** option. We will upload the COVID image we downloaded before by clicking **INSERT MEDIA** and selecting our file. Then we will ask it a question, here we asked \"Describe treatments for the item in this image\"." + ] + }, + { + "cell_type": "markdown", + "id": "ec72f9cd-0d06-47e4-b67c-45039372d967", + "metadata": { + "tags": [] + }, + "source": [ + "![Gemini3](../../../images/Gemini_3.png)" + ] + }, + { + "cell_type": "markdown", + "id": "4a88f06b-770a-43be-b875-b4054c3ccbf3", + "metadata": {}, + "source": [ + "To utilize Gemini Pro locate and click **Language** on the left side menu. You have the option to use a prompt or chat and if you would like to focus on text or code." + ] + }, + { + "cell_type": "markdown", + "id": "1952cd5c-c93d-428f-8036-ac658eebfba4", + "metadata": { + "tags": [] + }, + "source": [ + "![Gemini2](../../../images/Gemini_2.png)" + ] + }, + { + "cell_type": "markdown", + "id": "7810c0db-d145-4acd-a2cc-b6ce8231fa14", + "metadata": {}, + "source": [ + "Here we picked the **TEXT CHAT** option and asked the bot to describe covid and how it works." + ] + }, + { + "cell_type": "markdown", + "id": "36b4635d-d8f3-42df-959d-dff92259813c", + "metadata": { + "tags": [] + }, + "source": [ + "![Gemini4](../../../images/Gemini_4.png)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "acc53b99-e6ed-45f1-a0e1-9b146e18e46c", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "environment": { + "kernel": "python3", + "name": "common-cpu.m114", + "type": "gcloud", + "uri": "gcr.io/deeplearning-platform-release/base-cpu:m114" + }, + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}