Adding a Dockerfile for devcontainer (run-llama#43)

* add Dockerfile for devcontainer

* add nodejs installation

* more deps needed for node.js installation

* fix

* fix

* fix

* fix

* fix

* update README + add Codespace origin if running in codespaces

* use docker compose yaml

* rename file

* fix allow_origins set to include codespace origin

* use features + remove workspace compose yaml

* apt-get update

* remove npm & poetry install

* add nodejs feature

* address variety of more issues with running on codespaces + issues with make migate

* add one more step to backend readme

* remove loose console log

Author: sourabhdesai

.devcontainer/Dockerfile

@@ -0,0 +1,13 @@
+# https://hub.docker.com/_/python
+FROM python:3.11.3-slim-bullseye
+
+ENV PYTHONUNBUFFERED True
+# Install other backend deps
+RUN apt-get update
+RUN apt-get install libpq-dev gcc build-essential wkhtmltopdf s3fs -y
+RUN pip install poetry==1.6.1
+# Install frontend node modules
+ENV APP_HOME /app
+COPY . $APP_HOME
+
+CMD ["/bin/bash"]

.devcontainer/devcontainer.json

@@ -1,10 +1,15 @@
 {
-  "image": "mcr.microsoft.com/devcontainers/universal:2",
+  "name": "sec_insights",
+  "build": {
+    "dockerfile": "./Dockerfile",
+    "context": ".."
+  },
   "features": {
     "ghcr.io/devcontainers-contrib/features/pipx-package:1": {},
     "ghcr.io/devcontainers-contrib/features/poetry:2": {},
     "ghcr.io/warrenbuckley/codespace-features/sqlite:1": {},
     "ghcr.io/devcontainers/features/docker-in-docker:2": {},
-    "ghcr.io/devcontainers/features/aws-cli:1": {}
+    "ghcr.io/devcontainers/features/aws-cli:1": {},
+    "ghcr.io/devcontainers/features/node:1": {}
   }
 }

backend/Makefile

@@ -13,6 +13,8 @@ migrate:
 	docker compose create db
 	docker compose start db
 	poetry run python -m alembic upgrade head
+	# workaround for having PGVector create its tables
+	poetry run python -m scripts.build_vector_tables
 
 refresh_db:
 	# First ask for confirmation.

backend/README.md

@@ -3,7 +3,9 @@ Live at https://secinsights.ai/
 ## Setup Dev Workspace
 1. Install [pyenv](https://github.com/pyenv/pyenv#automatic-installer) and then use it to install the Python version in `.python-version`.
     1. install pyenv with `curl https://pyenv.run | bash`
+    * This step can be skipped if you're running from the devcontainer image in Github Codespaces
 1. [Install docker](https://docs.docker.com/engine/install/)
+    * This step can be skipped if you're running from the devcontainer image in Github Codespaces
 1. Run `poetry shell`
 1. Run `poetry install` to install dependencies for the project
 1. Create the `.env` file and source it. The `.env.development` file is a good template.
@@ -17,6 +19,13 @@ Live at https://secinsights.ai/
     - This spins up the Postgres 15 DB & Localstack in their own docker containers.
     - The server will not run in a container but will instead run directly on your OS.
         - This is to allow for use of debugging tools like `pdb`
+1. Lastly, you will likely want to populate your local database with some sample SEC filings
+    - We have a script for this! But first, open your `.env` file and replace the placeholder values for the `OPENAI_API_KEY` with your own OpenAI API key
+        - At some point you will want to do the same for the other secret keys in here like `POLYGON_IO_API_KEY`, `AWS_KEY`, & `AWS_SECRET`
+    - Source the file again with `set -a` then `source .env`
+    - Run `make seed_db_local`
+        - If this step fails, you may find it helpful to run `make refresh_db` to wipe your local database and re-start with emptied tables.
+    - Done 🏁! You can run `make run` again and you should see some documents loaded at http://localhost:8000/api/document
 
 ## Scripts
 The `scripts/` folder contains several scripts that are useful for both operations and development.
@@ -71,23 +80,28 @@ These steps assume you've already followed the steps above for setting up your d
 
 1. Setup AWS CLI
     1. Install AWS CLI
-        - `curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"`
-        - `unzip awscliv2.zip`
-        - `sudo ./aws/install`
+        - This step can be skipped if you're running from the devcontainer image in Github Codespaces
+        - Steps:
+            - `curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"`
+            - `unzip awscliv2.zip`
+            - `sudo ./aws/install`
     1. Configure AWS CLI
         - This is mainly to set the AWS credentials that will later be used by s3fs
         - Run `aws configure` and enter the access key & secret key for a AWS IAM user that has access to the PDFs where you want to store the SEC files.
             - set the default AWS region to `us-east-1` (what we're primarily using).
 1. Setup [`s3fs`](https://github.com/s3fs-fuse/s3fs-fuse)
     1. Install s3fs
+        - This step can be skipped if you're running from the devcontainer image in Github Codespaces
         - `sudo apt install s3fs`
     1. Setup a s3fs mounted folder
         - Create the mounted folder locally `mkdir ~/mounted_folder`
         - `s3fs llama-app-web-assets-preview ~/mounted_folder`
             - You can replace `llama-app-web-assets-preview` with the name of the S3 bucket you want to upload the files to.
 1. Install [`wkhtmltopdf`](https://wkhtmltopdf.org/)
-    - `sudo apt-get update`
-    - `sudo apt-get install wkhtmltopdf`
+    - This step can be skipped if you're running from the devcontainer image in Github Codespaces
+    - Steps:
+        - `sudo apt-get update`
+        - `sudo apt-get install wkhtmltopdf`
 1. Get into your poetry shell with `poetry shell` from the project's root directory.
 1. Run the script! `python scripts/download_sec_pdf.py -o ~/mounted_folder --file-types="['10-Q','10-K']"`
     - Take a 🚽 break while it's running, it'll take a while!

backend/app/core/config.py

@@ -70,6 +70,8 @@ class Settings(PreviewPrefixedSettings):
     LOG_LEVEL: str = "DEBUG"
     IS_PULL_REQUEST: bool = False
     RENDER: bool = False
+    CODESPACES: bool = False
+    CODESPACE_NAME: Optional[str]
     S3_BUCKET_NAME: str
     S3_ASSET_BUCKET_NAME: str
     CDN_BASE_URL: str

backend/app/main.py

@@ -97,10 +97,15 @@ async def lifespan(app: FastAPI):
 
 
 if settings.BACKEND_CORS_ORIGINS:
+    origins = settings.BACKEND_CORS_ORIGINS.copy()
+    if settings.CODESPACES and settings.CODESPACE_NAME and \
+        settings.ENVIRONMENT == AppEnvironment.LOCAL:
+        # add codespace origin if running in Github codespace
+        origins.append(f"https://{settings.CODESPACE_NAME}-3000.app.github.dev")
     # allow all origins
     app.add_middleware(
         CORSMiddleware,
-        allow_origins=settings.BACKEND_CORS_ORIGINS,
+        allow_origins=origins,
         allow_origin_regex="https://llama-app-frontend.*\.vercel\.app",
         allow_credentials=True,
         allow_methods=["*"],

backend/scripts/build_vector_tables.py

@@ -0,0 +1,19 @@
+from fire import Fire
+from app.schema import Document
+from app.db.session import SessionLocal
+from app.chat.pg_vector import get_vector_store_singleton
+import asyncio
+
+async def build_vector_tables():
+    vector_store = await get_vector_store_singleton()
+    await vector_store.run_setup()
+
+
+def main_build_vector_tables():
+    """
+    Script to build the PGVector table if they don't already exist
+    """
+    asyncio.run(build_vector_tables())
+
+if __name__ == "__main__":
+    Fire(main_build_vector_tables)

frontend/src/config.js

@@ -1,3 +1,11 @@
 import { env } from "~/env.mjs";
+
+if (env.NEXT_PUBLIC_CODESPACES === 'true' && env.NEXT_PUBLIC_CODESPACE_NAME) {
+    const suggestedUrl = `https://${env.NEXT_PUBLIC_CODESPACE_NAME}-8000.app.github.dev/`;
+    if (!env.NEXT_PUBLIC_BACKEND_URL.startsWith(suggestedUrl)) {
+        console.warn(`It looks like you're running on a Github codespace. You may want to set the NEXT_PUBLIC_BACKEND_URL environment variable to ${suggestedUrl}`);
+    }
+}
+
 export const backendUrl = env.NEXT_PUBLIC_BACKEND_URL;
 

frontend/src/env.mjs

@@ -17,6 +17,8 @@ export const env = createEnv({
    */
   client: {
     NEXT_PUBLIC_BACKEND_URL: z.string().min(1),
+    NEXT_PUBLIC_CODESPACES: z.string().default("false").optional(),
+    NEXT_PUBLIC_CODESPACE_NAME: z.string().optional(),
   },
 
   /**
@@ -26,6 +28,8 @@ export const env = createEnv({
   runtimeEnv: {
     NODE_ENV: process.env.NODE_ENV,
     NEXT_PUBLIC_BACKEND_URL: process.env.NEXT_PUBLIC_BACKEND_URL,
+    NEXT_PUBLIC_CODESPACES: process.env.CODESPACES,
+    NEXT_PUBLIC_CODESPACE_NAME: process.env.CODESPACE_NAME,
   },
   /**
    * Run `build` or `dev` with `SKIP_ENV_VALIDATION` to skip env validation.