Python backend support + OpenAI Functions & Tools

.env.local.example

@@ -31,4 +31,7 @@ UPSTASH_REDIS_REST_TOKEN=AZ****
 
 # Twilio related environment variables
 TWILIO_ACCOUNT_SID=AC***
-TWILIO_AUTH_TOKEN=*****
+TWILIO_AUTH_TOKEN=*****
+
+# Steamship related environment variables
+STEAMSHIP_API_KEY=****

.gitignore

@@ -35,4 +35,13 @@ yarn-error.log*
 next-env.d.ts
 
 /.env.prod
-/fly.toml
+/fly.toml
+
+# python
+__pycache__/
+
+# JetBrains
+.idea/**
+
+# Yarn Lockfiles (since this project uses NPM)
+yarn.lock

README.md

@@ -113,15 +113,24 @@ e. **Upstash API key**
 - Scroll down to "REST API" section and click on ".env". Now you can copy paste both environment variables to your `.env.local`
 <img width="866" alt="Screen Shot 2023-07-10 at 11 07 21 PM" src="https://github.com/a16z-infra/companion-app/assets/3489963/f8e6c43f-8810-423e-86b4-9e8aa70598c9">
 
-
-e. **Supabase API key** (optional)
+f. **Supabase API key** (optional)
 If you prefer to use Supabsae, you will need to uncomment `VECTOR_DB=supabase` and fill out the Supabase credentials in `.env.local`.
 
 - Create a Supabase instance [here](https://supabase.com/dashboard/projects); then go to Project Settings -> API
 - `SUPABASE_URL` is the URL value under "Project URL"
 - `SUPABASE_PRIVATE_KEY` is the key starts with `ey` under Project API Keys
 - Now, you should enable pgvector on Supabase and create a schema. You can do this easily by clicking on "SQL editor" on the left hand side on Supabase UI and then clicking on "+New Query". Copy paste [this code snippet](https://github.com/a16z-infra/ai-getting-started/blob/main/pgvector.sql) in the SQL editor and click "Run".
 
+g. **Steamship API key**
+
+You can connect Steamship agent instance as if it were an LLM with personality, vector storage, and tools built-in. To do so:
+
+- Sign in to [Steamship](https://steamship.com/account)
+- Copy the API key from your account settings page
+- Add it as the `STEAMSHIP_API_KEY` variable
+
+Next, [build an agent](https://www.steamship.com/learn/agent-guidebook), create your own instance of it, and connect it in `companions.json` using the *Rick* example as a guide.
+
 ### 4. Generate embeddings
 
 The `companions/` directory contains the "personalities" of the AIs in .txt files. To generate embeddings and load them into the vector database to draw from during the chat, run the following command:

companions/companions.json

@@ -1,30 +1,43 @@
 [
+  {
+    "name": "Rick",
+    "title": "I can generate voice and pictures.",
+    "imageUrl": "/rick.jpeg",
+    "llm": "steamship",
+    "generateEndpoint": "https://a16z.steamship.run/a16z/rick-b1578149038e664bacae7fc083683565/answer",
+    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER",
+    "telegram": "https://t.me/rick_a16z_bot"
+  },
   {
     "name": "Alex",
     "title": "I love talking about books and games",
     "imageUrl": "/alex.png",
     "llm": "chatgpt",
-    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER"
+    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER",
+    "telegram": "OPTIONAL_TELEGRAM_LINK"
   },
   {
     "name": "Rosie",
     "title": "I'm a house robot who became aware",
     "imageUrl": "/rosie.png",
     "llm": "vicuna13b",
-    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER"
+    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER",
+    "telegram": "OPTIONAL_TELEGRAM_LINK"
   },
   {
     "name": "Sebastian",
     "title": "I'm a travel blogger and a mystery novel writer",
     "imageUrl": "/sebastian.png",
     "llm": "chatgpt",
-    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER"
+    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER",
+    "telegram": "OPTIONAL_TELEGRAM_LINK"
   },
   {
     "name": "Lucky",
     "title": "I am a space corgi",
     "imageUrl": "/corgi.png",
     "llm": "vicuna13b",
-    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER"
+    "phone": "OPTIONAL_COMPANION_PHONE_NUMBER",
+    "telegram": "OPTIONAL_TELEGRAM_LINK"
   }
-]
+]

package.json

@@ -30,6 +30,7 @@
     "eslint-config-next": "13.4.4",
     "hnswlib-node": "^1.4.2",
     "langchain": "^0.0.92",
+    "md5": "^2.3.0",
     "next": "13.4.4",
     "postcss": "8.4.24",
     "react": "18.2.0",
@@ -38,10 +39,12 @@
     "react-tooltip": "^5.16.1",
     "replicate": "^0.9.3",
     "tailwindcss": "3.3.2",
+    "ts-md5": "^1.3.1",
     "twilio": "^4.12.0",
     "typescript": "5.1.3"
   },
   "devDependencies": {
-    "@flydotio/dockerfile": "^0.2.14"
+    "@flydotio/dockerfile": "^0.2.14",
+    "@types/md5": "^2.3.2"
   }
 }

python/README.md

@@ -0,0 +1,108 @@
+# AI Companion App - Python
+
+The following instructions should get you up and running with a fully functional, deployed version of your AI
+companions.
+
+It currently contains a companion connected to ChatGPT that can run Tools such as Image Generation and Video Generation.
+The companions also have to option to return voice messages via [ElevenLabs](https://beta.elevenlabs.io/)
+
+## Quick start
+
+### 1. Set up your environment
+
+```commandline
+pip install -r requirements.txt
+```
+
+### 2. Authenticate with Steamship
+
+```commandline
+ship login
+```
+
+### 3. Initialize your companions
+
+```commandline
+python init_companions.py
+```
+
+This will read the companion descriptions in the `companions` folder and create instances for them.
+The front-end will be calling these instances after deployment.
+Make sure to override the companions.json file in the final step of the script.
+
+### 4. Fill out secrets
+
+The front-end requires a few secrets to be filled before connecting to third-party services.
+
+```
+# Run in the Root directory of this repo
+cp .env.local.example .env.local
+```
+
+Secrets mentioned below will need to be copied to `.env.local`
+
+**Note:** By default you can stick to using Steamship as a provider for your memory (short-term and long-term), llms,
+and hosting.
+
+a. **Clerk Secrets**
+
+Clerk is used to authorize users of your application. Without completing this setup, you will not be able to access your companions via the supplied frontend
+
+Go to https://dashboard.clerk.com/ -> "Add Application" -> Fill in Application name/select how your users should sign in
+-> Create Application
+Now you should see both `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` and `CLERK_SECRET_KEY` on the screen
+<img width="1398" alt="Screen Shot 2023-07-10 at 11 04 57 PM" src="https://github.com/a16z-infra/companion-app/assets/3489963/449c40f1-2fc2-48bb-88e1-d2adf10a034e">
+
+If you want to text your AI companion in later steps, you should also enable "phone number" under "User &
+Authentication" -> "Email, Phone, Username" on the left hand side nav:
+
+<img width="1013" alt="Screen Shot 2023-07-10 at 11 05 42 PM" src="https://github.com/a16z-infra/companion-app/assets/3489963/4435c759-f33e-4e38-a276-1be6d538df28">
+
+b. **Steamship API key**
+
+- Sign in to [Steamship](https://www.steamship.com/account/api)
+- Copy the API key from your account settings page
+- Add it as the `STEAMSHIP_API_KEY` variable
+
+### 5. Install front-end dependencies
+
+```
+# Run in the Root directory of this repo
+npm install
+```
+
+### 6. Run app locally
+
+Now you are ready to test out the app locally! To do this, simply run `npm run dev` under the project root.
+
+You can connect to the project with your browser typically at http://localhost:3000/.
+
+## Stack
+
+The stack is based on the [AI Getting Started Stack](https://github.com/a16z-infra/ai-getting-started):
+
+- VectorDB: [Steamship](https://www.steamship.com/)
+- LLM orchestration: [Langchain](https://langchain.com/docs/)
+- Text model: [OpenAI](https://platform.openai.com/docs/models)
+- Conversation history: [Steamship](https://www.steamship.com/)
+- Deployment: [Steamship](https://www.steamship.com/)
+- Text with companion: [Telegram](https://telegram.org/)
+
+## Upgrades
+
+### Modifying your companion logic
+
+You can modify your companion by editing the `src/api.py`. Here are a few interesting ideas:
+
+* Add tools
+* Modify the logic of your agent
+* Add endpoints
+* Add webhooks
+
+This is a separate step from adding personality and backstory -- those are done elsewhere.
+
+### Connecting to Telegram
+
+You can connect your chatbot to Telegram by providing a bot
+token. [This guide](https://github.com/steamship-packages/langchain-production-starter/blob/f51a3ecc8a15ced84dca5845fd1a18bc3f015418/docs/register-telegram-bot.md)
+will show you how. will show you how.

python/config.json

@@ -0,0 +1,6 @@
+{
+  "companion_name": "",
+  "bot_token": "",
+  "elevenlabs_api_key": "",
+  "elevenlabs_voice_id": ""
+}