Judgment CLI

The Judgment CLI is a command-line tool that allows you to manage your Judgment resources and backend infrastructure.

Detailed documentation for this CLI can also be found here.

Installation

⚠️ Make sure you have Python installed on your system before proceeding with the installation.

To install the Judgment CLI, follow these steps:

1. Clone the repository:

   git clone https://github.com/JudgmentLabs/judgment-cli.git

2. Navigate to the project directory:

   cd judgment-cli

3. Set up a fresh Python virtual environment:

   Choose one of the following methods:

   Built-In venv

   python -m venv venv
   source venv/bin/activate  # On Windows, use: venv\Scripts\activate

   pipenv

   pipenv shell

   uv

   uv venv
   source .venv/bin/activate  # On Windows, use: .venv\Scripts\activate

4. Install the package:

   Built-In

   pip install -e .

   pipenv

   pipenv install -e .

   uv

   uv pip install -e .

Verifying the Installation

To verify that the CLI was installed correctly, run:

judgment --help

You should see a list of available commands and their descriptions.

Available Commands

The Judgment CLI provides the following commands:

Self-Hosting Commands

Command	Description
judgment self-host main	Deploy a self-hosted instance of Judgment (and optionally set up the HTTPS listener)
judgment self-host https-listener	Set up the HTTPS listener for a self-hosted Judgment instance

---

Self-Hosting

⚠️ If you are setting up self-hosting for the first time, please read the self-hosting documentation here before you get started with this section!

⚠️ Make sure the Judgment CLI is installed before proceeding.
Please refer to the Installation section above for more information.

⚠️ Do not delete .tfstate files generated during setup. They are used by Terraform to track the state of deployed infrastructure.

Introduction

The self-host command is used to deploy and manage your own self-hosted instance of Judgment.

• self-host main deploys a Supabase project and the Judgment AWS infrastructure.
• self-host https-listener configures an HTTPS listener for your self-hosted Judgment instance and is only needed if you skip the listener setup in the main command.

Usage

To see usage information, run:

judgment self-host --help
judgment self-host main --help
judgment self-host https-listener --help

1. Prerequisites

⚠️ Before proceeding, ensure you have:

1. An empty AWS account registered with us
2. An Osiris API key (optional, for evaluations)
3. A valid email address and app password (for sending invites)
4. A Supabase account and organization with admin access

As mentioned above, please read and follow the documentation here if any of the above aren't set up.

AWS CLI Setup

Install the AWS CLI:

macOS

brew install awscli

Windows

Download and run the installer from AWS CLI MSI

Linux

sudo apt install awscli

After installation, configure your local environment with the relevant AWS credentials:

aws configure

Terraform CLI Setup

Terraform is required for AWS infrastructure deployment.

macOS

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

Windows

choco install terraform

Linux

Follow: Terraform Install Guide

2. Deploying

1. Create a creds.json file with the following structure:

{
  "supabase_token": "your_supabase_personal_access_token_here",
  "org_id": "your_supabase_organization_id_here",
  "db_password": "your_desired_supabase_database_password_here",
  "invitation_sender_email": "email_address_to_send_org_invitations_from",
  "invitation_sender_app_password": "app_password_for_invitation_sender_email",
  "osiris_api_key": "your_osiris_api_key_here (optional)",
  "openai_api_key": "your_openai_api_key_here (optional)",
  "togetherai_api_key": "your_togetherai_api_key_here (optional)",
  "anthropic_api_key": "your_anthropic_api_key_here (optional)"
}

supabase_token: Use an existing one or generate a new one here

org_id: Extract this from the URL of your Supabase dashboard (make sure you have the correct organization selected in the top left corner). For example, if your organization URL is https://supabase.com/dashboard/org/uwqswwrmmkxgrkfjkdex, then your org_id is uwqswwrmmkxgrkfjkdex

db_password can be any password of your choice. It is necessary for creating the Supabase project and can be used later to directly connect to the project database

invitation_sender_email and invitation_sender_app_password are required because the only way to add users to the self-hosted Judgment instance is via email invitations

💡 The four LLM API keys are optional. If you are not planning to run evaluations with the models that require any of these API keys, you do not need to specify them.

2. Run the main self-host command. The command syntax is:

judgment self-host main [OPTIONS]

Required options:

• --root-judgment-email or -e: Email address for the root Judgment user
• --root-judgment-password or -p: Password for the root Judgment user
• --domain-name or -d: Domain name to request SSL certificate for (make sure you own this domain)

Optional options:

⚠️ Supabase free tier only supports nano for --supabase-compute-size. Larger sizes require a paid plan.

• --creds-file or -c: Path to credentials file (default: creds.json)
• --supabase-compute-size or -s: Size of the Supabase compute instance (default: small)
  • Available sizes: nano, micro, small, medium, large, xlarge, 2xlarge, 4xlarge, 8xlarge, 12xlarge, 16xlarge
• --invitation-email-service or -i: Email service for sending organization invitations (default: gmail)
  • Available services: gmail, outlook, yahoo, zoho, fastmail

Example usage:

judgment self-host main \
  --root-judgment-email [email protected] \
  --root-judgment-password password \
  --domain-name api.example.com \
  --creds-file creds.json \
  --supabase-compute-size nano \
  --invitation-email-service gmail

What this command does:

1. Creates a new Supabase project
2. Sets up the root Judgment user
3. Deploys AWS infrastructure
4. Connects AWS services to Supabase
5. Requests an SSL certificate
6. Optionally configures the HTTPS listener

📘 For SSL, you'll be given two DNS records to manually add to your DNS registrar.

🕒 After steps 1-5, you will be prompted to either continue with the HTTPS listener setup now or to come back later. If you choose to proceed with the setup now, the program will wait for the certificate to be issued before continuing.

3. Setting up the HTTPS Listener

⚠️ This step is optional; you can choose to have the HTTPS listener setup done as part of the main self-host command.

⚠️ This command will only work after judgment self-host main has already been run AND the ACM certificate has been issued. To accomplish this:

1. Add the two DNS records returned by the main self-host command to your DNS registrar/service
2. Monitor the ACM console here until the certificate has status ‘Issued’

After your ACM SSL certificate has been issued, set up the HTTPS listener by running the following:

judgment self-host https-listener

This command:

1. Sets up an HTTPS listener with your ACM-issued SSL certificate
2. Returns the final HTTPS endpoint of your self-hosted Judgment server