SOP: Provisioning Cloud GPU Workstation

[dan-homebrew]

Goal

• We have our own "GPU Cloud", and contributors can spin up their own "Desktop" to develop or test Jan or cortex.cpp
• We have our own test cloud with common OS and hardware types

[imtuyethan]

Overview

We maintain a private GPU cluster for developing and testing Jan (our Personal AI assistant) and cortex.cpp (our Local AI engine), as well as for our research team to train models. This setup allows our development and research teams to work efficiently with consistent, high-performance environments.

Use Case Highlights

• Supports development of Jan & Cortex.cpp
• Enables rapid prototyping and testing of AI models across various hardware configurations
• Powers our research team's model training efforts, including:

Fine-tuning LLMs
Experimenting with AI architectures
Developing more efficient, on-device AI models

Security Considerations

• Access to the cloud GPU workstations requires the company's VPN
• The current infrastructure is limited and primarily for internal team usage
• Network and firewall reorganization for internet-exposed VMs is needed
• Regular security audits and updates
• Strict access control and monitoring

Future Scalability

While currently for internal use, our setup is designed with scalability in mind:

• Ability to quickly adapt to new hardware or increased demand
• Potential to support external contributors or QA testers in the future
• Potential interest from partners in our process

We're open to discussions on how we can collaborate to enhance and scale this infrastructure, particularly in ways that could accelerate our research and development processes.

Provisioning Process (for Internal Use)

1. Request Initiation

• Team member submits request through Intra Github repo ticketing system: https://github.com/orgs/homebrewltd/projects/1
• Infra team reviews and approves based on project needs

2. Resource Allocation & Setup

@hiento09 to document

• Configure networking (VPN-accessible, grand permissions...)
• Assign virtual machines from our GPU cluster
• Further guidance if needed

We ensure a flexible resource allocation to balance between development and research needs.

Workstation Specifications

Window

1. 108 (windows10-test-app)

OS: Windows 10 Pro (Version 22H2, build: 19045.4780)
CPU: Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
RAM: 16.0 GB
System type: 64-bit operating system, x64-based processor

2. 114 (windows-dev-tensorRT-llm)

OS: Windows 11 Pro (Version 23H2, build 22631.4037)
CPU: AMD Ryzen Threadripper PRO 5955WX (16 cores)
RAM: 32 GB
GPU: NVIDIA GeForce RTX 3090
Storage: 599 GB local disk (C:)

3. 118 (test-windows-app-1)

OS: Windows 11 Pro (Version 23H2, build: 22631.4037)
CPU: Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
RAM: 16.0 GB
System type: 64-bit operating system, x64-based processor
Hardware Model: Standard PC (Q35 + ICH9, 2009)

4. 119 (test-windows-app-2)

Processor: Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz 2.40 GHz
CPU: Intel Xeon Silver 4214R @ 2.40GHz
RAM: 96GB
OS: 64-bit operating system
Architecture: x64-based processor
GPU: AMD Radeon RX 7600 (but Jan does not support)

Linux

1. 107 (test-ubuntu-app-cpu-2)

OS: Ubuntu 22.04.4 LTS (64-bit)
CPU: Intel Xeon(R) Silver 4214R @ 2.40GHz (16 cores)
RAM: 256.0 GiB
GPU: llvmpipe (LLVM 15.0.7, 256 bits)
Storage: 429.5 GB

2. 121 (test-ubuntu-app-cpu-1)

OS: Ubuntu 22.04.5 LTS (Jammy Jellyfish)
CPU: 2x Intel(R) Xeon(R) Silver 4214 processors
RAM: 16GiB System Memory
Storage: 268GB QEMU HARDDISK

3. 117 (test-ubuntu-2404-1)

Ubuntu server

4. 136 (test-ubuntu-20.04-3)

Ubuntu server