From 50e09f3b0e45e1d72fa9bfdd8997dc80b721941f Mon Sep 17 00:00:00 2001 From: peterschmidt85 Date: Wed, 15 May 2024 18:54:31 +0200 Subject: [PATCH] [Docs] Removed repeating examples; improved reference pages; updated the home page. --- README.md | 2 +- docker/server/README.md | 4 +- docs/assets/stylesheets/extra.css | 2 +- docs/assets/stylesheets/landing.css | 11 +- docs/docs/concepts/dev-environments.md | 36 +- docs/docs/concepts/pools.md | 12 +- docs/docs/concepts/services.md | 197 ++------ docs/docs/concepts/tasks.md | 170 +------ docs/docs/index.md | 31 +- docs/docs/installation/index.md | 548 +-------------------- docs/docs/quickstart.md | 4 +- docs/docs/reference/dstack.yml/service.md | 50 +- docs/docs/reference/dstack.yml/task.md | 13 +- docs/docs/reference/server/config.yml.md | 568 ++++++++++++++++------ docs/overrides/home.html | 320 ++++++------ docs/overrides/landing.html | 2 +- docs/overrides/main.html | 2 +- mkdocs.yml | 7 +- 18 files changed, 740 insertions(+), 1239 deletions(-) diff --git a/README.md b/README.md index 2e239640e..8838ce80c 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Orchestrate AI workloads in any cloud

Docs • -Examples • +Examples • Discord

diff --git a/docker/server/README.md b/docker/server/README.md index 898ef7083..8e8cbef83 100644 --- a/docker/server/README.md +++ b/docker/server/README.md @@ -94,8 +94,8 @@ _**️Note:** The use of Litestream requires that only one instance of the dstac For additional information and examples, see the following links: * [Docs](https://dstack.ai/docs/) -* [Examples](https://github.com/dstackai/dstack/blob/master/examples/README.md) -* [Changelog](https://dstack.ai/changelog) +* [Examples](https://github.com/dstackai/dstack/tree/master/examples) +* [Changelog](https://github.com/dstackai/dstack/releases) * [Discord](https://discord.gg/u8SmfwPpMd) ## Licence diff --git a/docs/assets/stylesheets/extra.css b/docs/assets/stylesheets/extra.css index 851990cd4..5945f4871 100644 --- a/docs/assets/stylesheets/extra.css +++ b/docs/assets/stylesheets/extra.css @@ -1402,7 +1402,7 @@ html .md-footer-meta.md-typeset a:is(:focus,:hover) { [dir=ltr] .md-typeset blockquote { border: 1px solid black; color: var(--md-default-fg-color); - padding: 0 20px; + padding: 8px 25px; border-radius: 12px; } diff --git a/docs/assets/stylesheets/landing.css b/docs/assets/stylesheets/landing.css index b9061fcdf..8a6e3b9d3 100644 --- a/docs/assets/stylesheets/landing.css +++ b/docs/assets/stylesheets/landing.css @@ -666,10 +666,10 @@ } .plans_card.dstack_sky { + background: -webkit-linear-gradient(45deg, rgba(0, 42, 255, 0.2), rgb(0 114 255 / 1%), rgba(0, 42, 255, 0.07)); } .plans_card.enterprise { - background: -webkit-linear-gradient(45deg, rgba(0, 42, 255, 0.0125), rgb(0 114 255 / 1%), rgba(0, 42, 255, 0.07)); } .plans_card { @@ -688,15 +688,18 @@ border-right: none; border-top-left-radius: 12px; border-bottom-left-radius: 12px; + margin-right: 5px; } .plans_card.dstack_sky { - } - - .plans_card.enterprise { border-left: none; border-top-right-radius: 12px; border-bottom-right-radius: 12px; + margin-left: 5px; + } + + .plans_card.enterprise { + margin: 0 -5px; } .tx-landing__plans_cards { diff --git a/docs/docs/concepts/dev-environments.md b/docs/docs/concepts/dev-environments.md index 7456df2e1..14dd722ca 100644 --- a/docs/docs/concepts/dev-environments.md +++ b/docs/docs/concepts/dev-environments.md @@ -26,37 +26,11 @@ resources: -> See the [.dstack.yml reference](../reference/dstack.yml/dev-environment.md) for more details. - If you don't specify your Docker image, `dstack` uses the [base](https://hub.docker.com/r/dstackai/base/tags) image (pre-configured with Python, Conda, and essential CUDA drivers). -### Environment variables - -Environment variables can be set either within the configuration file or passed via the CLI. - -```yaml -type: dev-environment - -env: - - HUGGING_FACE_HUB_TOKEN - - HF_HUB_ENABLE_HF_TRANSFER=1 - -python: "3.11" -ide: vscode - -resources: - gpu: 80GB -``` - -If you don't assign a value to an environment variable (see `HUGGING_FACE_HUB_TOKEN` above), -`dstack` will require the value to be passed via the CLI or set in the current process. - -For instance, you can define environment variables in a `.env` file and utilize tools like `direnv`. - -!!! info "Profiles" - In case you'd like to reuse certain parameters (such as spot policy, retry and max duration, - max price, regions, instance types, etc.) across runs, you can define them via [`.dstack/profiles.yml`](../reference/profiles.yml.md). +> See the [.dstack.yml reference](../reference/dstack.yml/dev-environment.md) +> for many examples on dev envioronment configuration. ## Running @@ -84,14 +58,14 @@ To open in VS Code Desktop, use this link: -When `dstack` provisions the dev environment, it uses the current folder contents. +When `dstack` provisions the dev environment, it mounts the project folder contents. !!! info ".gitignore" If there are large files or folders you'd like to avoid uploading, you can list them in `.gitignore`. -The `dstack run` command allows specifying many things, including spot policy, retry and max duration, -max price, regions, instance types, and [much more](../reference/cli/index.md#dstack-run). +> See the [CLI reference](../reference/cli/index.md#dstack-run) for more details +> on how `dstack run` works. ### VS Code diff --git a/docs/docs/concepts/pools.md b/docs/docs/concepts/pools.md index 7f46f7907..8c253cb5b 100644 --- a/docs/docs/concepts/pools.md +++ b/docs/docs/concepts/pools.md @@ -4,7 +4,7 @@ Pools enable the efficient reuse of cloud instances and on-premises servers acro ## Adding instances -### `dstack run` +### Automatic provisioning By default, when using the `dstack run` command, it tries to reuse an instance from a pool. If no idle instance meets the requirements, `dstack` automatically provisions a new cloud instance and adds it to the pool. @@ -18,7 +18,7 @@ requirements, `dstack` automatically provisions a new cloud instance and adds it This means that if the run is finished and the instance remains idle for longer than five minutes, it is automatically removed from the pool. To override the default idle duration, use `--idle-duration DURATION` with `dstack run`. -### `dstack pool add` +### Manual provisioning To manually provision a cloud instance and add it to a pool, use [`dstack pool add`](../reference/cli/index.md#dstack-pool-add): @@ -51,7 +51,7 @@ For more details on policies and their defaults, refer to [`.dstack/profiles.yml ??? info "Limitations" The `dstack pool add` command is not supported for Kubernetes, VastAI, and RunPod backends yet. -### `dstack pool add-ssh` +### Adding on-prem servers Any on-prem server that can be accessed via SSH can be added to a pool and used to run workloads. @@ -61,7 +61,7 @@ the SSH key.
```shell -$ dstack pool add-ssh ubuntu@54.73.155.119 -i ~/.ssh/id_rsa +$ dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu@54.73.155.119 ```
@@ -77,8 +77,6 @@ Once the instance is provisioned, you'll see it in the pool and will be able to If the instance remains idle for the configured idle duration, `dstack` removes it and deletes all cloud resources. -### `dstack pool remove` - To remove an instance from the pool manually, use the `dstack pool remove` command.
@@ -91,8 +89,6 @@ $ dstack pool remove <instance name> ## List instances -### `dstack pool ps` - The [`dstack pool ps`](../reference/cli/index.md#dstack-pool-ps) command lists active instances and their status (`busy` or `idle`). diff --git a/docs/docs/concepts/services.md b/docs/docs/concepts/services.md index 0f4b12433..69ed9f175 100644 --- a/docs/docs/concepts/services.md +++ b/docs/docs/concepts/services.md @@ -39,7 +39,7 @@ secure, and scalable endpoints. ## Configuration -First, create a YAML file in your project folder. Its name must end with `.dstack.yml` (e.g. `.dstack.yml` or `train.dstack.yml` +First, create a YAML file in your project folder. Its name must end with `.dstack.yml` (e.g. `.dstack.yml` or `serve.dstack.yml` are both acceptable).
@@ -47,144 +47,6 @@ are both acceptable). ```yaml type: service -# Specify the Docker image -image: ghcr.io/huggingface/text-generation-inference:latest - -# Specify environment variables -env: - - HUGGING_FACE_HUB_TOKEN # required to run gated models - - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.1 - -# The commands to run on start of the service -commands: - - text-generation-launcher --port 8000 --trust-remote-code - -# Specify the port of the service -port: 8000 - -# Specify GPU, disk, and other resource requirements -resources: - gpu: 80GB -``` - -
- -> See the [.dstack.yml reference](../reference/dstack.yml/service.md) for more details. - -If you don't specify your Docker image, `dstack` uses the [base](https://hub.docker.com/r/dstackai/base/tags) image -(pre-configured with Python, Conda, and essential CUDA drivers). - -### Environment variables - -Environment variables can be set either within the configuration file or passed via the CLI. - -```yaml -type: service - -image: ghcr.io/huggingface/text-generation-inference:latest - -env: - - HUGGING_FACE_HUB_TOKEN # required to run gated models - - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.1 - -commands: - - text-generation-launcher --port 8000 --trust-remote-code - -port: 8000 - -resources: - gpu: 80GB -``` - -If you don't assign a value to an environment variable (see `HUGGING_FACE_HUB_TOKEN` above), -`dstack` will require the value to be passed via the CLI or set in the current process. - -For instance, you can define environment variables in a `.env` file and utilize tools like `direnv`. - -### Model mapping - -By default, if you run a service, its endpoint is accessible at `https://.`. - -If you run a model, you can optionally configure the mapping to make it accessible via the -OpenAI-compatible interface. - -
- -```yaml -type: service - -image: ghcr.io/huggingface/text-generation-inference:latest -env: - - HUGGING_FACE_HUB_TOKEN # required to run gated models - - MODEL_ID=mistralai/Mistral-7B-Instruct-v0.1 -commands: - - text-generation-launcher --port 8000 --trust-remote-code -port: 8000 - -resources: - gpu: 80GB - -# Enable the OpenAI-compatible endpoint -model: - type: chat - name: mistralai/Mistral-7B-Instruct-v0.1 - format: tgi -``` - -
- -In this case, with such a configuration, once the service is up, you'll be able to access the model at -`https://gateway.` via the OpenAI-compatible interface. - -The `format` supports only `tgi` (Text Generation Inference) -and `openai` (if you are using Text Generation Inference or vLLM with OpenAI-compatible mode). - -??? info "Chat template" - - By default, `dstack` loads the [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating) - from the model's repository. If it is not present there, manual configuration is required. - - ```yaml - type: service - - image: ghcr.io/huggingface/text-generation-inference:latest - env: - - MODEL_ID=TheBloke/Llama-2-13B-chat-GPTQ - commands: - - text-generation-launcher --port 8000 --trust-remote-code --quantize gptq - port: 8000 - - resources: - gpu: 80GB - - # Enable the OpenAI-compatible endpoint - model: - type: chat - name: TheBloke/Llama-2-13B-chat-GPTQ - format: tgi - chat_template: "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<>\\n' + system_message + '\\n<>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' }}{% endif %}{% endfor %}" - eos_token: "" - ``` - - ##### Limitations - - Please note that model mapping is an experimental feature with the following limitations: - - 1. Doesn't work if your `chat_template` uses `bos_token`. As a workaround, replace `bos_token` inside `chat_template` with the token content itself. - 2. Doesn't work if `eos_token` is defined in the model repository as a dictionary. As a workaround, set `eos_token` manually, as shown in the example above (see Chat template). - - If you encounter any other issues, please make sure to file a [GitHub issue](https://github.com/dstackai/dstack/issues/new/choose). - -### Replicas and scaling - -By default, `dstack` runs a single replica of the service. -You can configure the number of replicas as well as the auto-scaling policy. - -
- -```yaml -type: service - python: "3.11" env: - MODEL=NousResearch/Llama-2-7b-chat-hf @@ -193,12 +55,10 @@ commands: - python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000 port: 8000 -replicas: 1..4 -scaling: - metric: rps - target: 10 +resources: + gpu: 80GB -# Enable the OpenAI-compatible endpoint +# (Optional) Enable the OpenAI-compatible endpoint model: format: openai type: chat @@ -207,14 +67,15 @@ model:
-If you specify the minimum number of replicas as `0`, the service will scale down to zero when there are no requests. +If you don't specify your Docker image, `dstack` uses the [base](https://hub.docker.com/r/dstackai/base/tags) image +(pre-configured with Python, Conda, and essential CUDA drivers). -[//]: # (??? info "Cold start") -[//]: # ( Scaling up from zero could take several minutes, considering the provisioning of a new instance and the pulling of a large model.) +By default, the service is deployed to a single instance. However, you can specify the +[number of replicas and scaling policy](../reference/dstack.yml/service.md#replicas-and-auto-scaling), so `dstack` +auto-scales it based on the load. -??? info "Profiles" - In case you'd like to reuse certain parameters (such as spot policy, retry and max duration, - max price, regions, instance types, etc.) across runs, you can define them via [`.dstack/profiles.yml`](../reference/profiles.yml.md). +> See the [.dstack.yml reference](../reference/dstack.yml/service.md) +> for many examples on service configuration. ## Running @@ -241,41 +102,47 @@ Service is published at https://yellow-cat-1.example.com
-When `dstack` submits the task, it uses the current folder contents. +When deploying the service, `dstack run` mounts the current folder's contents. -??? info ".gitignore" +!!! info ".gitignore" If there are large files or folders you'd like to avoid uploading, you can list them in `.gitignore`. -The `dstack run` command allows specifying many things, including spot policy, retry and max duration, -max price, regions, instance types, and [much more](../reference/cli/index.md#dstack-run). +> See the [CLI reference](../reference/cli/index.md#dstack-run) for more details +> on how `dstack run` works. ## Service endpoint One the service is up, its endpoint is accessible at `https://.`. -!!! info "Model endpoint" - In case the service has the [model mapping](#model-mapping) configured, you will also be able - to access the model at `https://gateway.` via the OpenAI-compatible interface. - -### Authorization - By default, the service endpoint requires the `Authorization` header with `"Bearer "`.
```shell -$ curl https://yellow-cat-1.example.com/generate \ - -X POST \ - -d '{"inputs":"<s>[INST] What is your favourite condiment?[/INST]"}' \ +$ curl https://yellow-cat-1.example.com/v1/chat/completions \ -H 'Content-Type: application/json' \ - -H 'Authorization: "Bearer <dstack token>"' + -H 'Authorization: "Bearer <dstack token>"' \ + -d '{ + "model": "NousResearch/Llama-2-7b-chat-hf", + "messages": [ + { + "role": "user", + "content": "Compose a poem that explains the concept of recursion in programming." + } + ] + }' ```
Authorization can be disabled by setting `auth` to `false` in the service configuration file. +### Model endpoint + +In case the service has the [model mapping](../reference/dstack.yml/service.md#model-mapping) configured, you will also be able +to access the model at `https://gateway.` via the OpenAI-compatible interface. + ## Managing runs **Stopping runs** @@ -290,4 +157,4 @@ The [`dstack ps`](../reference/cli/index.md#dstack-ps) command lists all running 1. Check the [Text Generation Inference :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/deployment/tgi/README.md){:target="_blank"} and [vLLM :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/deployment/vllm/README.md){:target="_blank"} examples 2. Check the [`.dstack.yml` reference](../reference/dstack.yml/service.md) for more details and examples -3. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/README.md){:target="_blank"} \ No newline at end of file +3. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/tree/master/examples){:target="_blank"} \ No newline at end of file diff --git a/docs/docs/concepts/tasks.md b/docs/docs/concepts/tasks.md index 732faf746..f65496cd5 100644 --- a/docs/docs/concepts/tasks.md +++ b/docs/docs/concepts/tasks.md @@ -15,64 +15,6 @@ are both acceptable). ```yaml type: task -# Specify the Python version, or your Docker image -python: "3.11" - -# Specify environment variables -env: - - HF_HUB_ENABLE_HF_TRANSFER=1 - -# The commands to run on start of the task -commands: - - pip install -r fine-tuning/qlora/requirements.txt - - python fine-tuning/qlora/train.py - -# Specify GPU, disk, and other resource requirements -resources: - gpu: 80GB -``` - - - -> See the [.dstack.yml reference](../reference/dstack.yml/task.md) for more details. - -If you don't specify your Docker image, `dstack` uses the [base](https://hub.docker.com/r/dstackai/base/tags) image -(pre-configured with Python, Conda, and essential CUDA drivers). - -### Environment variables - -Environment variables can be set either within the configuration file or passed via the CLI. - -```yaml -type: task - -python: "3.11" -env: - - HUGGING_FACE_HUB_TOKEN - - HF_HUB_ENABLE_HF_TRANSFER=1 -commands: - - pip install -r fine-tuning/qlora/requirements.txt - - python fine-tuning/qlora/train.py - -resources: - gpu: 80GB -``` - -If you don't assign a value to an environment variable (see `HUGGING_FACE_HUB_TOKEN` above), -`dstack` will require the value to be passed via the CLI or set in the current process. - -For instance, you can define environment variables in a `.env` file and utilize tools like `direnv`. - -### Ports - -A task can configure ports. In this case, if the task is running an application on a port, `dstack run` -will securely allow you to access this port from your local machine through port forwarding. - -
- -```yaml -type: task - python: "3.11" env: - HF_HUB_ENABLE_HF_TRANSFER=1 @@ -90,95 +32,15 @@ resources:
-When running it, `dstack run` forwards `6000` port to `localhost:6000`, enabling secure access. - -??? info "Port mapping" - By default, `dstack` uses the same ports on your local machine for port forwarding. However, you can override local ports using `--port`: - -
- - ```shell - $ dstack run . -f train.dstack.yml --port 6000:6001 - ``` - -
- - This will forward the task's port `6000` to `localhost:6001`. - -### Nodes - -By default, the task runs on a single node. However, you can run it on a cluster of nodes. - -
- -```yaml -type: task - -# The size of the cluster -nodes: 2 - -python: "3.11" -env: - - HF_HUB_ENABLE_HF_TRANSFER=1 -commands: - - pip install -r requirements.txt - - torchrun - --nproc_per_node=$DSTACK_GPUS_PER_NODE - --node_rank=$DSTACK_NODE_RANK - --nnodes=$DSTACK_NODES_NUM - --master_addr=$DSTACK_MASTER_NODE_IP - --master_port=8008 resnet_ddp.py - --num_epochs 20 - -resources: - gpu: 24GB -``` - -
- -If you run the task, `dstack` first provisions the master node and then runs the other nodes of the cluster. -All nodes are provisioned in the same region. - -??? info "Backends" - Running on multiple nodes is supported only with AWS, GCP, and Azure. - -### Args - -You can parameterize tasks with user arguments using `${{ run.args }}` in the configuration. - -
- -```yaml -type: task - -python: "3.11" -env: - - HF_HUB_ENABLE_HF_TRANSFER=1 -commands: - - pip install -r fine-tuning/qlora/requirements.txt - - python fine-tuning/qlora/train.py ${{ run.args }} - -resources: - gpu: 80GB -``` - -
- -Now, you can pass your arguments to the `dstack run` command: - -
- -```shell -$ dstack run . -f train.dstack.yml --train_batch_size=1 --num_train_epochs=100 -``` - -
+If you don't specify your Docker image, `dstack` uses the [base](https://hub.docker.com/r/dstackai/base/tags) image +(pre-configured with Python, Conda, and essential CUDA drivers). -The `dstack run` command will pass `--train_batch_size=1` and `--num_train_epochs=100` as arguments to `train.py`. +By default, the task runs on a single instance. However, you can specify the +[number of nodes](../reference/dstack.yml/task.md#_nodes) so `dstack` +provisions a cluster of instances. -??? info "Profiles" - In case you'd like to reuse certain parameters (such as spot policy, retry and max duration, - max price, regions, instance types, etc.) across runs, you can define them via [`.dstack/profiles.yml`](../reference/profiles.yml.md). +> See the [.dstack.yml reference](../reference/dstack.yml/task.md) +> for many examples on task configuration. ## Running @@ -200,6 +62,8 @@ Continue? [y/n]: y Provisioning... ---> 100% +TensorBoard 2.13.0 at http://localhost:6006/ (Press CTRL+C to quit) + Epoch 0: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969] Epoch 1: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969] Epoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969] @@ -207,22 +71,24 @@ Epoch 2: 100% 1719/1719 [00:18<00:00, 92.32it/s, loss=0.0981, acc=0.969] -When `dstack` submits the task, it uses the current folder contents. +If the task specifies `ports`, `dstack run` automatically forwards them to your local machine for +convenient and secure access. + +When running the task, `dstack run` mounts the current folder's contents. !!! info ".gitignore" If there are large files or folders you'd like to avoid uploading, you can list them in `.gitignore`. -The `dstack run` command allows specifying many things, including spot policy, retry and max duration, -max price, regions, instance types, and [much more](../reference/cli/index.md#dstack-run). +> See the [CLI reference](../reference/cli/index.md#dstack-run) for more details +> on how `dstack run` works. ## Managing runs **Stoping runs** -Once the run exceeds the max duration, -or when you use [`dstack stop`](../reference/cli/index.md#dstack-stop), -the task and its cloud resources are deleted. +Once you use [`dstack stop`](../reference/cli/index.md#dstack-stop) (or when the run exceeds the +`max_duration`), the instances return to the [pool](pools.md). **Listing runs** @@ -234,4 +100,4 @@ The [`dstack ps`](../reference/cli/index.md#dstack-ps) command lists all running 1. Check the [QLoRA :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/qlora/README.md){:target="_blank"} example 2. Check the [`.dstack.yml` reference](../reference/dstack.yml/task.md) for more details and examples -3. Browse [all examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/README.md){:target="_blank"} \ No newline at end of file +3. Browse [all examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/tree/master/examples){:target="_blank"} \ No newline at end of file diff --git a/docs/docs/index.md b/docs/docs/index.md index de86558c6..df4ca1a2a 100644 --- a/docs/docs/index.md +++ b/docs/docs/index.md @@ -5,22 +5,16 @@ > The supported cloud providers include AWS, GCP, Azure, Lambda, TensorDock, Vast.ai, CUDO, and RunPod. > You can also use `dstack` ro run workloads on on-prem servers. -Think of `dstack` as a lightweight alternative to Kubernetes or Slurm that provides a -straightforward interface for AI model development, training, and deployment. - -`dstack` supports dev environements, running tasks on clusters, and deployment with auto-scaling and built-in -authorization, all right out of the box. - -`dstack` is vendor-agnostic, allowing you to utilize any open-source libraries, frameworks, or tools -from your container. +`dstack` supports dev environements, running tasks on clusters, and deployment with auto-scaling and +authorization out of the box. ## Why use dstack? -1. Lightweight and easy-to-use compared to Kubernetes and Slurm -2. Supports all major GPU cloud providers -3. Scalable and reliable for production environments -4. Enables management of AI infrastructure across multiple clouds without vendor lock-in -5. Fully open-source +1. Simplifies development, training, and deployment of AI +2. Supports major GPU cloud providers as well as on-prem servers +3. Leverages the open-source AI ecosystem of libraries, frameworks, and models +4. Reduces GPU costs and improves workload efficiency +5. Allows the use of multiple cloud providers ## How does it work? @@ -28,17 +22,10 @@ from your container. 2. Define configurations such as [dev environments](concepts/dev-environments.md), [tasks](concepts/tasks.md), and [services](concepts/services.md). 3. Run configurations via `dstack`'s CLI or API. -[//]: # (The `dstack` server automatically provisions cloud resources, handles containers, logs, network, and everything else.) -4. Use [pools](concepts/pools.md) to manage instances and on-prem servers. - -[//]: # (### Coming soon) - -[//]: # (1. Multi-node tasks) -[//]: # (2. Auto-scalable services) -[//]: # (3. Integration with Kubernetes) +4. Use [pools](concepts/pools.md) to manage cloud instances and on-prem servers. ## Where do I start? 1. Follow [quickstart](quickstart.md) -2. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/README.md){:target="_blank"} +2. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/tree/master/examples){:target="_blank"} 3. Join the community via [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd){:target="_blank"} \ No newline at end of file diff --git a/docs/docs/installation/index.md b/docs/docs/installation/index.md index 7e94c7716..15356bb1e 100644 --- a/docs/docs/installation/index.md +++ b/docs/docs/installation/index.md @@ -10,13 +10,10 @@ To install `dstack`, use `pip`: -To use the open-source version of `dstack`, you have to configure -your cloud accounts via `~/.dstack/server/config.yml` and start the `dstack` server. - ## Configure backends -To configure cloud accounts, create the -`~/.dstack/server/config.yml` file, and configure each cloud under the `backends` property. +Before you can start the `dstack` server and let it provision instances in your cloud accounts, you have to +create the `~/.dstack/server/config.yml` file and configure a backend for each cloud account.
@@ -33,532 +30,8 @@ projects:
-Refer below for examples on how to configure a specific cloud provider. - -??? info "Projects" - For flexibility, `dstack` server permits you to configure backends for multiple projects. - If you intend to use only one project, name it `main`. - -[//]: # (!!! info "Default credentials") -[//]: # ( If you have default AWS, GCP, or Azure credentials on your machine, the `dstack` server will pick them up automatically.) -[//]: # ( Otherwise, you have to configure them manually.) - -### AWS - -There are two ways to configure AWS: using an access key or using the default credentials. - -=== "Access key" - - Create an access key by following the [this guide :material-arrow-top-right-thin:{ .external }](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-user.html#cli-authentication-user-get). - Once you've downloaded the `.csv` file with your IAM user's Access key ID and Secret access key, proceed to - configure the backend. - -
- - ```yaml - projects: - - name: main - backends: - - type: aws - creds: - type: access_key - access_key: KKAAUKLIZ5EHKICAOASV - secret_key: pn158lMqSBJiySwpQ9ubwmI6VUU3/W2fdJdFwfgO - ``` - -
- -=== "Default credentials" - - If you have default credentials set up (e.g. in `~/.aws/credentials`), configure the backend like this: - -
- - ```yaml - projects: - - name: main - backends: - - type: aws - creds: - type: default - ``` - -
- -??? info "VPC" - By default, `dstack` uses the default VPC. It's possible to customize it: - - === "vpc_name" - - ```yaml - projects: - - name: main - backends: - - type: aws - creds: - type: default - - vpc_name: my-vpc - ``` - - === "vpc_ids" - ```yaml - projects: - - name: main - backends: - - type: aws - creds: - type: default - - vpc_ids: - us-east-1: vpc-0a2b3c4d5e6f7g8h - us-east-2: vpc-9i8h7g6f5e4d3c2b - us-west-1: vpc-4d3c2b1a0f9e8d7 - ``` - - Note, the VPCs are required to have a public subnet. - -??? info "Required AWS permissions" - The following AWS policy permissions are sufficient for `dstack` to work: - - ``` - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "ec2:*" - ], - "Resource": "*" - }, - { - "Effect": "Allow", - "Action": [ - "servicequotas:*" - ], - "Resource": "*" - } - ] - } - ``` - -### Azure - -There are two ways to configure Azure: using a client secret or using the default credentials. - -=== "Client secret" - - A client secret can be created using the [Azure CLI :material-arrow-top-right-thin:{ .external }](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): - - ```shell - SUBSCRIPTION_ID=... - az ad sp create-for-rbac - --name dstack-app \ - --role $DSTACK_ROLE \ - --scopes /subscriptions/$SUBSCRIPTION_ID \ - --query "{ tenant_id: tenant, client_id: appId, client_secret: password }" - ``` - - Once you have `tenant_id`, `client_id`, and `client_secret`, go ahead and configure the backend. - -
- - ```yaml - projects: - - name: main - backends: - - type: azure - subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 - tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 - creds: - type: client - client_id: acf3f73a-597b-46b6-98d9-748d75018ed0 - client_secret: 1Kb8Q~o3Q2hdEvrul9yaj5DJDFkuL3RG7lger2VQ - ``` - -
- -=== "Default credentials" - - Obtain the `subscription_id` and `tenant_id` via the [Azure CLI :material-arrow-top-right-thin:{ .external }](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): - - ```shell - az account show --query "{subscription_id: id, tenant_id: tenantId}" - ``` - - Then proceed to configure the backend: - -
- - ```yaml - projects: - - name: main - backends: - - type: azure - subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 - tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 - creds: - type: default - ``` - -
- -!!! info "NOTE:" - If you don't know your `subscription_id`, run - - ```shell - az account show --query "{subscription_id: id}" - ``` - -??? info "Required Azure permissions" - The following Azure permissions are sufficient for `dstack` to work: - ``` - { - "properties": { - "roleName": "dstack-role", - "description": "Minimal reqired permissions for using Azure with dstack", - "assignableScopes": [ - "/subscriptions/${YOUR_SUBSCRIPTION_ID}" - ], - "permissions": [ - { - "actions": [ - "Microsoft.Authorization/*/read", - "Microsoft.Compute/availabilitySets/*", - "Microsoft.Compute/locations/*", - "Microsoft.Compute/virtualMachines/*", - "Microsoft.Compute/virtualMachineScaleSets/*", - "Microsoft.Compute/cloudServices/*", - "Microsoft.Compute/disks/write", - "Microsoft.Compute/disks/read", - "Microsoft.Compute/disks/delete", - "Microsoft.Network/networkSecurityGroups/*", - "Microsoft.Network/locations/*", - "Microsoft.Network/virtualNetworks/*", - "Microsoft.Network/networkInterfaces/*", - "Microsoft.Network/publicIPAddresses/*", - "Microsoft.Resources/subscriptions/resourceGroups/read", - "Microsoft.Resources/subscriptions/resourceGroups/write", - "Microsoft.Resources/subscriptions/read" - ], - "notActions": [], - "dataActions": [], - "notDataActions": [] - } - ] - } - } - ``` - -### GCP - -??? info "Enable APIs" - First, ensure the required APIs are enabled in your GCP `project_id`. - - ```shell - PROJECT_ID=... - gcloud config set project $PROJECT_ID - gcloud services enable cloudapis.googleapis.com - gcloud services enable compute.googleapis.com - ``` - -There are two ways to configure GCP: using a service account or using the default credentials. - -=== "Service account" - - To create a service account, follow [this guide :material-arrow-top-right-thin:{ .external }](https://cloud.google.com/iam/docs/service-accounts-create). - Make sure to grant it the `Service Account User` and `Compute Admin` roles. - - After setting up the service account [create a key :material-arrow-top-right-thin:{ .external }](https://cloud.google.com/iam/docs/keys-create-delete) for it - and download the corresponding JSON file. - - Then go ahead and configure the backend by specifying the downloaded file path. - -
- - ```yaml - projects: - - name: main - backends: - - type: gcp - project_id: gcp-project-id - creds: - type: service_account - filename: ~/.dstack/server/gcp-024ed630eab5.json - ``` - -
- -=== "Default credentials" - -
- - ```yaml - projects: - - name: main - backends: - - type: gcp - project_id: gcp-project-id - creds: - type: default - ``` - -
- -!!! info "NOTE:" - If you don't know your GCP project ID, run - - ```shell - gcloud projects list --format="json(projectId)" - ``` - -??? info "Required GCP permissions" - The following GCP permissions are sufficient for `dstack` to work: - - ``` - compute.disks.create - compute.firewalls.create - compute.images.useReadOnly - compute.instances.create - compute.instances.delete - compute.instances.get - compute.instances.setLabels - compute.instances.setMetadata - compute.instances.setTags - compute.networks.updatePolicy - compute.regions.list - compute.subnetworks.use - compute.subnetworks.useExternalIp - compute.zoneOperations.get - ``` - -### Lambda - -Log into your [Lambda Cloud :material-arrow-top-right-thin:{ .external }](https://lambdalabs.com/service/gpu-cloud) account, click API keys in the sidebar, and then click the `Generate API key` -button to create a new API key. - -Then, go ahead and configure the backend: - -
- -```yaml -projects: -- name: main - backends: - - type: lambda - creds: - type: api_key - api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p -``` - -
- -### TensorDock - -Log into your [TensorDock :material-arrow-top-right-thin:{ .external }](https://marketplace.tensordock.com/) account, click API in the sidebar, and use the `Create an Authorization` -section to create a new authorization key. - -Then, go ahead and configure the backend: - -
- -```yaml -projects: -- name: main - backends: - - type: tensordock - creds: - type: api_key - api_key: 248e621d-9317-7494-dc1557fa5825b-98b - api_token: FyBI3YbnFEYXdth2xqYRnQI7hiusssBC -``` - -
- -!!! info "NOTE:" - The `tensordock` backend supports on-demand instances only. Spot instance support coming soon. - -### Vast.ai - -Log into your [Vast.ai :material-arrow-top-right-thin:{ .external }](https://cloud.vast.ai/) account, click Account in the sidebar, and copy your -API Key. - -Then, go ahead and configure the backend: - -
- -```yaml -projects: -- name: main - backends: - - type: vastai - creds: - type: api_key - api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5 -``` - -
- -!!! info "NOTE:" - Also, the `vastai` backend supports on-demand instances only. Spot instance support coming soon. - -### CUDO - -Log into your [CUDO Compute :material-arrow-top-right-thin:{ .external }](https://compute.cudo.org/) account, click API keys in the sidebar, and click the `Create an API key` button. - -Ensure you've created a project with CUDO Compute, then proceed to configuring the backend. - -
- -```yaml -projects: -- name: main - backends: - - type: cudo - project_id: my-cudo-project - creds: - type: api_key - api_key: 7487240a466624b48de22865589 -``` - -
- -### RunPod - -Log into your [RunPod :material-arrow-top-right-thin:{ .external }](https://www.runpod.io/console/) console, click Settings in the sidebar, expand the `API Keys` section, and click -the button to create a key. - -Then proceed to configuring the backend. - -
- -```yaml -projects: -- name: main - backends: - - type: runpod - creds: - type: api_key - api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9 -``` - -
- -!!! warning "NOTE:" - If you're using a custom Docker image, its entrypoint cannot be anything other than `/bin/bash` or `/bin/sh`. - See the [issue :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/issues/1137) for more details. - -!!! info "NOTE:" - The `runpod` backend supports on-demand instances only. Spot instance support coming soon. - -### DataCrunch - -Log into your [DataCrunch :material-arrow-top-right-thin:{ .external }](https://cloud.datacrunch.io/signin) account, click Account Settings in the sidebar, find `REST API Credentials` area and then click the `Generate Credentials` button. - -Then, go ahead and configure the backend: - -
- -```yaml -projects: -- name: main - backends: - - type: datacrunch - creds: - type: api_key - client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T - client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8 -``` - -
- -### Kubernetes - -`dstack` supports both self-managed, and managed Kubernetes clusters. - -??? info "Prerequisite" - To use GPUs with Kubernetes, the cluster must be installed with the - [NVIDIA GPU Operator :material-arrow-top-right-thin:{ .external }](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html). - - [//]: # (TODO: Provide short yet clear instructions. Elaborate on whether it works with Kind.) - -To configure a Kubernetes backend, specify the path to the kubeconfig file, -and the port that `dstack` can use for proxying SSH traffic. -In case of a self-managed cluster, also specify the IP address of any node in the cluster. - -[//]: # (TODO: Mention that the Kind context has to be selected via `current-context` ) - -=== "Self-managed" - - Here's how to configure the backend to use a self-managed cluster. - -
- - ```yaml - projects: - - name: main - backends: - - type: kubernetes - kubeconfig: - filename: ~/.kube/config - networking: - ssh_host: localhost # The external IP address of any node - ssh_port: 32000 # Any port accessible outside of the cluster - ``` - -
- - The port specified to `ssh_port` must be accessible outside of the cluster. - - For example, if you are using Kind, make sure to add it via `extraPortMappings`: - -
- - ```yaml - kind: Cluster - apiVersion: kind.x-k8s.io/v1alpha4 - nodes: - - role: control-plane - extraPortMappings: - - containerPort: 32000 # Must be same as `ssh_port` - hostPort: 32000 # Must be same as `ssh_port` - ``` - -
- -[//]: # (TODO: Elaborate on the Kind's IP address on Linux) - -=== "Managed" - Here's how to configure the backend to use a managed cluster (AWS, GCP, Azure). - -
- - ```yaml - projects: - - name: main - backends: - - type: kubernetes - kubeconfig: - filename: ~/.kube/config - networking: - ssh_port: 32000 # Any port accessible outside of the cluster - ``` - -
- - The port specified to `ssh_port` must be accessible outside of the cluster. - - For example, if you are using EKS, make sure to add it via an ingress rule - of the corresponding security group: - - ```shell - aws ec2 authorize-security-group-ingress --group-id --protocol tcp --port 32000 --cidr 0.0.0.0/0 - ``` - -[//]: # (TODO: Elaborate on gateways, and what backends allow configuring them) - -[//]: # (TODO: Should we automatically detect ~/.kube/config) +> See the [server/config.yml reference](../reference/server/config.yml.md#examples) +> for more details on how to configure backends for all supported cloud providers. ## Start the server @@ -598,12 +71,12 @@ Once the `~/.dstack/server/config.yml` file is configured, proceed to start the > For more details on how to deploy `dstack` using Docker, check its [Docker repo](https://hub.docker.com/r/dstackai/dstack). -[//]: # (After you update `~/.dstack/server/config.yml`, make sure to restart the server.) +Once the `dstack` server is up, feel free to use the CLI or API to work with it. ## Configure the CLI -To point the CLI to the `dstack` server, you need to configure `~/.dstack/config.yml` -with the server address, user token and project name. +To point the CLI to the `dstack` server, configure `~/.dstack/config.yml` +with the server address, user token and project name:
@@ -622,6 +95,7 @@ Configuration is updated at ~/.dstack/config.yml ## What's next? -1. Follow [quickstart](../quickstart.md) -2. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/README.md) -3. Join the community via [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd) \ No newline at end of file +1. Check the [`server/config.yml` reference](../reference/server/config.yml.md) on how to configure backends +2. Follow [quickstart](../quickstart.md) +3. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/tree/master/examples) +4. Join the community via [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd) \ No newline at end of file diff --git a/docs/docs/quickstart.md b/docs/docs/quickstart.md index 7e3d4250d..96908b21c 100644 --- a/docs/docs/quickstart.md +++ b/docs/docs/quickstart.md @@ -1,6 +1,6 @@ # Quickstart -??? info "Prerequisites" +??? info "Installation" To use the open-source version, make sure to [install the server](installation/index.md) and configure backends. If you're using [dstack Sky :material-arrow-top-right-thin:{ .external }](https://sky.dstack.ai){:target="_blank"}, install the CLI and run the `dstack config` command: @@ -134,5 +134,5 @@ To exclude any files from uploading, use `.gitignore`. 1. Read about [dev environments](concepts/dev-environments.md), [tasks](concepts/tasks.md), [services](concepts/services.md), and [pools](concepts/pools.md) -2. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/blob/master/examples/README.md){:target="_blank"} +2. Browse [examples :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/tree/master/examples){:target="_blank"} 3. Join the community via [Discord :material-arrow-top-right-thin:{ .external }](https://discord.gg/u8SmfwPpMd) \ No newline at end of file diff --git a/docs/docs/reference/dstack.yml/service.md b/docs/docs/reference/dstack.yml/service.md index ba59460ef..f702cf675 100644 --- a/docs/docs/reference/dstack.yml/service.md +++ b/docs/docs/reference/dstack.yml/service.md @@ -2,10 +2,9 @@ The `service` configuration type allows running [services](../../concepts/services.md). -!!! info "Filename" - Configuration files must have a name ending with `.dstack.yml` (e.g., `.dstack.yml` or `serve.dstack.yml` are both acceptable) - and can be located in the project's root directory or any nested folder. - Any configuration can be run via [`dstack run`](../cli/index.md#dstack-run). +> Configuration files must have a name ending with `.dstack.yml` (e.g., `.dstack.yml` or `serve.dstack.yml` are both acceptable) +> and can be located in the project's root directory or any nested folder. +> Any configuration can be run via [`dstack run . -f PATH`](../cli/index.md#dstack-run). ## Examples @@ -69,7 +68,7 @@ port: 8000 port: 8000 ``` -### OpenAI-compatible interface +### OpenAI-compatible interface { #model-mapping } By default, if you run a service, its endpoint is accessible at `https://.`. @@ -104,7 +103,46 @@ model: In this case, with such a configuration, once the service is up, you'll be able to access the model at `https://gateway.` via the OpenAI-compatible interface. -See [services](../../concepts/services.md#configure-model-mapping) for more detail. + +The `format` supports only `tgi` (Text Generation Inference) +and `openai` (if you are using Text Generation Inference or vLLM with OpenAI-compatible mode). + +??? info "Chat template" + + By default, `dstack` loads the [chat template](https://huggingface.co/docs/transformers/main/en/chat_templating) + from the model's repository. If it is not present there, manual configuration is required. + + ```yaml + type: service + + image: ghcr.io/huggingface/text-generation-inference:latest + env: + - MODEL_ID=TheBloke/Llama-2-13B-chat-GPTQ + commands: + - text-generation-launcher --port 8000 --trust-remote-code --quantize gptq + port: 8000 + + resources: + gpu: 80GB + + # Enable the OpenAI-compatible endpoint + model: + type: chat + name: TheBloke/Llama-2-13B-chat-GPTQ + format: tgi + chat_template: "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = false %}{% endif %}{% for message in loop_messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if loop.index0 == 0 and system_message != false %}{% set content = '<>\\n' + system_message + '\\n<>\\n\\n' + message['content'] %}{% else %}{% set content = message['content'] %}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + content.strip() + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ ' ' + content.strip() + ' ' }}{% endif %}{% endfor %}" + eos_token: "" + ``` + + ##### Limitations + + Please note that model mapping is an experimental feature with the following limitations: + + 1. Doesn't work if your `chat_template` uses `bos_token`. As a workaround, replace `bos_token` inside `chat_template` with the token content itself. + 2. Doesn't work if `eos_token` is defined in the model repository as a dictionary. As a workaround, set `eos_token` manually, as shown in the example above (see Chat template). + + If you encounter any other issues, please make sure to file a [GitHub issue](https://github.com/dstackai/dstack/issues/new/choose). + ### Replicas and auto-scaling diff --git a/docs/docs/reference/dstack.yml/task.md b/docs/docs/reference/dstack.yml/task.md index cb080d162..f1fadb460 100644 --- a/docs/docs/reference/dstack.yml/task.md +++ b/docs/docs/reference/dstack.yml/task.md @@ -167,7 +167,7 @@ The following environment variables are available in any run and are passed by ` | `DSTACK_NODE_RANK` | The rank of the node | | `DSTACK_MASTER_NODE_IP` | The internal IP address the master node | -### Nodes +### Nodes { #_nodes } By default, the task runs on a single node. However, you can run it on a cluster of nodes. @@ -226,8 +226,15 @@ commands:
-Now, you can pass your arguments to the `dstack run` command. -See [tasks](../../concepts/tasks.md#parametrize-tasks) for more detail. +Now, you can pass your arguments to the `dstack run` command: + +
+ +```shell +$ dstack run . -f train.dstack.yml --train_batch_size=1 --num_train_epochs=100 +``` + +
### Web applications diff --git a/docs/docs/reference/server/config.yml.md b/docs/docs/reference/server/config.yml.md index 26a3bbdba..bc1615bb5 100644 --- a/docs/docs/reference/server/config.yml.md +++ b/docs/docs/reference/server/config.yml.md @@ -3,16 +3,24 @@ The `~/.dstack/server/config.yml` file is used by the `dstack` server to [configure](../../installation/index.md#configure-backends) cloud accounts. -!!! info "Projects" - For flexibility, `dstack` server permits you to configure backends for multiple projects. - If you intend to use only one project, name it `main`. +> The `dstack` server allows you to configure backends for multiple projects. +> If you don't need multiple projects, use only the `main` project. -### Examples +Each cloud account must be configured under the `backends` property of the respective project. +See the examples below. -#### AWS +## Examples + +### AWS + +There are two ways to configure AWS: using an access key or using the default credentials. === "Access key" + Create an access key by following the [this guide :material-arrow-top-right-thin:{ .external }](https://docs.aws.amazon.com/cli/latest/userguide/cli-authentication-user.html#cli-authentication-user-get). + Once you've downloaded the `.csv` file with your IAM user's Access key ID and Secret access key, proceed to + configure the backend. +
```yaml @@ -30,6 +38,8 @@ to [configure](../../installation/index.md#configure-backends) cloud accounts. === "Default credentials" + If you have default credentials set up (e.g. in `~/.aws/credentials`), configure the backend like this: +
```yaml @@ -43,9 +53,82 @@ to [configure](../../installation/index.md#configure-backends) cloud accounts.
-#### Azure +??? info "VPC" + By default, `dstack` uses the default VPC. It's possible to customize it: -=== "Client" + === "vpc_name" + + ```yaml + projects: + - name: main + backends: + - type: aws + creds: + type: default + + vpc_name: my-vpc + ``` + + === "vpc_ids" + ```yaml + projects: + - name: main + backends: + - type: aws + creds: + type: default + + vpc_ids: + us-east-1: vpc-0a2b3c4d5e6f7g8h + us-east-2: vpc-9i8h7g6f5e4d3c2b + us-west-1: vpc-4d3c2b1a0f9e8d7 + ``` + + Note, the VPCs are required to have a public subnet. + +??? info "Required AWS permissions" + The following AWS policy permissions are sufficient for `dstack` to work: + + ``` + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ec2:*" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "servicequotas:*" + ], + "Resource": "*" + } + ] + } + ``` + +### Azure + +There are two ways to configure Azure: using a client secret or using the default credentials. + +=== "Client secret" + + A client secret can be created using the [Azure CLI :material-arrow-top-right-thin:{ .external }](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): + + ```shell + SUBSCRIPTION_ID=... + az ad sp create-for-rbac + --name dstack-app \ + --role $DSTACK_ROLE \ + --scopes /subscriptions/$SUBSCRIPTION_ID \ + --query "{ tenant_id: tenant, client_id: appId, client_secret: password }" + ``` + + Once you have `tenant_id`, `client_id`, and `client_secret`, go ahead and configure the backend.
@@ -66,183 +149,392 @@ to [configure](../../installation/index.md#configure-backends) cloud accounts. === "Default credentials" + Obtain the `subscription_id` and `tenant_id` via the [Azure CLI :material-arrow-top-right-thin:{ .external }](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli): + + ```shell + az account show --query "{subscription_id: id, tenant_id: tenantId}" + ``` + + Then proceed to configure the backend: + +
+ + ```yaml + projects: + - name: main + backends: + - type: azure + subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 + tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 + creds: + type: default + ``` + +
+ + !!! info "NOTE:" + If you don't know your `subscription_id`, run + + ```shell + az account show --query "{subscription_id: id}" + ``` + + ??? info "Required Azure permissions" + The following Azure permissions are sufficient for `dstack` to work: + ``` + { + "properties": { + "roleName": "dstack-role", + "description": "Minimal reqired permissions for using Azure with dstack", + "assignableScopes": [ + "/subscriptions/${YOUR_SUBSCRIPTION_ID}" + ], + "permissions": [ + { + "actions": [ + "Microsoft.Authorization/*/read", + "Microsoft.Compute/availabilitySets/*", + "Microsoft.Compute/locations/*", + "Microsoft.Compute/virtualMachines/*", + "Microsoft.Compute/virtualMachineScaleSets/*", + "Microsoft.Compute/cloudServices/*", + "Microsoft.Compute/disks/write", + "Microsoft.Compute/disks/read", + "Microsoft.Compute/disks/delete", + "Microsoft.Network/networkSecurityGroups/*", + "Microsoft.Network/locations/*", + "Microsoft.Network/virtualNetworks/*", + "Microsoft.Network/networkInterfaces/*", + "Microsoft.Network/publicIPAddresses/*", + "Microsoft.Resources/subscriptions/resourceGroups/read", + "Microsoft.Resources/subscriptions/resourceGroups/write", + "Microsoft.Resources/subscriptions/read" + ], + "notActions": [], + "dataActions": [], + "notDataActions": [] + } + ] + } + } + ``` + + ### GCP + + ??? info "Enable APIs" + First, ensure the required APIs are enabled in your GCP `project_id`. + + ```shell + PROJECT_ID=... + gcloud config set project $PROJECT_ID + gcloud services enable cloudapis.googleapis.com + gcloud services enable compute.googleapis.com + ``` + + There are two ways to configure GCP: using a service account or using the default credentials. + + === "Service account" + + To create a service account, follow [this guide :material-arrow-top-right-thin:{ .external }](https://cloud.google.com/iam/docs/service-accounts-create). + Make sure to grant it the `Service Account User` and `Compute Admin` roles. + + After setting up the service account [create a key :material-arrow-top-right-thin:{ .external }](https://cloud.google.com/iam/docs/keys-create-delete) for it + and download the corresponding JSON file. + + Then go ahead and configure the backend by specifying the downloaded file path. + +
+ + ```yaml + projects: + - name: main + backends: + - type: gcp + project_id: gcp-project-id + creds: + type: service_account + filename: ~/.dstack/server/gcp-024ed630eab5.json + ``` + +
+ + === "Default credentials" + +
+ + ```yaml + projects: + - name: main + backends: + - type: gcp + project_id: gcp-project-id + creds: + type: default + ``` + +
+ + !!! info "NOTE:" + If you don't know your GCP project ID, run + + ```shell + gcloud projects list --format="json(projectId)" + ``` + + ??? info "Required GCP permissions" + The following GCP permissions are sufficient for `dstack` to work: + + ``` + compute.disks.create + compute.firewalls.create + compute.images.useReadOnly + compute.instances.create + compute.instances.delete + compute.instances.get + compute.instances.setLabels + compute.instances.setMetadata + compute.instances.setTags + compute.networks.updatePolicy + compute.regions.list + compute.subnetworks.use + compute.subnetworks.useExternalIp + compute.zoneOperations.get + ``` + + ### Lambda + + Log into your [Lambda Cloud :material-arrow-top-right-thin:{ .external }](https://lambdalabs.com/service/gpu-cloud) account, click API keys in the sidebar, and then click the `Generate API key` + button to create a new API key. + + Then, go ahead and configure the backend: +
- + ```yaml projects: - name: main - backends: - - type: azure - subscription_id: 06c82ce3-28ff-4285-a146-c5e981a9d808 - tenant_id: f84a7584-88e4-4fd2-8e97-623f0a715ee1 + backends: + - type: lambda creds: - type: default + type: api_key + api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p ``` - +
-#### GCP + ### TensorDock -=== "Service account" + Log into your [TensorDock :material-arrow-top-right-thin:{ .external }](https://marketplace.tensordock.com/) account, click API in the sidebar, and use the `Create an Authorization` + section to create a new authorization key. + + Then, go ahead and configure the backend:
- + ```yaml projects: - name: main - backends: - - type: gcp - project_id: gcp-project-id + backends: + - type: tensordock creds: - type: service_account - filename: ~/.dstack/server/gcp-024ed630eab5.json + type: api_key + api_key: 248e621d-9317-7494-dc1557fa5825b-98b + api_token: FyBI3YbnFEYXdth2xqYRnQI7hiusssBC ``` - +
-=== "Default credentials" + !!! info "NOTE:" + The `tensordock` backend supports on-demand instances only. Spot instance support coming soon. + + ### Vast.ai + + Log into your [Vast.ai :material-arrow-top-right-thin:{ .external }](https://cloud.vast.ai/) account, click Account in the sidebar, and copy your + API Key. + + Then, go ahead and configure the backend:
- + ```yaml projects: - name: main - backends: - - type: gcp - project_id: gcp-project-id + backends: + - type: vastai creds: - type: default + type: api_key + api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5 ``` - -
- -#### Lambda - -
-```yaml -projects: -- name: main - backends: - - type: lambda - creds: - type: api_key - api_key: eersct_yrpiey-naaeedst-tk-_cb6ba38e1128464aea9bcc619e4ba2a5.iijPMi07obgt6TZ87v5qAEj61RVxhd0p -``` +
-
+ !!! info "NOTE:" + Also, the `vastai` backend supports on-demand instances only. Spot instance support coming soon. -#### TensorDock + ### CUDO -
+ Log into your [CUDO Compute :material-arrow-top-right-thin:{ .external }](https://compute.cudo.org/) account, click API keys in the sidebar, and click the `Create an API key` button. -```yaml -projects: -- name: main - backends: - - type: tensordock - creds: - type: api_key - api_key: 248e621d-9317-7494-dc1557fa5825b-98b - api_token: FyBI3YbnFEYXdth2xqYRnQI7hiusssBC -``` + Ensure you've created a project with CUDO Compute, then proceed to configuring the backend. -
+
-#### Vast.ai + ```yaml + projects: + - name: main + backends: + - type: cudo + project_id: my-cudo-project + creds: + type: api_key + api_key: 7487240a466624b48de22865589 + ``` -
+
-```yaml -projects: -- name: main - backends: - - type: vastai - creds: - type: api_key - api_key: d75789f22f1908e0527c78a283b523dd73051c8c7d05456516fc91e9d4efd8c5 -``` + ### RunPod -
+ Log into your [RunPod :material-arrow-top-right-thin:{ .external }](https://www.runpod.io/console/) console, click Settings in the sidebar, expand the `API Keys` section, and click + the button to create a key. -#### CUDO + Then proceed to configuring the backend. -
+
-```yaml -projects: -- name: main - backends: - - type: cudo - project_id: my-cudo-project - creds: - type: api_key - api_key: 7487240a466624b48de22865589 -``` + ```yaml + projects: + - name: main + backends: + - type: runpod + creds: + type: api_key + api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9 + ``` -
+
-#### DataCrunch + !!! warning "NOTE:" + If you're using a custom Docker image, its entrypoint cannot be anything other than `/bin/bash` or `/bin/sh`. + See the [issue :material-arrow-top-right-thin:{ .external }](https://github.com/dstackai/dstack/issues/1137) for more details. -
+ !!! info "NOTE:" + The `runpod` backend supports on-demand instances only. Spot instance support coming soon. -```yaml -projects: -- name: main - backends: - - type: datacrunch - creds: - type: api_key - client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T - client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8 -``` + ### DataCrunch -
+ Log into your [DataCrunch :material-arrow-top-right-thin:{ .external }](https://cloud.datacrunch.io/signin) account, click Account Settings in the sidebar, find `REST API Credentials` area and then click the `Generate Credentials` button. -#### Kubernetes + Then, go ahead and configure the backend: -=== "Self-managed"
```yaml projects: - name: main - backends: - - type: kubernetes - kubeconfig: - filename: ~/.kube/config - networking: - ssh_host: localhost # The external IP address of any node - ssh_port: 32000 # Any port accessible outside of the cluster + backends: + - type: datacrunch + creds: + type: api_key + client_id: xfaHBqYEsArqhKWX-e52x3HH7w8T + client_secret: B5ZU5Qx9Nt8oGMlmMhNI3iglK8bjMhagTbylZy4WzncZe39995f7Vxh8 ```
-=== "Managed" -
+ ### Kubernetes - ```yaml - projects: - - name: main - backends: - - type: kubernetes - kubeconfig: - filename: ~/.kube/config - networking: - ssh_port: 32000 # Any port accessible outside of the cluster - ``` + `dstack` supports both self-managed, and managed Kubernetes clusters. -
+ ??? info "Prerequisite" + To use GPUs with Kubernetes, the cluster must be installed with the + [NVIDIA GPU Operator :material-arrow-top-right-thin:{ .external }](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html). + + [//]: # (TODO: Provide short yet clear instructions. Elaborate on whether it works with Kind.) + + To configure a Kubernetes backend, specify the path to the kubeconfig file, + and the port that `dstack` can use for proxying SSH traffic. + In case of a self-managed cluster, also specify the IP address of any node in the cluster. + + [//]: # (TODO: Mention that the Kind context has to be selected via `current-context` ) + + === "Self-managed" + + Here's how to configure the backend to use a self-managed cluster. + +
+ + ```yaml + projects: + - name: main + backends: + - type: kubernetes + kubeconfig: + filename: ~/.kube/config + networking: + ssh_host: localhost # The external IP address of any node + ssh_port: 32000 # Any port accessible outside of the cluster + ``` + +
+ + The port specified to `ssh_port` must be accessible outside of the cluster. + + For example, if you are using Kind, make sure to add it via `extraPortMappings`: + +
+ + ```yaml + kind: Cluster + apiVersion: kind.x-k8s.io/v1alpha4 + nodes: + - role: control-plane + extraPortMappings: + - containerPort: 32000 # Must be same as `ssh_port` + hostPort: 32000 # Must be same as `ssh_port` + ``` + +
+ + [//]: # (TODO: Elaborate on the Kind's IP address on Linux) + + === "Managed" + Here's how to configure the backend to use a managed cluster (AWS, GCP, Azure). + +
+ + ```yaml + projects: + - name: main + backends: + - type: kubernetes + kubeconfig: + filename: ~/.kube/config + networking: + ssh_port: 32000 # Any port accessible outside of the cluster + ``` + +
+ + The port specified to `ssh_port` must be accessible outside of the cluster. + + For example, if you are using EKS, make sure to add it via an ingress rule + of the corresponding security group: + + ```shell + aws ec2 authorize-security-group-ingress --group-id --protocol tcp --port 32000 --cidr 0.0.0.0/0 + ``` + + [//]: # (TODO: Elaborate on gateways, and what backends allow configuring them) -For more details on configuring clouds, please refer to [Installation](../../installation/index.md#configure-backends). + [//]: # (TODO: Should we automatically detect ~/.kube/config) -### Root reference +## Root reference #SCHEMA# dstack._internal.server.services.config.ServerConfig overrides: show_root_heading: false -### `projects[n]` { #projects data-toc-label="projects" } +## `projects[n]` { #projects data-toc-label="projects" } #SCHEMA# dstack._internal.server.services.config.ProjectConfig overrides: @@ -250,7 +542,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins backends: type: 'Union[AWSConfigInfoWithCreds, AzureConfigInfoWithCreds, GCPConfigInfoWithCreds, LambdaConfigInfoWithCreds, TensorDockConfigInfoWithCreds, VastAIConfigInfoWithCreds, KubernetesConfig]' -### `projects[n].backends[type=aws]` { #aws data-toc-label="backends[type=aws]" } +## `projects[n].backends[type=aws]` { #aws data-toc-label="backends[type=aws]" } #SCHEMA# dstack._internal.server.services.config.AWSConfig overrides: @@ -259,7 +551,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: aws- -### `projects[n].backends[type=aws].creds` { #aws-creds data-toc-label="backends[type=aws].creds" } +## `projects[n].backends[type=aws].creds` { #aws-creds data-toc-label="backends[type=aws].creds" } === "Access key" #SCHEMA# dstack._internal.core.models.backends.aws.AWSAccessKeyCreds @@ -275,7 +567,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=azure]` { #azure data-toc-label="backends[type=azure]" } +## `projects[n].backends[type=azure]` { #azure data-toc-label="backends[type=azure]" } #SCHEMA# dstack._internal.server.services.config.AzureConfig overrides: @@ -284,7 +576,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: azure- -### `projects[n].backends[type=azure].creds` { #azure-creds data-toc-label="backends[type=azure].creds" } +## `projects[n].backends[type=azure].creds` { #azure-creds data-toc-label="backends[type=azure].creds" } === "Client" #SCHEMA# dstack._internal.core.models.backends.azure.AzureClientCreds @@ -300,7 +592,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=datacrunch]` { #datacrunch data-toc-label="backends[type=datacrunch]" } +## `projects[n].backends[type=datacrunch]` { #datacrunch data-toc-label="backends[type=datacrunch]" } #SCHEMA# dstack._internal.server.services.config.DataCrunchConfig overrides: @@ -309,7 +601,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: datacrunch- -### `projects[n].backends[type=datacrunch].creds` { #datacrunch-creds data-toc-label="backends[type=datacrunch].creds" } +## `projects[n].backends[type=datacrunch].creds` { #datacrunch-creds data-toc-label="backends[type=datacrunch].creds" } #SCHEMA# dstack._internal.core.models.backends.datacrunch.DataCrunchAPIKeyCreds overrides: @@ -317,7 +609,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=gcp]` { #gcp data-toc-label="backends[type=gcp]" } +## `projects[n].backends[type=gcp]` { #gcp data-toc-label="backends[type=gcp]" } #SCHEMA# dstack._internal.server.services.config.GCPConfig overrides: @@ -326,7 +618,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: gcp- -### `projects[n].backends[type=gcp].creds` { #gcp-creds data-toc-label="backends[type=gcp].creds" } +## `projects[n].backends[type=gcp].creds` { #gcp-creds data-toc-label="backends[type=gcp].creds" } === "Service account" #SCHEMA# dstack._internal.server.services.config.GCPServiceAccountCreds @@ -342,7 +634,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=lambda]` { #lambda data-toc-label="backends[type=lambda]" } +## `projects[n].backends[type=lambda]` { #lambda data-toc-label="backends[type=lambda]" } #SCHEMA# dstack._internal.server.services.config.LambdaConfig overrides: @@ -351,7 +643,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: lambda- -### `projects[n].backends[type=lambda].creds` { #lambda-creds data-toc-label="backends[type=lambda].creds" } +## `projects[n].backends[type=lambda].creds` { #lambda-creds data-toc-label="backends[type=lambda].creds" } #SCHEMA# dstack._internal.core.models.backends.lambdalabs.LambdaAPIKeyCreds overrides: @@ -359,7 +651,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=tensordock]` { #tensordock data-toc-label="backends[type=tensordock]" } +## `projects[n].backends[type=tensordock]` { #tensordock data-toc-label="backends[type=tensordock]" } #SCHEMA# dstack._internal.server.services.config.TensorDockConfig overrides: @@ -368,7 +660,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: tensordock- -### `projects[n].backends[type=tensordock].creds` { #tensordock-creds data-toc-label="backends[type=tensordock].creds" } +## `projects[n].backends[type=tensordock].creds` { #tensordock-creds data-toc-label="backends[type=tensordock].creds" } #SCHEMA# dstack._internal.core.models.backends.tensordock.TensorDockAPIKeyCreds overrides: @@ -376,7 +668,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=vastai]` { #vastai data-toc-label="backends[type=vastai]" } +## `projects[n].backends[type=vastai]` { #vastai data-toc-label="backends[type=vastai]" } #SCHEMA# dstack._internal.server.services.config.VastAIConfig overrides: @@ -385,7 +677,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins required: true item_id_prefix: vastai- -### `projects[n].backends[type=vastai].creds` { #vastai-creds data-toc-label="backends[type=vastai].creds" } +## `projects[n].backends[type=vastai].creds` { #vastai-creds data-toc-label="backends[type=vastai].creds" } #SCHEMA# dstack._internal.core.models.backends.vastai.VastAIAPIKeyCreds overrides: @@ -393,7 +685,7 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=kubernetes]` { #kubernetes data-toc-label="backends[type=kubernetes]" } +## `projects[n].backends[type=kubernetes]` { #kubernetes data-toc-label="backends[type=kubernetes]" } #SCHEMA# dstack._internal.server.services.config.KubernetesConfig overrides: @@ -401,13 +693,13 @@ For more details on configuring clouds, please refer to [Installation](../../ins type: required: true -### `projects[n].backends[type=kubernetes].kubeconfig` { #kubeconfig data-toc-label="kubeconfig" } +## `projects[n].backends[type=kubernetes].kubeconfig` { #kubeconfig data-toc-label="kubeconfig" } ##SCHEMA# dstack._internal.server.services.config.KubeconfigConfig overrides: show_root_heading: false -### `projects[n].backends[type=kubernetes].networking` { #networking data-toc-label="networking" } +## `projects[n].backends[type=kubernetes].networking` { #networking data-toc-label="networking" } ##SCHEMA# dstack._internal.core.models.backends.kubernetes.KubernetesNetworkingConfig overrides: diff --git a/docs/overrides/home.html b/docs/overrides/home.html index 561e76d3e..52d59b097 100644 --- a/docs/overrides/home.html +++ b/docs/overrides/home.html @@ -279,6 +279,88 @@

Pools

+
+
+

Why community dstack

+ +
+
+
+ +
+

Andrew Spott

+ +

ML Engineer at Stealth Startup

+ +

+ Thanks to @dstack, I get the convenience of having a personal Slurm cluster + and using budget-friendly cloud GPUs, without paying the super-high premiums charged by the big three. +

+
+ +
+
+ +
+

Alvaro Bartolome

+ +

ML Engineer at Argilla

+ +

+ With @dstack it's incredibly easy to define a configuration within a repository + and run it without worrying about GPU availability. It lets you focus on + data and your research. +

+
+ +
+
+ +
+

Park Chansung

+ +

ML Researcher at ETRI

+ +

+ Thanks to @dstack, I can effortlessly access the top GPU options across different clouds, + saving me time and money while pushing my AI work forward. +

+
+ +
+
+ +
+

Eckart Burgwedel

+ +

CEO at Uberchord

+ +

+ With @dstack, running an open-source LLM or a dev environment on a cloud GPU is as + easy as running a local Docker container. + It combines the ease of Docker with the auto-scaling capabilities of K8s. +

+
+ +
+
+ +
+

Peter Hill

+ +

Co-Founder at CUDO Compute

+ +

+ @dstack is instrumental in simplifying infrastructure provisioning and AI + model development. + if your organization is on the lookout for an platform to speed up the adoption of AI, I + wholeheartedly recommend @dstack +

+
+
+
+
+

Examples

@@ -325,8 +407,6 @@

Deploy LLMs as endpoints using vLLM. - -

@@ -402,7 +482,7 @@

- Deploy Mixtral 7Bx8 as an endpoint using Ollama. +Deploy Mixtral 7Bx8 as an endpoint using Ollama.

@@ -413,7 +493,7 @@

- More examples

@@ -451,8 +531,6 @@

Get started in a minute

Open-source
Use your own cloud accounts or data centers. -
-
@@ -501,8 +579,8 @@

Get started in a minute

-
CLI & API
Self-hosted
+
CLI & API
@@ -512,17 +590,16 @@

Get started in a minute

-
-
dstack Sky
+
+
Enterprise
- Access GPUs from our marketplace at the best rate. - + Use your own cloud accounts or data centers.
+ style="padding-top: 5px">
AWS
@@ -537,32 +614,81 @@

Get started in a minute

- -
Lambda
+ +
K8S
+
-
- -
TensorDock
+
+
+
+
Web console
+
Single sign-on
+
Audit logs
+
Self-hosted
+
CLI & API
+
-
- -
Vast.ai
+ Request a trial +
Get a free 60-day trial. Or book a demo.
+
+
-
- -
CUDO
-
+
+
dstack Sky
+
+ Access GPUs at the best rates through our global marketplace. +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
-
CLI & API
-
Hosted by dstack
Web console
+
Hosted by dstack
+
CLI & API
@@ -576,63 +702,19 @@

Get started in a minute

Sign up now -
Pay per compute. No commision.
-
-
- -
-
Enterprise
-
- Use your own cloud accounts or data centers. -

+
Pay per compute. No comission.
- -
-
- -
AWS
-
- -
- -
Azure
-
- -
- -
GCP
-
- -
- -
K8S
-
-
- -
-
-
-
CLI & API
-
Self-hosted
-
Web console
-
Single sign-on
-
Audit logs
-
-
- - Get a demo -
Contact us for a demo, pricing, and trial.
-

- Working with a university or research lab on open-source? - Apply to use dstack Enterprise for free. + Need help, have a question, or just want to stay updated? +
+ + Join Discord +

@@ -666,92 +748,6 @@

FAQ

--> - -
-
-

Why community dstack

- -
-
-
- -
-

Andrew Spott

- -

ML Engineer at Stealth Startup

- -

- Thanks to @dstack, I get the convenience of having a personal Slurm cluster - and using budget-friendly cloud GPUs, without paying the super-high premiums charged by the big three. -

-
- -
-
- -
-

Alvaro Bartolome

- -

ML Engineer at Argilla

- -

- With @dstack it's incredibly easy to define a configuration within a repository - and run it without worrying about GPU availability. It lets you focus on - data and your research. -

-
- -
-
- -
-

Park Chansung

- -

ML Researcher at ETRI

- -

- Thanks to @dstack, I can effortlessly access the top GPU options across different clouds, - saving me time and money while pushing my AI work forward. -

-
- -
-
- -
-

Eckart Burgwedel

- -

CEO at Uberchord

- -

- With @dstack, running an open-source LLM or a dev environment on a cloud GPU is as - easy as running a local Docker container. - It combines the ease of Docker with the auto-scaling capabilities of K8s. -

-
- -
-
- -
-

Peter Hill

- -

Co-Founder at CUDO Compute

- -

- @dstack is instrumental in simplifying infrastructure provisioning and AI - model development. - if your organization is on the lookout for an platform to speed up the adoption of AI, I - wholeheartedly recommend @dstack -

-
-
- - - Join Discord - -
-
{% endblock %} diff --git a/docs/overrides/landing.html b/docs/overrides/landing.html index bda69d681..e8b217d96 100644 --- a/docs/overrides/landing.html +++ b/docs/overrides/landing.html @@ -43,7 +43,7 @@
Resources
Docs - Examples + Examples Changelog Blog
diff --git a/docs/overrides/main.html b/docs/overrides/main.html index 07225e8cb..2c52529c9 100644 --- a/docs/overrides/main.html +++ b/docs/overrides/main.html @@ -98,7 +98,7 @@
Resources
Docs - Examples + Examples Changelog Blog
diff --git a/mkdocs.yml b/mkdocs.yml index 14aec5c19..a6f007e3e 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -195,18 +195,19 @@ nav: - Services: docs/concepts/services.md - Pools: docs/concepts/pools.md - Reference: + - server/config.yml: docs/reference/server/config.yml.md - .dstack.yml: - dev-environment: docs/reference/dstack.yml/dev-environment.md - task: docs/reference/dstack.yml/task.md - service: docs/reference/dstack.yml/service.md - - gateway: docs/reference/dstack.yml/gateway.md +# - gateway: docs/reference/dstack.yml/gateway.md - profiles.yml: docs/reference/profiles.yml.md - CLI: docs/reference/cli/index.md - - server/config.yml: docs/reference/server/config.yml.md + - profiles.yml: docs/reference/profiles.yml.md - API: - Python API: docs/reference/api/python/index.md - REST API: docs/reference/api/rest/index.md - - Examples: https://github.com/dstackai/dstack/blob/master/examples/README.md" target="_blank + - Examples: https://github.com/dstackai/dstack/tree/master/examples" target="_blank - Changelog: https://github.com/dstackai/dstack/releases" target="_blank - Blog: - blog/index.md