Skip to content

Releases: dstackai/dstack

0.18.6

18 Jul 14:44
@peterschmidt85 peterschmidt85
0.18.6
50d6d41
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.18.6

Major fixes

  • Support for GitLab's authorization when the repo is using HTTP/HTTPS by @jvstme in #1412
  • Add a multi-node example to the Hugging Alignment Handbook example by @deep-diver in #1409
  • Fix the issue where idle instances weren't offered (occurred when a GPU name was in upper case). by @jvstme in #1417
  • Fix the issue where an exception is thrown for non-standard Git repo host URLs using HTTP/HTTPS @jvstme in #1410
  • Support H100 with the gcp backend by @jvstme in #1405

Warning

If you have idle instances in your pool, it is recommended to re-create them after upgrading to version 0.18.6. Otherwise, there is a risk that these instances won't be able to execute jobs.

Other

Full changelog: 0.18.5...0.18.6

Contributors

deep-diver and jvstme
Assets 2
Loading

0.18.5

12 Jul 13:09
@peterschmidt85 peterschmidt85
0.18.5
41557ca
Compare
Choose a tag to compare
0.18.5

Read below about its new features and bug-fixes.

Volumes

When you run anything with dstack, it allows you to configure the disk size. However, once the run is finished, if you haven't stored your data in any external storage, all the data on disk will be erased. With 0.18.5, we're adding support for network volumes that allow data to persist across runs.

Once you've created a volume (e.g. named my-new-volume), you can attach it to a dev environment, task, or service.

type: dev-environment
ide: vscode
volumes:
  - name: my-new-volume
    path: /volume_data

The data stored in the volume will persist across runs.

dstack allows you to create new volumes and register existing ones. To learn more about how volumes work, check out the docs.

Important

Volumes are currently experimental and only work with the aws backend. Support for other backends is coming soon.

PostgreSQL

By default, dstack stores its state in ~/.dstack/server/data using SQLite. With this update, it's now possible to configure dstack to store its state in PostgreSQL. Just pass the DSTACK_DATABASE_URL environment variable.

DSTACK_DATABASE_URL="postgresql+asyncpg://myuser:mypassword@localhost:5432/mydatabase" dstack server

Important

Despite PostgreSQL support, dstack still requires that you run only one instance of the dstack server. However, this requirement will be lifted in a future update.

On-prem clusters

Previously, dstack didn't allow the use of on-prem clusters (added via dstack pool add-ssh) if there were no backends configured. This update fixes that bug. Now, you don't have to configure any backends if you only plan to use on-prem clusters.

Supported GPUs

Previously, dstack didn't support L4 and H100 GPUs with AWS. Now you can use them.

Full changelog

See more: 0.18.4...0.18.5

Contributors

r4victor and jvstme
Assets 2
Loading

0.18.5rc1

08 Jul 11:06
@peterschmidt85 peterschmidt85
0.18.5rc1
41557ca
Compare
Choose a tag to compare
0.18.5rc1 Pre-release
Pre-release

This is a release candidate build of the upcoming 0.18.5 release. Read below to learn about its new features and bug-fixes.

Volumes

When you run anything with dstack, it allows you to configure the disk size. However, once the run is finished, if you haven't stored your data in any external storage, all the data on disk will be erased. With 0.18.5, we're adding support for network volumes that allow data to persist across runs.

Once you've created a volume (e.g. named my-new-volume), you can attach it to a dev environment, task, or service.

type: dev-environment
ide: vscode
volumes:
  - name: my-new-volume
    path: /volume_data

The data stored in the volume will persist across runs.

dstack allows you to create new volumes and register existing ones. To learn more about how volumes work, check out the docs.

Important

Volumes are currently experimental and only work with the aws backend. Support for other backends is coming soon.

PostgreSQL

By default, dstack stores its state in /root/.dstack/server/data using SQLite. With this update, it's now possible to configure dstack to store its state in PostgreSQL. Just pass the DSTACK_DATABASE_URL environment variable.

DSTACK_DATABASE_URL="postgresql+asyncpg://myuser:mypassword@localhost:5432/mydatabase" dstack server

Important

Despite PostgreSQL support, dstack still requires that you run only one instance of the dstack server. However, this requirement will be lifted in a future update.

On-prem clusters

Previously, dstack didn't allow the use of on-prem clusters (added via dstack pool add-ssh) if there were no backends configured. This update fixes that bug. Now, you don't have to configure any backends if you only plan to use on-prem clusters.

Supported GPUs

Previously, dstack didn't support L4 and H100 GPUs with AWS. Now you can use them.

Full changelog

See more: 0.18.4...0.18.5rc1

Contributors

r4victor and jvstme
Assets 2
Loading

0.18.4

27 Jun 12:14
@peterschmidt85 peterschmidt85
0.18.4
f6395c6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.18.4

Google Cloud TPU

This update introduces initial support for Google Cloud TPU.

To request a TPU, specify the TPU architecture prefixed by tpu- (in gpu under resources):

type: task

python: "3.11"

commands:
  - pip install torch~=2.3.0 torch_xla[tpu]~=2.3.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html
  - git clone --recursive https://github.com/pytorch/xla.git
  - python3 xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=1

resources:
  gpu:  tpu-v2-8

Important

Currently, you can't specify other than 8 TPU cores. This means only single TPU device workloads are supported. Support for multiple TPU devices is coming soon.

Private subnets with GCP

Additionally, the update allows configuring the gcp backend to use only private subnets. To achieve this, set public_ips to false.

projects:
  - name: main
    backends:
      - type: gcp
        creds:
          type: default

        public_ips: false

Major bug-fixes

Besides TPU, the update fixes a few important bugs.

Other

New contributors

Full changelog: 0.18.3...0.18.4

Contributors

Bihan, r4victor, and 4 other contributors
Assets 2
Loading

0.18.4rc3

26 Jun 14:49
@peterschmidt85 peterschmidt85
0.18.4rc3
3e89218
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.18.4rc3 Pre-release
Pre-release

This is a preview build of the upcoming 0.18.4 release. See below to see what's new.

TPU

One of the major new features in this update is the initial support for Google Cloud TPU.

To request a TPU, you simply need to specify the system architecture of the required TPU prefixed by tpu- in gpu:

type: task

python: "3.11"

commands:
  - pip install torch~=2.3.0 torch_xla[tpu]~=2.3.0 torchvision -f https://storage.googleapis.com/libtpu-releases/index.html
  - git clone --recursive https://github.com/pytorch/xla.git
  - python3 xla/test/test_train_mp_imagenet.py --fake_data --model=resnet50 --num_epochs=1

resources:
  gpu:  tpu-v2-8

Important

You cannot request multiple nodes (for running parallel on multiple TPU devices) for tasks. This feature is coming soon.

You're very welcome to try the initial support and share your feedback.

Major bug-fixes

Besides TPU, the update fixes a few important bugs.

Other

New contributors

Full changelog: 0.18.3...0.18.4rc3

Contributors

Bihan, r4victor, and 4 other contributors
Assets 2
Loading

0.18.3

06 Jun 10:55
@peterschmidt85 peterschmidt85
0.18.3
36655f0
Compare
Choose a tag to compare
0.18.3

Oracle Cloud Infrastructure

With the new update, it is now possible to run workloads with your Oracle Cloud Infrastructure (OCI) account. The backend is called oci and can be configured as follows:

projects:
  - name: main
    backends:
      - type: oci
        creds:
          type: default

The supported credential types include default and client. In case default is used, dstack automatically picks the default OCI credentials from ~/.oci/config.

Just like other backends, oci supports dev environments, tasks, and services:

Note

Support for spot instances, multi-node tasks, and gateways is coming soon.

Find more documentation on using Oracle Cloud Infrastructure on the reference page.

Retry policy

We have reworked how to configure the retry policy and how it is applied to runs. Here's an example:

type: task

commands: 
  - python train.py

retry:
  on_events: [no-capacity]
  duration: 2h

Now, if you run such a task, dstack will keep trying to find capacity within 2 hours. Once capacity is found, dstack will run the task.

The on_events property also supports error (in case the run fails with an error) and interruption (if the run is using a spot instance and it was interrupted).

Previously, dstack only allowed retries when spot instances were interrupted.

RunPod

Previously, the runpod backend only allowed the use of Docker images with /bin/bash or /bin/sh as the entrypoint. Thanks to the fix on the RunPod's side, dstack now allows the use of any Docker images.

Additionally, the runpod backend now also supports spot instances.

GCP

The gcp backend now also allows configuring VPCs:

projects:
  - name: main
    backends:
      - type: gcp

        project_id: my-awesome-project
        creds:
          type: default

        vpc_name: my-custom-vpc

The VPC should belong to the same project. If you would like to use a shared VPC from another project, you can also specify vpc_project_id.

AWS

Last but not least, for the aws backend, it is now possible to configure VPCs for selected regions and use the default VPC in other regions:

projects:
  - name: main
    backends:
      - type: aws
        creds:
          type: default

        vpc_ids:
          us-east-1: vpc-0a2b3c4d5e6f7g8h

        default_vpcs: true

You just need to set default_vpcs to true.

Other changes

Contributors

r4victor, jvstme, and peterschmidt85
Assets 2
Loading

0.18.3rc1

05 Jun 09:37
@peterschmidt85 peterschmidt85
0.18.3rc1
9d7c42f
Compare
Choose a tag to compare
0.18.3rc1 Pre-release
Pre-release

OCI

With the new update, it is now possible to run workloads with your Oracle Cloud Infrastructure (OCI) account. The backend is called oci and can be configured as follows:

projects:
  - name: main
    backends:
      - type: oci
        creds:
          type: default

The supported credential types include default and client. In case default is used, dstack automatically picks the default OCI credentials from ~/.oci/config.

Warning

OCI support does not yet include spot instances, multi-node tasks, and gateways. These features will be added in upcoming updates.

Retry policy

We have reworked how to configure the retry policy and how it is applied to runs. Here's an example:

type: task

commands: 
  - python train.py

retry:
  on_events: [no-capacity]
  duration: 2h

Now, if you run such a task, dstack will keep trying to find capacity within 2 hours. Once capacity is found, dstack will run the task.

The on_events property also supports error (in case the run fails with an error) and interruption (if the run is using a spot instance and it was interrupted).

Previously, dstack only allowed retries when spot instances were interrupted.

VPC

GCP

The gcp backend now also allows configuring VPCs:

projects:
  - name: main
    backends:
      - type: gcp

        project_id: my-awesome-project
        creds:
          type: default

        vpc_name: my-custom-vpc

The VPC should belong to the same project. If you would like to use a shared VPC from another project, you can also specify vpc_project_id.

AWS

Last but not least, for the aws backend, it is now possible to configure VPCs for selected regions and use the default VPC in other regions:

projects:
  - name: main
    backends:
      - type: aws
        creds:
          type: default

        vpc_ids:
          us-east-1: vpc-0a2b3c4d5e6f7g8h

        default_vpcs: true

You just need to set default_vpcs to true.

Other changes

Full changelog: 0.18.2...0.18.3rc1

Warning

This is an RC build. Please report any bugs to the issue tracker. The final release is planned for later this week, and the official documentation and examples will be updated then.

Contributors

r4victor, jvstme, and peterschmidt85
Assets 2
Loading

0.18.2

13 May 12:30
@peterschmidt85 peterschmidt85
0.18.2
86b41b2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
0.18.2

On-prem clusters

Network

The dstack pool add-ssh command now supports the --network argument. Use this argument if you want to use multiple instances that share the same private network as a cluster to run multi-node tasks.

The --network argument accepts the IP address range (CIDR) of the private network of the instance.

Example:

dstack pool add-ssh -i ~/.ssh/id_rsa [email protected] --network 10.0.0.0/24

Once you've added multiple instances with the same network value, you'll be able to use them as a cluster to run multi-node tasks.

Private subnets

By default, dstack uses public IPs for SSH access to running instances, requiring public subnets in the VPC. The new update allows AWS instances to use private subnets instead.

To create instances only in private subnets, set public_ips to false in the AWS backend settings:

type: aws
  creds:
    type: default
  vpc_ids:
     ...
  public_ips: false

Note

  • Both dstack server and the dstack CLI should have access to the private subnet to access instances.
  • If you want running instances to access the Internet, the private subnets need to have a NAT gateway.

Gateways

dstack apply

Previously, to create or update gateways, one had to use the dstack gateway create or dstack gateway update commands.
Now, it's possible to define a gateway configuration via YAML and create or update it using the dstack apply command.

Example:

type: gateway
name: example-gateway

backend: gcp
region: europe-west1
domain: example.com
dstack apply -f examples/deployment/gateway.dstack.yml

For now, the dstack apply command only supports the gateway configuration type. Soon, it will also support dev-environment, task, and service, replacing the dstack run command.

The dstack destroy command can be used to delete resources.

Private gateways

By default, gateways are deployed using public subnets. Since 0.18.2, it is now possible to deploy gateways using private subnets. To do this, you need to set public_ips to false and specify the ARN of a certificate from AWS Certificate Manager.

type: gateway
name: example-gateway

backend: aws
region: eu-west-1
domain: "example.com"

public_ip: false
certificate:
  type: acm
  arn: "arn:aws:acm:eu-west-1:3515152512515:certificate/3251511125--1241-1224-121251515125"

In this case, dstack will deploy the gateway in a private subnet behind a load balancer using the specified certificate.

Note

Private gateways are currently supported only for AWS.

What's changed

New Contributors

Full Changelog: 0.18.1...0.18.2

Contributors

TheBits, Bihan, and 3 other contributors
Assets 2
Loading

0.18.1

29 Apr 15:47
@peterschmidt85 peterschmidt85
0.18.1
2761a90
Compare
Choose a tag to compare
0.18.1

On-prem servers

Now you can add your own servers as pool instances:

dstack pool add-ssh -i ~/.ssh/id_rsa [email protected]

Note

The server should be pre-installed with CUDA 12.1 and NVIDIA Docker.

Configuration

All .dstack/profiles.yml properties now can be specified via run configurations:

type: dev-environment

ide: vscode

spot_policy: auto
backends: ["aws"]

regions: ["eu-west-1", "eu-west-2"]

instance_types: ["p3.8xlarge", "p3.16xlarge"]
max_price: 2.0

max_duration: 1d

New examples 🔥🔥

Thanks to the contribution from @deep-diver, we got two new examples:

Other

  • Configuring VPCs using their IDs (via vpc_ids in server/config.yml)
  • Support for global profiles (via ~/.dstack/profiles.yml)
  • Updated the default environment variables (DSTACK_RUN_NAME, DSTACK_GPUS_NUM, DSTACK_NODES_NUM, DSTACK_NODE_RANK, and DSTACK_MASTER_NODE_IP)
  • It’s now possible to use NVIDIA A10 GPU on Azure
  • More granular permissions for Azure

What's changed

Full Changelog: 0.18.0...0.18.1rc2

Contributors

TheBits, Bihan, and 3 other contributors
Assets 2
Loading

0.18.0

10 Apr 15:46
@peterschmidt85 peterschmidt85
0.18.0
f9b941c
Compare
Choose a tag to compare
0.18.0

RunPod

The update adds the long-awaited integration with RunPod, a distributed GPU cloud that offers GPUs at affordable prices.

To use RunPod, specify your RunPod API key in ~/.dstack/server/config.yml:

projects:
- name: main
  backends:
  - type: runpod
    creds:
      type: api_key
      api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9

Once the server is restarted, go ahead and run workloads.

Clusters

Another major change with the update is the ability to run multi-node tasks over an interconnected cluster of instances.

type: task

nodes: 2

commands:
  - git clone https://github.com/r4victor/pytorch-distributed-resnet.git
  - cd pytorch-distributed-resnet
  - mkdir -p data
  - cd data
  - wget -c --quiet https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
  - tar -xvzf cifar-10-python.tar.gz
  - cd ..
  - pip3 install -r requirements.txt torch
  - mkdir -p saved_models
  - torchrun --nproc_per_node=$DSTACK_GPUS_PER_NODE 
     --node_rank=$DSTACK_NODE_RANK 
     --nnodes=$DSTACK_NODES_NUM
     --master_addr=$DSTACK_MASTER_NODE_IP
     --master_port=8008 resnet_ddp.py 
     --num_epochs 20

resources:
  gpu: 1

Currently supported providers for this feature include AWS, GCP, and Azure.

Other

  • The commands property is now not required for tasks and services if you use an image that has a default entrypoint configured.
  • The permissions required for using dstack with GCP are more granular.

What's changed

Full changelog: 0.17.0...0.18.0

Contributors

TheBits, Bihan, and 3 other contributors
Assets 2
Loading