Code

[WIP] Slurm agent#3005

Code

Draft

JiangJiaWei1103 wants to merge 31 commits intoflyteorg:masterflyteorg/flytekit:masterfrom JiangJiaWei1103:slurm-agent-devJiangJiaWei1103/flytekit:slurm-agent-devCopy head branch name to clipboard

+764-12

Commits

Commits on Dec 14, 2024

• Add slurm plugin blank components

  Show description for 421d1b8
  JiangJiaWei1103

  committed
  421d1b8

  Copy full SHA for 421d1b8

Commits on Dec 16, 2024

• feat: Add naive slurm agent create and get with rest api

  Show description for 1d1f806
  JiangJiaWei1103

  committed
  1d1f806

  Copy full SHA for 1d1f806

Commits on Dec 17, 2024

• Use asyncssh instead of REST API

  Show description for 5d97126
  JiangJiaWei1103

  committed
  5d97126

  Copy full SHA for 5d97126

Commits on Dec 18, 2024

• Test ssh communication and run sbatch

  Show description for 2e7f0f2
  JiangJiaWei1103

  committed
  2e7f0f2

  Copy full SHA for 2e7f0f2

Commits on Dec 19, 2024

• Add delete method and support slurm job state

  Show description for 9644b99
  JiangJiaWei1103

  committed
  9644b99

  Copy full SHA for 9644b99

Commits on Dec 27, 2024

• feat: Submit and run SlurmTask on a remote Slurm cluster

  Show description for e41b181
  JiangJiaWei1103

  committed
  e41b181

  Copy full SHA for e41b181

Commits on Dec 28, 2024

• refactor: Remove redundant task_module transfer

  Show description for 6db24dc
  JiangJiaWei1103

  committed
  6db24dc

  Copy full SHA for 6db24dc

• refactor: Remove redundant env var

  Show description for 122c7f1
  JiangJiaWei1103

  committed
  122c7f1

  Copy full SHA for 122c7f1

Commits on Dec 30, 2024

• docs: Add env setup guide for local test

  Show description for e9760a7
  JiangJiaWei1103

  committed
  e9760a7

  Copy full SHA for e9760a7

• docs: Add links and figures

  Show description for e68fda9
  JiangJiaWei1103

  committed
  e68fda9

  Copy full SHA for e68fda9

• docs: Fix commit sha

  Show description for 470637c
  JiangJiaWei1103

  committed
  470637c

  Copy full SHA for 470637c

• docs: Fix commit sha for demo guide

  Show description for 1579ab4
  JiangJiaWei1103

  committed
  1579ab4

  Copy full SHA for 1579ab4

• docs: Fix links

  Show description for 0e538f0
  JiangJiaWei1103

  committed
  0e538f0

  Copy full SHA for 0e538f0

Commits on Dec 31, 2024

• feat: Support SSH config in task config

  Show description for 8229418
  JiangJiaWei1103

  committed
  8229418

  Copy full SHA for 8229418

• docs: Include ssh config in demo example

  Show description for 9e6d8a6
  JiangJiaWei1103

  committed
  9e6d8a6

  Copy full SHA for 9e6d8a6

Commits on Jan 7, 2025

• refactor: Reduce ssh_conf option to slurm_host only

  Show description for e07b09a
  JiangJiaWei1103

  committed
  e07b09a

  Copy full SHA for e07b09a

• feat: Support Slurm agent with ShellTask

  Show description for 3a7eb6d
  JiangJiaWei1103

  committed
  3a7eb6d

  Copy full SHA for 3a7eb6d

Commits on Jan 9, 2025

• feat: Simplify Slurm job submission logic

  Show description for a815fd9
  JiangJiaWei1103

  committed
  a815fd9

  Copy full SHA for a815fd9

Commits on Jan 10, 2025

• Added script args to agent and task

  Show description for a3ea014
  pryce-turner

  committed
  a3ea014

  Copy full SHA for a3ea014

Commits on Jan 11, 2025

• Add asyncssh to dependencies

  Show description for a109bd8
  JiangJiaWei1103

  committed
  a109bd8

  Copy full SHA for a109bd8

• docs: Update setup and demo for a basic use case

  Show description for e5da665
  JiangJiaWei1103

  committed
  e5da665

  Copy full SHA for e5da665

• docs: Update basic arch figure path

  Show description for 0a3d9f1
  JiangJiaWei1103

  committed
  0a3d9f1

  Copy full SHA for 0a3d9f1

• docs: Fix typo and hyperlink

  Show description for 1b0f6df
  JiangJiaWei1103

  committed
  1b0f6df

  Copy full SHA for 1b0f6df

• fix: A tmp workaround to test agent locally without container_image

  Show description for 26cc201
  JiangJiaWei1103

  committed
  26cc201

  Copy full SHA for 26cc201

Commits on Jan 14, 2025

• feat: Support user-defined batch script content with SlurmShellTask

  Show description for 16d953e
  JiangJiaWei1103

  committed
  16d953e

  Copy full SHA for 16d953e

Commits on Jan 15, 2025

• feat: Fall back to PythonTask for naive use cases

  Show description for c743917
  JiangJiaWei1103

  committed
  c743917

  Copy full SHA for c743917

• refactor: Define Slurm as a base task config and extend for remote script

  Show description for e365dee
  JiangJiaWei1103

  committed
  e365dee

  Copy full SHA for e365dee

Commits on Jan 16, 2025

• feat: Support PythonFunctionTask and reorganize agent structure

  Show description for c1064d4
  JiangJiaWei1103

  committed
  c1064d4

  Copy full SHA for c1064d4

Commits on Jan 22, 2025

• Use poetry virtual env to avoid contamination

  Show description for 361fbd1
  JiangJiaWei1103

  committed
  361fbd1

  Copy full SHA for 361fbd1

Commits on Jan 23, 2025

• docs: Complete local test env setup process

  Show description for fc3e34e
  JiangJiaWei1103

  committed
  fc3e34e

  Copy full SHA for fc3e34e

• docs: Add use cases ranging from basic to advanced

  Show description for 5cd58ec
  JiangJiaWei1103

  committed
  5cd58ec

  Copy full SHA for 5cd58ec