Code

[POC] Updated PR for spin steps#2209

Code

Open

talsperre wants to merge 28 commits intomasterNetflix/metaflow:masterfrom dev/spin-stepsNetflix/metaflow:dev/spin-stepsCopy head branch name to clipboard

+1,837-71

Commits

Commits on Jan 23, 2025

• Add static and runtime dag info, API to fetch ancestor tasks

  talsperre

  committed
  171013f

  Copy full SHA for 171013f

• Add API to get immediate successors

  talsperre

  committed
  a842dc1

  Copy full SHA for a842dc1

• Add API for getting closest siblings

  talsperre

  committed
  3f9fdfd

  Copy full SHA for 3f9fdfd

• Update metadata API params

  talsperre

  committed
  4d0298d

  Copy full SHA for 4d0298d

• Refactor ancestor and successor client code

  talsperre

  committed
  a42f31f

  Copy full SHA for a42f31f

• Remove unneccessary prints

  talsperre

  committed
  493c3fa

  Copy full SHA for 493c3fa

• Support querying ancestors and successors in local metadata provider

  talsperre

  committed
  26a0a6a

  Copy full SHA for 26a0a6a

• Refactor and simplify client code

  talsperre

  committed
  5f092e7

  Copy full SHA for 5f092e7

• Make query logic more descriptive

  talsperre

  committed
  890ff92

  Copy full SHA for 890ff92

• Add core tests for ancestor task API

  talsperre

  committed
  7f91bf8

  Copy full SHA for 7f91bf8

• Add core test for immediate successor API

  talsperre

  committed
  0cfea97

  Copy full SHA for 0cfea97

• Add endpoint in OSS metadata service

  talsperre

  committed
  6bd0778

  Copy full SHA for 6bd0778

• Add logs to tests

  talsperre

  committed
  b92581b

  Copy full SHA for b92581b

• Log for each stack to metadata, update query logic

  talsperre

  committed
  6002395

  Copy full SHA for 6002395

• Add more comments to code

  talsperre

  committed
  d505d87

  Copy full SHA for d505d87

• Run black formatting

  talsperre

  committed
  f71e26b

  Copy full SHA for f71e26b

• Set monitor to None in filter tasks API

  talsperre

  committed
  536278d

  Copy full SHA for 536278d

• import urlencode

  talsperre

  committed
  2559999

  Copy full SHA for 2559999

• Address comments

  talsperre

  committed
  c172ef2

  Copy full SHA for c172ef2

• Update logic for siblings, make it work for static splits as well

  talsperre

  committed
  c825d66

  Copy full SHA for c825d66

• Initial commit for spin steps

  talsperre

  committed
  81aca61

  Copy full SHA for 81aca61

• Make it work with subprocess

  talsperre

  committed
  5cb5292

  Copy full SHA for 5cb5292

• Make it work with subprocess and truncated buffers

  talsperre

  committed
  23d7f12

  Copy full SHA for 23d7f12

• dummy commit

  talsperre

  committed
  f4df5db

  Copy full SHA for f4df5db

Commits on Jan 25, 2025

• Dummy commit

  talsperre

  committed
  30f170c

  Copy full SHA for 30f170c

Commits on Jan 26, 2025

• Working implementation

  talsperre

  committed
  2b821d0

  Copy full SHA for 2b821d0

• Use polling for spin logging

  talsperre

  committed
  6f57368

  Copy full SHA for 6f57368

Commits on Jan 27, 2025

• Make spin work with runner API

  talsperre

  committed
  86b872e

  Copy full SHA for 86b872e