Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

'Remove step' feature for local workspaces #588

Merged
merged 38 commits into from
Oct 5, 2023
Merged
Changes from 1 commit
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
ab72783
remove step feature for local workspaces
Jul 12, 2023
2f04716
cleaned code for remove step cache for local workspaces
Jul 13, 2023
f9d112a
resolved comments for local workspace
Jul 14, 2023
1565658
remove cached step for memory workspaces
Jul 14, 2023
3795678
tests added
Jul 18, 2023
6f3d60c
fixed style issues
Jul 18, 2023
ddcbf9e
fixed style issues +1
Jul 18, 2023
fe06aeb
step cache place holder
Jul 18, 2023
48661a6
adding tests
Jul 18, 2023
1c68405
fixed styling in tests
Jul 18, 2023
bc289c2
abstraction tag added
Jul 19, 2023
c2bf197
cache remove for GCS remote space
Jul 20, 2023
525bdb7
fixed style and type issues
Jul 20, 2023
bebf5ce
style issues
Jul 21, 2023
b31c230
cache remove for beaker workspace
Jul 25, 2023
df5462b
style check
Jul 25, 2023
da9202f
suggested changes + tests
Jul 27, 2023
9b90608
style changes fixed
Jul 27, 2023
59d44e4
tests for remote workspaces
Jul 27, 2023
0cbd3f9
style and type checks
Jul 27, 2023
e4749a3
incorporated suggestions
Jul 27, 2023
889298c
adding PR review changes
Jul 27, 2023
e1a37bd
Update tango/integrations/beaker/workspace.py
pranjali97 Sep 13, 2023
df93c21
Update tango/step_caches/local_step_cache.py
pranjali97 Sep 14, 2023
9be5ff8
Update tango/step_cache.py
pranjali97 Sep 14, 2023
b5a65d6
Update tango/step_caches/remote_step_cache.py
pranjali97 Sep 14, 2023
0c2d101
make lint happy
AkshitaB Sep 14, 2023
9b21a09
fix type
AkshitaB Sep 14, 2023
97e9909
Update tests/integrations/beaker/workspace_test.py
pranjali97 Sep 15, 2023
b164eb6
add missing import
AkshitaB Sep 15, 2023
1e59e14
we shouldn't have to update this list everytime a new model is added …
AkshitaB Sep 15, 2023
945451b
ignore type error
AkshitaB Sep 15, 2023
0294f8d
explicitly add jaxlib
AkshitaB Sep 15, 2023
107dd94
wrong place
AkshitaB Sep 15, 2023
d03a50b
Update CHANGELOG.md
pranjali97 Sep 15, 2023
93dafde
update cache
AkshitaB Sep 18, 2023
dd8e852
Update main.yml
AkshitaB Sep 18, 2023
bb2a61d
Merge branch 'main' into remove_step_branch
AkshitaB Oct 4, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions tango/workspace.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
from urllib.parse import ParseResult, urlparse

import pytz
from sqlitedict import SqliteDict
import shutil

from .common import Registrable
from .common.from_params import FromParams
Expand Down Expand Up @@ -419,6 +421,29 @@ def step_result(self, step_name: str) -> Any:
return self.step_cache[run.steps[step_name]]
raise KeyError(f"No step named '{step_name}' found in previous runs")


def remove_step(self, step_name: str) -> Any:
pranjali97 marked this conversation as resolved.
Show resolved Hide resolved
"""
Get Step name (Unique ID) from the user and remove the step information from cache
:raises KeyError: If no step with the unique name found in the cache dir
"""
# get path to dir dynamically
sqlite_path = self.dir / "stepinfo.sqlite"
with SqliteDict(sqlite_path) as d:
try:
step_location = d[step_name].result_location
# remove step info from the sqlite dict
del d[step_name]
d.commit()
# remove cache directory
try:
shutil.rmtree(step_location)
except OSError:
raise OSError('Step Cache folder not found')
return('Step deleted')
pranjali97 marked this conversation as resolved.
Show resolved Hide resolved
except KeyError:
raise KeyError(f"No step named '{step_name}' found")

def capture_logs_for_run(self, name: str) -> ContextManager[None]:
"""
Should return a context manager that can be used to capture the logs for a run.
Expand Down