Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve speed for workflows with hundreds of items #264

Closed
wants to merge 6 commits into from
Closed

Improve speed for workflows with hundreds of items #264

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
90f4991
ton of logging, narrowed hot spots down to a couple of spots
rebrowning Jan 13, 2024
4c0f102
clean up current log statements
rebrowning Jan 16, 2024
2b2d48c
a little bit more cleanup
rebrowning Jan 16, 2024
d7db46c
fix linting error
rebrowning Jan 18, 2024
52b8245
a bit of cleanup based on feedback
rebrowning Jan 23, 2024
f8d2021
cleanup
rebrowning Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion orquesta/conducting.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,8 @@ def get_staged_tasks(self, filtered=True):
if not filtered:
return self.staged

return [x for x in self.staged if x["ready"] and not x.get("completed", False)]
resp = [x for x in self.staged if x["ready"] and not x.get("completed", False)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this change is not needed, correct?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct, I'll revert some of the additional changes that were made (just moved variables out of return statements to get timing on them, didn't move back)

return resp

@property
def has_staged_tasks(self):
Expand Down
12 changes: 7 additions & 5 deletions orquesta/specs/native/v1/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ def has_retry(self):
def render(self, in_ctx):
action_specs = []

item_ctx_value = ctx_util.copy_context(in_ctx)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

having the copy here will slow down non with items . move the copy in the else clause.


if not self.has_items():
action_spec = {
"action": expr_base.evaluate(self.action, in_ctx),
Expand Down Expand Up @@ -189,14 +191,14 @@ def render(self, in_ctx):
elif item_keys and len(item_keys) == 1:
item = {item_keys[0]: item}

item_ctx_value = ctx_util.set_current_item(in_ctx, item)

item_ctx_value = ctx_util.set_current_item(item_ctx_value, item)
action = expr_base.evaluate(self.action, item_ctx_value)
gen_input = expr_base.evaluate(getattr(self, "input", {}), item_ctx_value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is not needed? were you just testing the variable value here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

item_ctx_value is needed i see. 195 and 196 can be reverted.

action_spec = {
"action": expr_base.evaluate(self.action, item_ctx_value),
"input": expr_base.evaluate(getattr(self, "input", {}), item_ctx_value),
"action": action,
"input": gen_input,
"item_id": idx,
}

action_specs.append(action_spec)

return self, action_specs
Expand Down
10 changes: 9 additions & 1 deletion orquesta/utils/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,19 @@ def set_current_task(context, task):
return ctx


def set_current_item(context, item):
def copy_context(context):
if context and not isinstance(context, dict):
raise TypeError("The context is not type of dict.")

ctx = json_util.deepcopy(context) if context else dict()
return ctx


def set_current_item(context, item):
if context and not isinstance(context, dict):
raise TypeError("The context is not type of dict.")

ctx = {**context}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a shallow instead of deep copy , correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would actually leave this function as is and just do the operation in render.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I'll move it over. And yes it's a shallow copy, but intentional. The rest of the sub-objects that are copied in a deep copy don't actually change between iteration from what I saw in testing, but the top level is what needs to change which a shallow copy still facilitates.

Copy link
Author

@rebrowning rebrowning Jan 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guzzijones thoughts on undoing the changes to set_current_item, that function is not used anywhere else in orquesta. set_current_item would not work at the current location that copy_context is being used, so we could either remove the set_current_item function and move the logic into the render function, or leave set_current_item in the current form

had to resend with the correct profile, switched to my laptop and forgot to switch

ctx["__current_item"] = item

return ctx