Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve speed for workflows with hundreds of items #264

Closed
wants to merge 6 commits into from
Closed

Improve speed for workflows with hundreds of items #264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
90f4991
ton of logging, narrowed hot spots down to a couple of spots
rebrowning Jan 13, 2024
4c0f102
clean up current log statements
rebrowning Jan 16, 2024
2b2d48c
a little bit more cleanup
rebrowning Jan 16, 2024
d7db46c
fix linting error
rebrowning Jan 18, 2024
52b8245
a bit of cleanup based on feedback
rebrowning Jan 23, 2024
f8d2021
cleanup
rebrowning Jan 23, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion orquesta/specs/native/v1/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ def render(self, in_ctx):

action_specs.append(action_spec)
else:
item_ctx_value = ctx_util.copy_context(in_ctx)
items_spec = self.get_items_spec()

if " in " not in items_spec.items:
Expand All @@ -189,7 +190,7 @@ def render(self, in_ctx):
elif item_keys and len(item_keys) == 1:
item = {item_keys[0]: item}

item_ctx_value = ctx_util.set_current_item(in_ctx, item)
item_ctx_value = ctx_util.set_current_item(item_ctx_value, item)

action_spec = {
"action": expr_base.evaluate(self.action, item_ctx_value),
Expand Down
10 changes: 9 additions & 1 deletion orquesta/utils/context.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,19 @@ def set_current_task(context, task):
return ctx


def set_current_item(context, item):
def copy_context(context):
if context and not isinstance(context, dict):
raise TypeError("The context is not type of dict.")

ctx = json_util.deepcopy(context) if context else dict()
return ctx


def set_current_item(context, item):
if context and not isinstance(context, dict):
raise TypeError("The context is not type of dict.")

ctx = {**context}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a shallow instead of deep copy , correct?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would actually leave this function as is and just do the operation in render.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, I'll move it over. And yes it's a shallow copy, but intentional. The rest of the sub-objects that are copied in a deep copy don't actually change between iteration from what I saw in testing, but the top level is what needs to change which a shallow copy still facilitates.

Copy link
Author

@rebrowning rebrowning Jan 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guzzijones thoughts on undoing the changes to set_current_item, that function is not used anywhere else in orquesta. set_current_item would not work at the current location that copy_context is being used, so we could either remove the set_current_item function and move the logic into the render function, or leave set_current_item in the current form

had to resend with the correct profile, switched to my laptop and forgot to switch

ctx["__current_item"] = item

return ctx