Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Service option update #378

Merged
merged 232 commits into from
Sep 27, 2024
Merged

Service option update #378

merged 232 commits into from
Sep 27, 2024

Conversation

ZergLev
Copy link
Collaborator

@ZergLev ZergLev commented Jul 30, 2024

Description

  • ServiceGroups can now run async components in groups within itself, instead of not doing that if there's even one sequential component inside. This removes the need to make lots of ServiceGroups just to run some functions in parallel.
    Basically, [a, s, a, a, s] -> a, s, (a, a), s, where 'a' and 's' are 'asynchronous' and 'sequential' components respectively. So, it's easier to define async components in ServiceGroups now. The user can mark any of them as asynchronous, by default all PipelineComponents are sequential except the Actor.
  • The 'asynchronous' field will not be inherited by ServiceGroups that contain a component with 'sequential' = True. True, the user can make a mistake, but it allows for the program to be faster, so it's a reasonable tradeoff.
  • Reduced PipelineComponent __call__ signature to (ctx: Context) -> None -- since 'Core Rework' Pipeline can be accessed from Context; component info should be obtained from self via subclassing from Service and defining a custom call(self, ctx: Context) method. BaseProcessing is also accepted in place of the service.handler (Service, ServiceGroup and Actor are all instances of PipelineComponent)
  • Made service conditions inherit from BaseCondition -- able to reuse built-in conditions for services
  • Changed service_successful_condition to ServiceFinished condition which awaits until the service is either finished or failed.
  • All service paths are recalculated during pipeline._run_pipeline() just to be safe.
  • Pipeline tutorials now have logs in console output, they are no longer red boxes of text.
  • Increased Service test coverage to 100%.
  • Numerous minor changes / fixes.

Checklist

  • "Service Option Update" done
  • "Service Update" done and merged
  • Updated relevant tutorials
  • Updated relevant API
  • Full Service test coverage done.
  • Self-review done
  • Review changes done

To Consider

  • Represent service state dict to make it collapse finished groups; leave only failed services in failed groups
  • There's potential to subclass Service from BaseProcessing.
  • There was an opinion that Extra Handlers could just be made into Services. (I think it'd take a lot of time to change now and I also think Extra Handlers do have some value)
  • Update API reference / tutorials / guides
  • Update .ignore files, scripts (such as lint), distribution manifest (if files are added/deleted)
  • Search for references to changed entities in the codebase

…s to from_script() before, that's not handled right now)
…uted_fields from Pipeline, they're not working as intended
This is error-prone and not needed.
Now renaming does not happen if there are no collisions;
Postfix is added with "#" to further avoid collisions.
Use add_extra_handler on group instead
@RLKRo RLKRo merged commit 1392a28 into dev Sep 27, 2024
10 of 17 checks passed
@RLKRo RLKRo mentioned this pull request Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants