Improved Automation Experience

[Fryguy]

This feature spec proposes a new automation experience that is much less complicated. The underlying driver for this change is actually to create an improved service catalog experience. However, as one builds out their service catalog, the first stumbling block is the need to build a dialog, and the automation behind it. Writing Automate is incredibly complicated, and requires a lot of knowledge of the system in addition to Automate itself (there's literally an entire book). This feature spec is to design a new experience that is much more user friendly, and allows users to build automation in a much more natural way.

Tracking issues:

• [RFE] ManageIQ Workflows - Backend #22311
• [RFE] ManageIQ Workflows - UI manageiq-ui-classic#8715

Overview

We will create a new a new step based automation system. Each step will execute a container. We will support import/export and even in-browser development by serializing to Amazon State Languages (ASL). It is the responsibility of the end user to build and support the containers with the execution of each step.

Considerations

There are some considerations about the current system that we need to take into account.

• One of the more widely used features of Automate is the ability to create and modify a State Machine, being able to execute custom code at each state. Currently users create and modify existing state machines, but doing so within Automate is complicated and the interface is not user friendly.
• Users want to reuse existing state machines within other state machines.
• Users want to be able to execute in any language they are familiar with, and not just Ruby or Ansible, however Ruby and Ansible will be required for existing users that have already written automation in Automate.

Design

For the purposes of this document I am referring to this step-function-like automation as "workflows".

Backend

ASL Runtime/Orchestration

• Gem https://github.com/ManageIQ/manageiq-floe
• ASL compliant runner
• An ASL Task Resource will be a docker container
  • Needs to run on podified (kube), appliances (podman), and local development (podman/docker)
• Safely handle credentials when giving them to the container

Modeling

• Built as a separate "Embedded Automation Provider" - https://github.com/ManageIQ/manageiq-providers-workflows
• "repositories" model for git repository management
  • Reuse the ConfigurationScriptSource model as a base
  • Detect presence of workflows in repository, and synchronize them to "Workflow" records.
• "credentials" model for the credentials to be used by workflows
  • Reuse the Authentication model as a base
  • These credentials will be tied to the provider instance, and will be used by the workflow when it is executed
  • Need to be able to associate specific credential records to the workflow
• "workflows" model for individual workflow ASL files
  • Reuse the ConfigurationScriptPayload model as a base
  • payload column to be used to embed the ASL directly
  • credentials column to be used for credential mapping needed by the workflow to know which fields map to which credentials
    • Validation on deletions of credentials should not be allowed if associated to a workflow
• "workflow instances" model for the execution of a workflow
  • Reuse the ConfigurationScript model as a base
  • context field will store the running context of the workflow. context encapsulates the entire workspace of the workflow

API

• Reuse existing API endpoints where possible
  • /api/configuration_script_sources for repositories
  • /api/configuration_script_payloads for workflows
  • /api/configuration_scripts for workflow instances
  • /api/authentications (or /api/providers/:id/authentications) for credentials

UI

• Modify existing integration points
  • Anywhere a user can choose a workflow, we need to replace that with a way to choose a Workflow or an Automate Entry point.
    • Service Dialog fields
    • Provisioning/Reconfigure/Retirement Entry points in a service catalog item
• New Menu & UI for Workflows Automate -> Embedded Workflows (can be mostly copied from Embedded Ansible)
  • Repositories
    • Import from git
    • Copy from Embedded Ansible to start. Later we may want to consolidate with Embedded Ansible's Repositories
  • Workflows
    • Text view
      • JSON representation of ASL color coded with CodeMirror
    • Graph view
      • Nodes and lines view (https://www.npmjs.com/package/@tshepomgaga/aws-sfn-graph)
    • The above 2 forms can be represented with tabs or with an icon indicator similar to how we used to do Grid-Tile-List view
    • Copy from Embedded Ansible to start. Later we may want to consolidate with Embedded Ansible's Playbooks
  • Credentials
    • Credential types should come from the provider similar to how Embedded Ansible credentials come from the provider

Documentation

• All of the above
• Need a way to document adding docker registries and their credential
  • How to add to an appliance
  • How to add to a kubernetes cluster

Phase 2

• New UI for workflow instances
  • Not sure where this should live.
    • Requests? / My Tasks?
    • A tab in a request similar to how Ansible playbooks show their output?
  • Show the current runtime state of the workflow
    • Text version is a table showing the state's start/end time, inputs/outputs errors, logs, etc
      • Also need to be able to see the JSON format of the workflow that was used to execute the instance as it may be different from the original workflow.
    • Graph UI (reuse the graph UI for runtime presentation)
      • Highlight current step
      • Have details available if a previous step is clicked on
        • Show start/end time, inputs/outputs, errors, logs, etc
      • If a step fails we can highlight it red, if a step succeeds highlight it green, current step highlight in yellow?

Future Plans

• Out of the box base containers for at least Ruby (later: Shell, Ansible Execution Environments (EE), Python, Javascript)
• Predefined Steps
  • Send Email
  • Call Webhook
  • Maybe more complex stuff like Open a ServiceNow ticket?
• Dev/Test/Prod workflow
  • How to test / dry-run workflows; Simulate action? (or is this just Launch with a dry-run parameter?)
  • Rely on git branches?
• Allow providers to bring their own workflows, such as provisioning.
• Optimization to fetch all images ahead of time
• Authoring without a git repository
  • UI under "Workflows" to create/update/delete a workflow or import from a JSON file
  • Export workflows to .zip or .tgz
• More details in graph view of a workflow
  • Details in right side panel when a node is clicked
  • In authoring mode have a toolbar palette of different node types
    • All ASL node types
    • In the future, allow for selection other workflows to embed one workflow within another.
• More integration points
  • Automate methods
  • Event handlers
  • Policy actions

Backward Compatibility

We do not intend for this to be directly backward compatible with Automate, and therefore we don't intend for there to be a migration plan. If this new Workflow system is better, we will manually migrate existing things, such as the provisioning and retirement Automate models.

Alternatives considered

• Apache AirFlow
• GitHub Actions
• Tekton

Videos

• Workflows PoC 1st demo (no voiceover): https://youtu.be/SzjqrGBft6A
• Workflows PoC 1st demo (voiceover during Sprint Review): https://youtu.be/G_Zt8C5z3lc?t=600

[chessbyte]

Exciting initiative and long overdue!

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 3 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[Fryguy]

Marking this as completed. All future changes will be done in a follow up issues/PRs.