Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add service for event and integration with Prefect event #4493

Merged
merged 1 commit into from
Sep 30, 2024

Conversation

dgarros
Copy link
Collaborator

@dgarros dgarros commented Sep 29, 2024

Initial integration of Infrahub events with prefect event

To avoid having to define/send event twice, I've introduced a new event service that will automatically send event to the message bus and to prefect.
This PR also introduces a new InfrahubEvent dataclass which support both formats: internal bus and prefect

For now I've only implemented the node mutated event for testing.

I'm still exploring how events are working in prefect and what is the best way to integrate with them.
in prefect an event must have be associated with a resource and can have multiple related resources.
Both resource and related resources are defined as a dictionary of key: value pair with the following constraints

  • resource must include at least prefect.resource.id
  • related resources must include at least prefect.resource.id and prefect.resource.role

below is the payload of the event generated after creating a new node in prefect

{
  "id": "7388044d-2faa-4d83-ba03-7225bf5ea276",
  "event": "infrahub.node.added",
  "occurred": "2024-09-29T15:39:22.972Z",
  "payload": {
    "id": "17f9c134-2f6a-c4d3-38ed-c517bcdee703",
    "name": {
      "id": "17f9c134-4ef8-faa3-38e7-c513cc3cb884",
      "owner": null,
      "value": "BLUE",
      "source": null,
      "__typename": "Text",
      "is_visible": true,
      "is_protected": false
    },
    "type": "BuiltinTag",
    "__typename": "BuiltinTag",
    "description": {
      "id": "17f9c134-4ef9-be70-38e2-c51048ea7d81",
      "owner": null,
      "source": null,
      "__typename": "Text",
      "is_visible": true,
      "is_protected": false
    },
    "display_label": "BLUE"
  },
  "received": "2024-09-29T15:39:22.994Z",
  "related": [
    {
      "prefect.resource.id": "infrahub.request.10e771ce781f4314acbf2307029a42ba",
      "prefect.resource.role": "request"
    },
    {
      "prefect.resource.id": "infrahub.source.80af6c07-61f4-461c-84db-ed85ff874daa",
      "prefect.resource.role": "event_source"
    },
    {
      "prefect.resource.id": "infrahub.branch",
      "prefect.resource.name": "main",
      "prefect.resource.role": "branch"
    }
  ],
  "resource": {
    "infrahub.node.kind": "BuiltinTag",
    "prefect.resource.id": "infrahub.node.17f9c134-2f6a-c4d3-38ed-c517bcdee703",
    "infrahub.node.action": "added"
  }
}

@github-actions github-actions bot added the group/backend Issue related to the backend (API Server, Git Agent) label Sep 29, 2024
@dgarros dgarros marked this pull request as ready for review September 29, 2024 18:44
@dgarros dgarros requested a review from a team September 29, 2024 18:44
backend/infrahub/events/models.py Outdated Show resolved Hide resolved
branch=context.branch.name,
kind=obj._schema.kind,
namespace=obj._schema.namespace,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might have a bit limiting previously to only have kind but I think the addition of namespace here gets a bit weird if we keep kind. We should either just leave it as "kind" or remove kind if we have "name" and "namespace.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed it for now, we can revisit that later but having just the kind is not enough in my perspective if we need to do some filtering

backend/infrahub/services/adapters/event/__init__.py Outdated Show resolved Hide resolved
resource=event.get_resource(),
related=event.get_related(),
payload=event.get_payload(),
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks a bit problematic, is there an async version of emit_event?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I could find quickly but as far as I understand they are sending the event in a non blocking way. I'm planning to dive deeper into it soon.
We may have to create our own function to send the event but for now since we are sending the events in a background task I don't think it will make a big different

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right the background tasks are run in a separate thread regardless. But we'll have to keep it in mind if used elsewhere, though if Prefects emit_event does the same it should be fine.

@dgarros dgarros merged commit 8313077 into develop Sep 30, 2024
31 checks passed
@dgarros dgarros deleted the dga-20240929-event-service branch September 30, 2024 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
group/backend Issue related to the backend (API Server, Git Agent)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants