feat(): Introducing support for DataHub Events Source

[jjoyce0510]

As of v0.3.7 of DataHub Cloud, connecting DataHub actions is supported.

Summary

• Add new DataHubEventsSource for reading events from DataHub Cloud versions > v0.3.7.
• Add unit tests
• Add documentation

QA & Validation

Verified that the integration is working correctly with longtailcompanions.acryl.io:

Hello world! Received event:
{
    "event_type": "EntityChangeEvent_v1",
    "event": {
        "entityType": "assertion",
        "entityUrn": "urn:li:assertion:aed3ec48-5dbb-40d5-8942-e44a2ff3fa78",
        "category": "RUN",
        "operation": "COMPLETED",
        "auditStamp": {
            "time": 1737418020285,
            "actor": "urn:li:corpuser:__datahub_system"
        },
        "version": 0,
        "parameters": {
            "asserteeUrn": "urn:li:dataset:(urn:li:dataPlatform:snowflake,test_db.public.purchase_event,PROD)",
            "runId": "urn:li:assertion:aed3ec48-5dbb-40d5-8942-e44a2ff3fa78-1737418020282",
            "runResult": "COMPLETE",
            "assertionResult": "SUCCESS"
        }
    },
    "meta": {
        "batch_id": 1,
        "msg_id": 2
    }
}

[github-actions]

Unit Test Results (build & test)

94 tests  +31   94 ✅ +31   14s ⏱️ +10s
 1 suites ± 0    0 💤 ± 0 
 1 files   ± 0    0 ❌ ± 0 

Results for commit 226a305. ± Comparison against base commit c0fc442.

♻️ This comment has been updated with latest results.

[gabe-lyons]

does this need any try/catch?

[jjoyce0510]

no - this should raise an exception if it fails to parse. all i would do here is wrap and raise the exception. i can do that to add context but this indicates a fundamental issue

[gabe-lyons]

prefer urn_builder here

[jjoyce0510]

This is a pretty rarely constructured urn from python, so we don't have a utility for it: https://github.com/datahub-project/datahub/blob/8ac35fa9ee6229ab7b8c0f4b2a0e669113574cd1/metadata-ingestion/src/datahub/emitter/mce_builder.py#L125 (unless you're referring to another set of utils)

[gabe-lyons]

why not grant this to all Admins? Admins already have the ability to read all metadata. This seems like it will be a constant source of frustration.

[jjoyce0510]

Currently, the root user does have it but they are the only ones. We can grant to admins once we validate that everything is working. I agree it could be frustrating

[jjoyce0510]

e.g. the Admin user

[hsheth2]

looks like this was added to dev deps, not framework_common