Lineage problem (need to cleanup or reset lineages)

[mchristianto]

First, I ingest Big Query lineages without filtering user_email to specific service account (that run production queries), so i think Openmetadata ingest all the queries executed by all users.

All the lineages is created successfully but it has wrong nodes and edges, also the query lineage show the wrong query, because it shows create table as select (CTAS) query statement from the old table that even didn't exists anymore in Openmetadata.

I edit the lineage ingestion config and added filter user_email so it just scanned query history executed by specific service account, the ingestion is success but when i see the lineage its not changing, still with the wrong sql query lineage.

I think the first lineage cannot be replaced by the new one with filter, is there any way to delete all lineage, reset or cleanup so i can ingest fresh lineage from Big Query?

[Prajwal214]

To reset or clean up the existing lineage in OpenMetadata and ingest fresh lineage from BigQuery, you need to manually delete the outdated lineage edges before re-running the ingestion job. Here are the steps:

1. Delete Existing Lineage:
   a. Navigate to the Lineage Tab: Go to the data asset's details page and click on the "Lineage" tab.
   b. Edit Lineage: Click on the "Edit" icon to enter the lineage editor mode.
   c. Delete Edges: Select the edge (line) you want to delete and click on the delete option to remove it.

2. Modify and Re-run Ingestion: After deleting the outdated edges, re-run the ingestion job with the updated configuration that includes the filter for user_email.

[Prajwal214]

@mchristianto Can you share the OM Version?

[Prajwal214]

You need to click on edit Lineage, And then you will get the cross button to delete the node. You can run the ingestion once the Lineage node is deleted.

[mchristianto]

i'm using version 1.4.5

[mchristianto]

i already click on the edit mode, if its "full edge" i can remove it

but there are an edge with no connection to another nodes, it cannot be remove

there is no edit edge option when i clicked on it, so i cannot remove the edges on both side

[Prajwal214]

Can you try deleting the edge using the API,
https://docs.open-metadata.org/swagger.html#operation/deleteLineageEdge