Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: update ml system UI #12334

Merged
merged 57 commits into from
Jan 29, 2025
Merged
Changes from 1 commit
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
7008a23
[wip] update ui changes
yoonhyejin Jan 14, 2025
6148597
fix typo
yoonhyejin Jan 14, 2025
bbcc2d4
add mlmodellineageinfo
yoonhyejin Jan 14, 2025
963b060
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 14, 2025
8dd3c25
update mapper for lineageinfo and add uni tests
yoonhyejin Jan 15, 2025
2046691
[wip] add script for creating mlmodel
yoonhyejin Jan 15, 2025
a5f9a3f
spotlessapply
yoonhyejin Jan 15, 2025
054bc1f
feat(graphql/versioning): Add versioning support to graphql; mutation…
asikowitz Jan 15, 2025
06bc8c1
add back fragments.graphql
yoonhyejin Jan 15, 2025
5a6923c
fix tests
RyanHolstien Jan 16, 2025
69da53a
Merge remote-tracking branch 'andrew-fork/versioning-graphql' into fe…
yoonhyejin Jan 16, 2025
314b869
fix fragments.graphql
yoonhyejin Jan 16, 2025
43e3d07
[wip] fix ui bugs
yoonhyejin Jan 16, 2025
e365b06
fix rendering issues
yoonhyejin Jan 17, 2025
88db564
add run event in script
yoonhyejin Jan 17, 2025
b62ef19
update script and add islinegae:True back
yoonhyejin Jan 17, 2025
b5066c6
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 17, 2025
d4dc426
add mlModel to DPIinput relationships entityTypes
yoonhyejin Jan 20, 2025
31dd659
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 20, 2025
3c1c3b7
fix container dpi preview
yoonhyejin Jan 21, 2025
cdc23a8
spotlessapply
yoonhyejin Jan 21, 2025
8a92827
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 21, 2025
0209b7e
fix prefect test
yoonhyejin Jan 21, 2025
2adba1d
fix linting
yoonhyejin Jan 21, 2025
fe1e9cd
fix linting
yoonhyejin Jan 21, 2025
757b914
fix yarn lint
yoonhyejin Jan 21, 2025
c4999be
fix ruff check
yoonhyejin Jan 21, 2025
7a3bdf5
[wip] fix linting
yoonhyejin Jan 21, 2025
f7336bc
fix linting
yoonhyejin Jan 21, 2025
a4c600c
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 22, 2025
da7aee9
update golden files for dbt & fivetran
yoonhyejin Jan 22, 2025
534e3b4
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 22, 2025
01f17c3
update golden files for airflow
yoonhyejin Jan 22, 2025
24a9070
update golden files for dags
yoonhyejin Jan 22, 2025
28359e3
fix airflow tests to include dataplatforminstance
yoonhyejin Jan 22, 2025
43fe5e2
fix linitng
yoonhyejin Jan 22, 2025
0501a2b
hacky fix for search filters test
shirshanka Jan 22, 2025
398450c
fix searchfilters cypress test
shirshanka Jan 22, 2025
8cdc27d
some golden file updates
shirshanka Jan 23, 2025
c52a462
Add argparser for token (#6)
ryota-cloud Jan 23, 2025
b198047
update goldens
shirshanka Jan 23, 2025
e48a7a6
fix preview entity
yoonhyejin Jan 23, 2025
426d80c
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 23, 2025
0d99464
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 24, 2025
2f4aaea
remove commented out code
yoonhyejin Jan 24, 2025
9f7f298
fix empty ml model message
yoonhyejin Jan 24, 2025
946854c
remove console logs
yoonhyejin Jan 24, 2025
fa713df
pr feedback pair programming
asikowitz Jan 28, 2025
a1d48c2
revert metadata-ingestion/ & metadata-ingestion-modules/ files to head
yoonhyejin Jan 28, 2025
8a42b7e
revert some of the changes manually
yoonhyejin Jan 28, 2025
a33ef73
revert some of the changes manually
yoonhyejin Jan 28, 2025
d4af718
update stylings
yoonhyejin Jan 28, 2025
07adba4
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 28, 2025
e43ed52
Merge branch 'master' into feat/update-mlflow-ui
yoonhyejin Jan 28, 2025
a1c0125
revert search filters
asikowitz Jan 28, 2025
dfd66bf
fix properties.name mapping
yoonhyejin Jan 28, 2025
2b9a418
fix yarnlint
yoonhyejin Jan 29, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update goldens
shirshanka committed Jan 23, 2025
commit b1980477fdbcaa04f6409d45152dfe6fde5cb38c
Original file line number Diff line number Diff line change
@@ -368,7 +368,7 @@
"name": "simple_dag_task_1_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1733528983395,
"time": 1737616797588,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -386,6 +386,18 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:fdbbbcd638bc0e91bbf8d7775efbecaf",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:fdbbbcd638bc0e91bbf8d7775efbecaf",
@@ -459,7 +471,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733528983395,
"timestampMillis": 1737616797588,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -476,7 +488,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733528984355,
"timestampMillis": 1737616797965,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -770,7 +782,7 @@
"name": "simple_dag_run_another_data_task_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1733528992448,
"time": 1737616800543,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -788,14 +800,26 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:888f71b79d9a0b162fe44acad7b2c2ae",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:888f71b79d9a0b162fe44acad7b2c2ae",
"changeType": "UPSERT",
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733528992448,
"timestampMillis": 1737616800543,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -812,7 +836,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733528993380,
"timestampMillis": 1737616800696,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
Original file line number Diff line number Diff line change
@@ -125,7 +125,7 @@
"downstream_task_ids": "[]",
"inlets": "[]",
"outlets": "[]",
"openlineage_job_facet_sql": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.22.0/integration/airflow\", \"_schemaURL\": \"https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet\", \"query\": \"\\n CREATE OR REPLACE TABLE processed_costs AS\\n SELECT\\n id,\\n month,\\n total_cost,\\n area,\\n total_cost / area as cost_per_area\\n FROM costs\\n \"}"
"openlineage_job_facet_sql": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet\", \"query\": \"\\n CREATE OR REPLACE TABLE processed_costs AS\\n SELECT\\n id,\\n month,\\n total_cost,\\n area,\\n total_cost / area as cost_per_area\\n FROM costs\\n \"}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=athena_operator&_flt_3_task_id=transform_cost_table",
"name": "transform_cost_table",
@@ -311,7 +311,7 @@
"name": "athena_operator_transform_cost_table_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1733121901482,
"time": 1737617084849,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -329,6 +329,18 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:9cd4fbcec3a50a4988ffc5841beaf0ad",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:9cd4fbcec3a50a4988ffc5841beaf0ad",
@@ -388,7 +400,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733121901482,
"timestampMillis": 1737617084849,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -405,14 +417,14 @@
"aspectName": "operation",
"aspect": {
"json": {
"timestampMillis": 1733121901625,
"timestampMillis": 1737617084987,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
},
"actor": "urn:li:corpuser:airflow",
"operationType": "CREATE",
"lastUpdatedTimestamp": 1733121901625
"lastUpdatedTimestamp": 1737617084987
}
}
},
@@ -435,7 +447,7 @@
"downstream_task_ids": "[]",
"inlets": "[]",
"outlets": "[]",
"openlineage_job_facet_sql": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.22.0/integration/airflow\", \"_schemaURL\": \"https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet\", \"query\": \"\\n CREATE OR REPLACE TABLE processed_costs AS\\n SELECT\\n id,\\n month,\\n total_cost,\\n area,\\n total_cost / area as cost_per_area\\n FROM costs\\n \"}"
"openlineage_job_facet_sql": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://raw.githubusercontent.com/OpenLineage/OpenLineage/main/spec/OpenLineage.json#/definitions/SqlJobFacet\", \"query\": \"\\n CREATE OR REPLACE TABLE processed_costs AS\\n SELECT\\n id,\\n month,\\n total_cost,\\n area,\\n total_cost / area as cost_per_area\\n FROM costs\\n \"}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=athena_operator&_flt_3_task_id=transform_cost_table",
"name": "transform_cost_table",
@@ -656,7 +668,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1733121901675,
"timestampMillis": 1737617085044,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
Original file line number Diff line number Diff line change
@@ -24,6 +24,17 @@
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,custom_operator_dag,prod)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,custom_operator_dag,prod)",
@@ -130,7 +141,7 @@
"downstream_task_ids": "[]",
"inlets": "[]",
"outlets": "[]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.18.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"CustomOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"execution_timeout\": \"<<non-serializable: timedelta>>\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"unusual_prefix_eab30e0aa660e38948d8dc30c6e600c764319a2e_custom_operator_dag.CustomOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"custom_task_id\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"CustomOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"execution_timeout\": \"<<non-serializable: timedelta>>\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"unusual_prefix_1e2714c5716f252b1bd6e3624b5d06c66bcef953_custom_operator_dag.CustomOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"custom_task_id\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=custom_operator_dag&_flt_3_task_id=custom_task_id",
"name": "custom_task_id",
@@ -141,6 +152,17 @@
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,custom_operator_dag,prod),custom_task_id)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,custom_operator_dag,prod),custom_task_id)",
@@ -222,7 +244,7 @@
"name": "custom_operator_dag_custom_task_id_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1723716446564,
"time": 1737617021993,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -240,14 +262,26 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:07a4aaeffa3875a24cccd1fec6fc7c8c",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:07a4aaeffa3875a24cccd1fec6fc7c8c",
"changeType": "UPSERT",
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1723716446564,
"timestampMillis": 1737617021993,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -276,7 +310,7 @@
"downstream_task_ids": "[]",
"inlets": "[Dataset(platform='snowflake', name='mydb.schema.tableA', env='PROD', platform_instance=None), Dataset(platform='snowflake', name='mydb.schema.tableB', env='PROD', platform_instance=None)]",
"outlets": "[Dataset(platform='snowflake', name='mydb.schema.tableD', env='PROD', platform_instance=None)]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.18.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"CustomOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"execution_timeout\": \"<<non-serializable: timedelta>>\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"unusual_prefix_eab30e0aa660e38948d8dc30c6e600c764319a2e_custom_operator_dag.CustomOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"custom_task_id\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"CustomOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"execution_timeout\": \"<<non-serializable: timedelta>>\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"unusual_prefix_1e2714c5716f252b1bd6e3624b5d06c66bcef953_custom_operator_dag.CustomOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"custom_task_id\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=custom_operator_dag&_flt_3_task_id=custom_task_id",
"name": "custom_task_id",
@@ -287,6 +321,17 @@
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,custom_operator_dag,prod),custom_task_id)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,custom_operator_dag,prod),custom_task_id)",
@@ -391,7 +436,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1723716446701,
"timestampMillis": 1737617022175,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
Original file line number Diff line number Diff line change
@@ -24,6 +24,17 @@
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
@@ -71,23 +82,19 @@
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"changeType": "UPSERT",
"aspectName": "browsePathsV2",
"aspectName": "status",
"aspect": {
"json": {
"path": [
{
"id": "simple_dag"
}
]
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
@@ -96,6 +103,21 @@
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "browsePathsV2",
"aspect": {
"json": {
"path": [
{
"id": "simple_dag"
}
]
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
@@ -115,7 +137,7 @@
"downstream_task_ids": "['run_another_data_task']",
"inlets": "[Dataset(platform='snowflake', name='mydb.schema.tableA', env='PROD', platform_instance=None), Urn(_urn='urn:li:dataset:(urn:li:dataPlatform:snowflake,mydb.schema.tableC,PROD)'), Urn(_urn='urn:li:dataJob:(urn:li:dataFlow:(airflow,test_dag,PROD),test_task)')]",
"outlets": "[Dataset(platform='snowflake', name='mydb.schema.tableD', env='PROD', platform_instance=None)]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.18.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"['run_another_data_task']\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"task_1\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"['run_another_data_task']\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"task_1\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=simple_dag&_flt_3_task_id=task_1",
"name": "task_1",
@@ -128,7 +150,7 @@
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
@@ -260,7 +282,7 @@
"name": "simple_dag_task_1_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1717179933913,
"time": 1737616884628,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -278,6 +300,18 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:fdbbbcd638bc0e91bbf8d7775efbecaf",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:fdbbbcd638bc0e91bbf8d7775efbecaf",
@@ -351,7 +385,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1717179933913,
"timestampMillis": 1737616884628,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -368,14 +402,55 @@
"aspectName": "operation",
"aspect": {
"json": {
"timestampMillis": 1717179934145,
"timestampMillis": 1737616884737,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
},
"actor": "urn:li:corpuser:airflow",
"operationType": "CREATE",
"lastUpdatedTimestamp": 1717179934145
"lastUpdatedTimestamp": 1737616884737
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"changeType": "UPSERT",
"aspectName": "dataJobInfo",
"aspect": {
"json": {
"customProperties": {
"depends_on_past": "False",
"email": "None",
"label": "'task_1'",
"execution_timeout": "None",
"sla": "None",
"task_id": "'task_1'",
"trigger_rule": "<TriggerRule.ALL_SUCCESS: 'all_success'>",
"wait_for_downstream": "False",
"downstream_task_ids": "['run_another_data_task']",
"inlets": "[Dataset(platform='snowflake', name='mydb.schema.tableA', env='PROD', platform_instance=None), Urn(_urn='urn:li:dataset:(urn:li:dataPlatform:snowflake,mydb.schema.tableC,PROD)'), Urn(_urn='urn:li:dataJob:(urn:li:dataFlow:(airflow,test_dag,PROD),test_task)')]",
"outlets": "[Dataset(platform='snowflake', name='mydb.schema.tableD', env='PROD', platform_instance=None)]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"['run_another_data_task']\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"task_1\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"[]\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=simple_dag&_flt_3_task_id=task_1",
"name": "task_1",
"type": {
"string": "COMMAND"
},
"env": "PROD"
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
@@ -481,7 +556,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1717179934378,
"timestampMillis": 1737616884839,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -494,6 +569,42 @@
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "dataFlowInfo",
"aspect": {
"json": {
"customProperties": {
"_access_control": "None",
"catchup": "False",
"description": "'A simple DAG that runs a few fake data tasks.'",
"doc_md": "None",
"fileloc": "<fileloc>",
"is_paused_upon_creation": "None",
"start_date": "DateTime(2023, 1, 1, 0, 0, 0, tzinfo=Timezone('UTC'))",
"tags": "[]",
"timezone": "Timezone('UTC')"
},
"externalUrl": "http://airflow.example.com/tree?dag_id=simple_dag",
"name": "simple_dag",
"description": "A simple DAG that runs a few fake data tasks.",
"env": "PROD"
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
@@ -529,6 +640,54 @@
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),task_1)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataFlow",
"entityUrn": "urn:li:dataFlow:(airflow,simple_dag,prod)",
"changeType": "UPSERT",
"aspectName": "browsePathsV2",
"aspect": {
"json": {
"path": [
{
"id": "simple_dag"
}
]
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
@@ -548,7 +707,7 @@
"downstream_task_ids": "[]",
"inlets": "[]",
"outlets": "[]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.18.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"run_another_data_task\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"['task_1']\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"run_another_data_task\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"['task_1']\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=simple_dag&_flt_3_task_id=run_another_data_task",
"name": "run_another_data_task",
@@ -559,6 +718,17 @@
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
@@ -638,7 +808,7 @@
"name": "simple_dag_run_another_data_task_manual_run_test",
"type": "BATCH_AD_HOC",
"created": {
"time": 1717179938499,
"time": 1737616886933,
"actor": "urn:li:corpuser:datahub"
}
}
@@ -656,14 +826,26 @@
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:888f71b79d9a0b162fe44acad7b2c2ae",
"changeType": "UPSERT",
"aspectName": "dataPlatformInstance",
"aspect": {
"json": {
"platform": "urn:li:dataPlatform:airflow",
"instance": "urn:li:dataPlatformInstance:(urn:li:dataPlatform:airflow,prod)"
}
}
},
{
"entityType": "dataProcessInstance",
"entityUrn": "urn:li:dataProcessInstance:888f71b79d9a0b162fe44acad7b2c2ae",
"changeType": "UPSERT",
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1717179938499,
"timestampMillis": 1737616886933,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"
@@ -673,6 +855,47 @@
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"changeType": "UPSERT",
"aspectName": "dataJobInfo",
"aspect": {
"json": {
"customProperties": {
"depends_on_past": "False",
"email": "None",
"label": "'run_another_data_task'",
"execution_timeout": "None",
"sla": "None",
"task_id": "'run_another_data_task'",
"trigger_rule": "<TriggerRule.ALL_SUCCESS: 'all_success'>",
"wait_for_downstream": "False",
"downstream_task_ids": "[]",
"inlets": "[]",
"outlets": "[]",
"openlineage_run_facet_unknownSourceAttribute": "{\"_producer\": \"https://github.com/OpenLineage/OpenLineage/tree/1.25.0/integration/airflow\", \"_schemaURL\": \"https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/BaseFacet\", \"unknownItems\": [{\"name\": \"BashOperator\", \"properties\": {\"depends_on_past\": false, \"downstream_task_ids\": \"[]\", \"executor_config\": {}, \"ignore_first_depends_on_past\": true, \"mapped\": false, \"operator_class\": \"airflow.operators.bash.BashOperator\", \"owner\": \"airflow\", \"priority_weight\": 1, \"queue\": \"default\", \"retries\": 0, \"retry_exponential_backoff\": false, \"task_id\": \"run_another_data_task\", \"trigger_rule\": \"all_success\", \"upstream_task_ids\": \"['task_1']\", \"wait_for_downstream\": false, \"weight_rule\": \"downstream\"}, \"type\": \"operator\"}]}"
},
"externalUrl": "http://airflow.example.com/taskinstance/list/?flt1_dag_id_equals=simple_dag&_flt_3_task_id=run_another_data_task",
"name": "run_another_data_task",
"type": {
"string": "COMMAND"
},
"env": "PROD"
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
"changeType": "UPSERT",
"aspectName": "status",
"aspect": {
"json": {
"removed": false
}
}
},
{
"entityType": "dataJob",
"entityUrn": "urn:li:dataJob:(urn:li:dataFlow:(airflow,simple_dag,prod),run_another_data_task)",
@@ -731,7 +954,7 @@
"aspectName": "dataProcessInstanceRunEvent",
"aspect": {
"json": {
"timestampMillis": 1717179939057,
"timestampMillis": 1737616887242,
"partitionSpec": {
"partition": "FULL_TABLE_SNAPSHOT",
"type": "FULL_TABLE"

Large diffs are not rendered by default.


Unchanged files with check annotations Beta

entityType === EntityType.MlfeatureTable ||
entityType === EntityType.MlmodelGroup ||
entityType === EntityType.GlossaryTerm ||
entityType === EntityType.DataProcessInstance ||

Check warning on line 69 in datahub-web-react/src/app/entity/EntityPage.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/EntityPage.tsx#L69

Added line #L69 was not covered by tests
entityType === EntityType.GlossaryNode;
return (
(rel) => rel.type === 'InstanceOf' && rel.entity?.type === EntityType.DataJob,
);
if (!parentEntity || !parentEntity.entity) {
return [];
}

Check warning on line 37 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L35-L37

Added lines #L35 - L37 were not covered by tests
// First cast to unknown, then to Entity with proper type
return [parentEntity.entity as unknown as Entity<DataJob>];

Check warning on line 40 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L39-L40

Added lines #L39 - L40 were not covered by tests
};
/**
// name: 'Documentation',
// component: DocumentationTab,
// },
{
name: 'Summary',
component: SummaryTab,
},

Check warning on line 102 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L99-L102

Added lines #L99 - L102 were not covered by tests
{
name: 'Lineage',
component: LineageTab,
renderPreview = (_: PreviewType, data: DataProcessInstance) => {
const genericProperties = this.getGenericEntityProperties(data);
const parentEntities = getParentEntities(data);
type PreviewEntity = {
urn: string;
type: EntityType;
};

Check warning on line 157 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L154-L157

Added lines #L154 - L157 were not covered by tests
return (
<Preview
urn={data.urn}
dataProduct={getDataProduct(genericProperties?.dataProduct)}
externalUrl={data.properties?.externalUrl}
parentContainers={data.parentContainers}
parentEntities={parentEntities as unknown as PreviewEntity[]}

Check warning on line 175 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L175

Added line #L175 was not covered by tests
container={data.container || undefined}
// health={data.health}
/>
const data = result.entity as DataProcessInstance;
const genericProperties = this.getGenericEntityProperties(data);
const parentEntities = getParentEntities(data);
const firstState = data?.state && data.state.length > 0 ? data.state[0] : undefined;
type PreviewEntity = {
urn: string;
type: EntityType;
};

Check warning on line 192 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L186-L192

Added lines #L186 - L192 were not covered by tests
return (
<Preview
urn={data.urn}
degree={(result as any).degree}
paths={(result as any).paths}
parentContainers={data.parentContainers}
parentEntities={parentEntities as unknown as PreviewEntity[]}

Check warning on line 215 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L215

Added line #L215 was not covered by tests
container={data.container || undefined}
duration={firstState?.durationMillis}
status={firstState?.result?.resultType}
startTime={firstState?.timestampMillis}

Check warning on line 219 in datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/DataProcessInstanceEntity.tsx#L217-L219

Added lines #L217 - L219 were not covered by tests
// health={data.health}
/>
);
health,
parentEntities,
parentContainers,
duration,
status,
startTime,
}: {

Check warning on line 45 in datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx#L42-L45

Added lines #L42 - L45 were not covered by tests
urn: string;
name: string;
subType?: string | null;
health?: Health[] | null;
parentEntities?: Array<GeneratedEntity> | null;
parentContainers?: ParentContainersResult | null;
duration?: number | null;
status?: string | null;
startTime?: number | null;

Check warning on line 69 in datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx#L67-L69

Added lines #L67 - L69 were not covered by tests
}): JSX.Element => {
const entityRegistry = useEntityRegistry();
return (
paths={paths}
health={health || undefined}
parentEntities={parentEntities}
duration={duration}
status={status}
startTime={startTime}

Check warning on line 100 in datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/preview/Preview.tsx#L98-L100

Added lines #L98 - L100 were not covered by tests
/>
);
};
},
];
export default function MLModelSummary() {
const baseEntity = useBaseEntity<GetDataProcessInstanceQuery>();
const dpi = baseEntity?.dataProcessInstance;
console.log('dpi', dpi);
const formatDate = (timestamp?: number) => {
if (!timestamp) return '-';
const milliseconds = timestamp < 10000000000 ? timestamp * 1000 : timestamp;
return new Date(milliseconds).toISOString().slice(0, 19).replace('T', ' ');
};
const formatStatus = (state) => {
if (!state || state.length === 0) return '-';
const result = state[0]?.result?.resultType;
const statusColor = result === 'SUCCESS' ? 'green' : 'red';
return <Pill label={capitalize(result)} colorScheme={statusColor} />;
};
const formatDuration = (state) => {
if (!state || state.length === 0) return '-';
return formatDetailedDuration(state[0]?.durationMillis);
};
return (
<TabContent>
<Space direction="vertical" style={{ width: '100%' }} size="large">
<Typography.Title level={3}>Details</Typography.Title>
<InfoItemContainer justifyContent="left">
<InfoItem title="Created At">
<InfoItemContent>{formatDate(dpi?.properties?.created?.time)}</InfoItemContent>
</InfoItem>
<InfoItem title="Status">
<InfoItemContent>{formatStatus(dpi?.state)}</InfoItemContent>
</InfoItem>
<InfoItem title="Duration">
<InfoItemContent>{formatDuration(dpi?.state)}</InfoItemContent>
</InfoItem>
<InfoItem title="Run ID">
<InfoItemContent>{dpi?.mlTrainingRunProperties?.id}</InfoItemContent>
</InfoItem>
<InfoItem title="Created By">
<InfoItemContent>{dpi?.properties?.created?.actor}</InfoItemContent>
</InfoItem>
</InfoItemContainer>
<InfoItemContainer justifyContent="left">
<InfoItem title="Artifacts Location">
<InfoItemContent>{dpi?.mlTrainingRunProperties?.outputUrls}</InfoItemContent>
</InfoItem>
</InfoItemContainer>
<Typography.Title level={3}>Training Metrics</Typography.Title>
<Table
pagination={false}
columns={propertyTableColumns}
dataSource={dpi?.mlTrainingRunProperties?.trainingMetrics as MlMetric[]}
/>
<Typography.Title level={3}>Hyper Parameters</Typography.Title>
<Table
pagination={false}
columns={propertyTableColumns}
dataSource={dpi?.mlTrainingRunProperties?.hyperParams as MlHyperParam[]}
/>
</Space>
</TabContent>
);
}

Check warning on line 105 in datahub-web-react/src/app/entity/dataProcessInstance/profile/DataProcessInstanceSummary.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/dataProcessInstance/profile/DataProcessInstanceSummary.tsx#L40-L105

Added lines #L40 - L105 were not covered by tests
};
displayName = (data: MlModel) => {
return data.properties?.name || data.name || data.urn;

Check warning on line 154 in datahub-web-react/src/app/entity/mlModel/MLModelEntity.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModel/MLModelEntity.tsx#L154

Added line #L154 was not covered by tests
};
getGenericEntityProperties = (mlModel: MlModel) => {
export default function MLModelSummary() {
const baseEntity = useBaseEntity<GetMlModelQuery>();
const model = baseEntity?.mlModel;
const entityRegistry = useEntityRegistry();
console.log('model', model);

Check warning on line 52 in datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx#L50-L52

Added lines #L50 - L52 were not covered by tests
const propertyTableColumns = [
{
},
];
const formatDate = (timestamp?: number) => {
if (!timestamp) return '-';
const milliseconds = timestamp < 10000000000 ? timestamp * 1000 : timestamp;
return new Date(milliseconds).toISOString().slice(0, 19).replace('T', ' ');
};
const renderTrainingJobs = () => {
const trainingJobs =
model?.trainedBy?.relationships?.map((relationship) => relationship.entity).filter(notEmpty) || [];
if (trainingJobs.length === 0) return '-';
return (
<div>
{trainingJobs.map((job, index) => (
<span key={(job as { urn: string }).urn}>
<JobLink
to={entityRegistry.getEntityUrl(
EntityType.DataProcessInstance,
(job as { urn: string }).urn,
)}
>
{(job as { name?: string })?.name || (job as { urn: string }).urn}
</JobLink>
{index < trainingJobs.length - 1 && ', '}
</span>
))}
</div>
);
};

Check warning on line 96 in datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx#L66-L96

Added lines #L66 - L96 were not covered by tests
return (
<TabContent>
<Space direction="vertical" style={{ width: '100%' }} size="large">
<Typography.Title level={3}>Model Details</Typography.Title>
<InfoItemContainer justifyContent="left">
<InfoItem title="Version">
<InfoItemContent>{model?.versionProperties?.version?.versionTag}</InfoItemContent>
</InfoItem>
<InfoItem title="Registered At">
<InfoItemContent>{formatDate(model?.properties?.created?.time)}</InfoItemContent>
</InfoItem>
<InfoItem title="Last Modified At">
<InfoItemContent>{formatDate(model?.properties?.lastModified?.time)}</InfoItemContent>
</InfoItem>
<InfoItem title="Created By">
<InfoItemContent>{model?.properties?.created?.actor}</InfoItemContent>
</InfoItem>
</InfoItemContainer>
<InfoItemContainer justifyContent="left">
<InfoItem title="Aliases">
<InfoItemContent>
{model?.versionProperties?.aliases?.map((alias) => (
<VersionTagContainer key={alias.versionTag}>{alias.versionTag}</VersionTagContainer>
))}
</InfoItemContent>
</InfoItem>
<InfoItem title="Source Run">
<InfoItemContent>{renderTrainingJobs()}</InfoItemContent>
</InfoItem>
</InfoItemContainer>

Check warning on line 126 in datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModel/profile/MLModelSummary.tsx#L100-L126

Added lines #L100 - L126 were not covered by tests
<Typography.Title level={3}>Training Metrics</Typography.Title>
<Table
pagination={false}
export default function MLGroupModels() {
const baseEntity = useBaseEntity<GetMlModelGroupQuery>();
const entityRegistry = useEntityRegistry();
const modelGroup = baseEntity?.mlModelGroup;
const models =
baseEntity?.mlModelGroup?.incoming?.relationships
?.map((relationship) => relationship.entity)
.filter(notEmpty) || [];
const formatDate = (timestamp?: number) => {
if (!timestamp) return '-';
const milliseconds = timestamp < 10000000000 ? timestamp * 1000 : timestamp;
return new Date(milliseconds).toISOString().slice(0, 19).replace('T', ' ');
};
console.log('modelGroup', modelGroup);
console.log('models', models);
const columns = [
{
title: 'Name',
dataIndex: 'name',
key: 'name',
width: 300,
render: (name, record) => (
<NameContainer>
<NameLink href={entityRegistry.getEntityUrl(EntityType.Mlmodel, record.urn)}>{name}</NameLink>
</NameContainer>
),
},
{
title: 'Version',
key: 'version',
width: 70,
render: (_: any, record: any) => (
<VersionContainer>{record.versionProperties?.version?.versionTag || '-'}</VersionContainer>
),
},
{
title: 'Created At',
key: 'createdAt',
width: 150,
render: (_: any, record: any) => (
<Typography.Text>{formatDate(record.properties?.createdTS?.time)}</Typography.Text>
),
},
{
title: 'Aliases',
key: 'aliases',
width: 200,
render: (_: any, record: any) => {
const aliases = record.versionProperties?.aliases || [];
return (
<TagContainer>
{aliases.map((alias) => (
<AliasPill key={alias.versionTag}>{alias.versionTag}</AliasPill>
))}
</TagContainer>
);
},
},
{
title: 'Tags',
key: 'tags',
width: 200,
render: (_: any, record: any) => {
const tags = record.properties?.tags || [];
return (
<TagContainer>
{tags.map((tag) => (
<TagPill key={tag}>{tag}</TagPill>
))}
</TagContainer>
);
},
},
{
title: 'Description',
dataIndex: 'description',
key: 'description',
width: 300,
render: (_: any, record: any) => {
const editableDesc = record.editableProperties?.description;
const originalDesc = record.description;
return <Typography.Text>{editableDesc || originalDesc || '-'}</Typography.Text>;
},
},
];

Check warning on line 178 in datahub-web-react/src/app/entity/mlModelGroup/profile/ModelGroupModels.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModelGroup/profile/ModelGroupModels.tsx#L90-L178

Added lines #L90 - L178 were not covered by tests
return (
<ModelsContainer>
<Typography.Title level={3}>Model Group Details</Typography.Title>
<InfoItemContainer justifyContent="left">
<InfoItem title="Created At">
<InfoItemContent>
{modelGroup?.properties?.created?.time ? formatDate(modelGroup.properties?.created?.time) : '-'}
</InfoItemContent>
</InfoItem>
<InfoItem title="Last Modified At">
<InfoItemContent>
{modelGroup?.properties?.lastModified?.time
? formatDate(modelGroup.properties?.lastModified?.time)
: '-'}
</InfoItemContent>
</InfoItem>
{modelGroup?.properties?.created?.actor && (
<InfoItem title="Created By">
<InfoItemContent>{modelGroup.properties.created?.actor}</InfoItemContent>
</InfoItem>
)}
</InfoItemContainer>
<Typography.Title level={3}>Models</Typography.Title>
<StyledTable
columns={columns}
dataSource={models}
pagination={false}
rowKey="urn"
expandable={{
defaultExpandAllRows: true,
expandRowByClick: true,
}}
locale={{
emptyText: <EmptyTab tab="mlModel" />,
}}
/>
</ModelsContainer>

Check warning on line 216 in datahub-web-react/src/app/entity/mlModelGroup/profile/ModelGroupModels.tsx

Codecov / codecov/patch

datahub-web-react/src/app/entity/mlModelGroup/profile/ModelGroupModels.tsx#L181-L216

Added lines #L181 - L216 were not covered by tests
);
}
{shouldShowRightColumn && (
<RightColumn key="right-column">
{startTime && (
<Popover
content={<div style={popoverStyles.contentStyle}>{toLocalDateTimeString(startTime)}</div>}
title={<div style={popoverStyles.titleStyle}>Start Time</div>}
trigger="hover"
overlayInnerStyle={popoverStyles.overlayInnerStyle}
overlayStyle={popoverStyles.overlayStyle}
>
<StatContainer>{toRelativeTimeString(startTime)}</StatContainer>
</Popover>

Check warning on line 436 in datahub-web-react/src/app/preview/DefaultPreviewCard.tsx

Codecov / codecov/patch

datahub-web-react/src/app/preview/DefaultPreviewCard.tsx#L428-L436

Added lines #L428 - L436 were not covered by tests
)}
{duration && (
<Popover
content={<div style={popoverStyles.contentStyle}>{formatDetailedDuration(duration)}</div>}
title={<div style={popoverStyles.titleStyle}>Duration</div>}
trigger="hover"
overlayInnerStyle={popoverStyles.overlayInnerStyle}
overlayStyle={popoverStyles.overlayStyle}
>
<StatContainer>{formatDuration(duration)}</StatContainer>
</Popover>

Check warning on line 447 in datahub-web-react/src/app/preview/DefaultPreviewCard.tsx

Codecov / codecov/patch

datahub-web-react/src/app/preview/DefaultPreviewCard.tsx#L439-L447

Added lines #L439 - L447 were not covered by tests
)}
{status && (
<>
<StatContainer>
<Pill label={capitalize(status)} colorScheme={statusPillColor} />
</StatContainer>
</>

Check warning on line 454 in datahub-web-react/src/app/preview/DefaultPreviewCard.tsx

Codecov / codecov/patch

datahub-web-react/src/app/preview/DefaultPreviewCard.tsx#L450-L454

Added lines #L450 - L454 were not covered by tests
)}
{topUsers && topUsers?.length > 0 && (
<>
}
export function formatDuration(durationMs: number): string {
const seconds = Math.floor((durationMs / 1000) % 60);
const minutes = Math.floor((durationMs / 1000 / 60) % 60);
const hours = Math.floor(durationMs / 1000 / 3600);
if (hours === 0 && minutes === 0) {
return `${seconds}secs`;
}
if (hours === 0) {
return minutes === 1 ? `${minutes} min` : `${minutes} mins`;
}
const minuteStr = minutes > 0 ? ` ${minutes} mins` : '';
return hours === 1 ? `${hours} hr${minuteStr}` : `${hours} hrs${minuteStr}`;
}

Check warning on line 225 in datahub-web-react/src/app/shared/time/timeUtils.tsx

Codecov / codecov/patch

datahub-web-react/src/app/shared/time/timeUtils.tsx#L211-L225

Added lines #L211 - L225 were not covered by tests
export function formatDetailedDuration(durationMs: number): string {
const seconds = Math.floor((durationMs / 1000) % 60);
const minutes = Math.floor((durationMs / 1000 / 60) % 60);
const hours = Math.floor(durationMs / 1000 / 3600);
const parts: string[] = [];
if (hours > 0) {
parts.push(hours === 1 ? `${hours} hr` : `${hours} hrs`);
}
if (minutes > 0) {
parts.push(minutes === 1 ? `${minutes} min` : `${minutes} mins`);
}
if (seconds > 0) {
parts.push(`${seconds} secs`);
}
return parts.join(' ');
}

Check warning on line 244 in datahub-web-react/src/app/shared/time/timeUtils.tsx

Codecov / codecov/patch

datahub-web-react/src/app/shared/time/timeUtils.tsx#L228-L244

Added lines #L228 - L244 were not covered by tests
yield mcp
if self.subtype:
mcp = MetadataChangeProposalWrapper(

Check warning on line 323 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L323

Added line #L323 was not covered by tests
entityUrn=str(self.urn), aspect=SubTypesClass(typeNames=[self.subtype])
)
yield mcp

Check warning on line 326 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L326

Added line #L326 was not covered by tests
if self.container_urn:
mcp = MetadataChangeProposalWrapper(

Check warning on line 329 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L329

Added line #L329 was not covered by tests
entityUrn=str(self.urn),
aspect=ContainerClass(container=self.container_urn),
)
yield mcp

Check warning on line 333 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L333

Added line #L333 was not covered by tests
yield from self.generate_inlet_outlet_mcp(materialize_iolets=materialize_iolets)
:param clone_outlets: (bool) whether to clone datajob's outlets
:return: DataProcessInstance
"""
dpi: DataProcessInstance = DataProcessInstance(

Check warning on line 411 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L411

Added line #L411 was not covered by tests
id=id,
orchestrator=DataPlatformUrn.from_string(
container_key.platform
container_urn=container_key.as_urn(),
)
return dpi

Check warning on line 420 in metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py

Codecov / codecov/patch

metadata-ingestion/src/datahub/api/entities/dataprocess/dataprocess_instance.py#L420

Added line #L420 was not covered by tests
@staticmethod
def from_dataflow(dataflow: DataFlow, id: str) -> "DataProcessInstance":