feat(ingestion/powerbi): ingest powerbi app

metadata-ingestion/docs/sources/powerbi/powerbi_pre.md

@@ -10,16 +10,17 @@
 
 ## Concept mapping 
 
-| PowerBI              | Datahub                 |                                                                                               
-|-----------------------|---------------------|
-| `Dashboard`           | `Dashboard`         |
-| `Dataset's Table`     | `Dataset`           |
-| `Tile`                | `Chart`             |
-| `Report.webUrl`       | `Chart.externalUrl` |
-| `Workspace`           | `Container`         |
-| `Report`              | `Dashboard`         |
-| `PaginatedReport`     | `Dashboard`         |
-| `Page`                | `Chart`             |
+| PowerBI           | Datahub             |                                                                                               
+|-------------------|---------------------|
+| `Dashboard`       | `Dashboard`         |
+| `Dataset's Table` | `Dataset`           |
+| `Tile`            | `Chart`             |
+| `Report.webUrl`   | `Chart.externalUrl` |
+| `Workspace`       | `Container`         |
+| `Report`          | `Dashboard`         |
+| `PaginatedReport` | `Dashboard`         |
+| `Page`            | `Chart`             |
+| `App`             | `Dashboard`         |
 
 - If `Tile` is created from report then `Chart.externalUrl` is set to Report.webUrl.
 - The `Page` is unavailable for PowerBI PaginatedReport.
@@ -102,7 +103,7 @@ combine_result
 
 `Pattern-2` is *not* supported for upstream table lineage extraction as it uses nested item-selector i.e. {Source{[Schema="public",Item="book"]}[Data], Source{[Schema="public",Item="issue_history"]}[Data]} as argument to M-QUery table function i.e. Table.Combine
 
-`Pattern-1` is supported as it first assign the table from schema to variable and then variable is used in M-Query Table function i.e. Table.Combine
+`Pattern-1` is supported as it first assigns the table from schema to variable and then variable is used in M-Query Table function i.e. Table.Combine
 
 ## Extract endorsements to tags
 
@@ -112,18 +113,20 @@ Please note that the default implementation overwrites tags for the ingested ent
 
 ## Profiling
 
-The profiling implementation is done through querying [DAX query endpoint](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/execute-queries). Therefore the principal needs to have permission to query the datasets to be profiled. Usually this means that the service principal should have `Contributor` role for the workspace to be ingested. Profiling is done with column based queries to be able to handle wide datasets without timeouts.
+The profiling implementation is done through querying [DAX query endpoint](https://learn.microsoft.com/en-us/rest/api/power-bi/datasets/execute-queries). Therefore, the principal needs to have permission to query the datasets to be profiled. Usually this means that the service principal should have `Contributor` role for the workspace to be ingested. Profiling is done with column-based queries to be able to handle wide datasets without timeouts.
 
-Take into account that the profiling implementation exeutes fairly big amount of DAX queries and for big datasets this is substantial load to the PowerBI system.
+Take into account that the profiling implementation executes a fairly big number of DAX queries, and for big datasets this is a significant load to the PowerBI system.
 
-The `profiling_pattern` setting may be used to limit profiling actions to only a certain set of resources in PowerBI. Both allow and deny rules are matched against following pattern for every table in a PowerBI Dataset: `workspace_name.dataset_name.table_name`. User may limit profiling with these settings at table level, dataset level or workspace level.
+The `profiling_pattern` setting may be used to limit profiling actions to only a certain set of resources in PowerBI. Both allowed and deny rules are matched against the following pattern for every table in a PowerBI Dataset: `workspace_name.dataset_name.table_name`. Users may limit profiling with these settings at table level, dataset level or workspace level.
 
 ## Admin Ingestion vs. Basic Ingestion
 PowerBI provides two sets of API i.e. [Basic API and Admin API](https://learn.microsoft.com/en-us/rest/api/power-bi/). 
 
-The Basic API returns metadata of PowerBI resources where service principal has granted access explicitly on resources whereas Admin API returns metadata of all PowerBI resources irrespective of whether service principal has granted or doesn't granted access explicitly  on resources.
+The Basic API returns metadata of PowerBI resources where service principal has granted access explicitly on resources,
+whereas Admin API returns metadata of all PowerBI resources irrespective of whether service principal has granted
+or doesn't grant access explicitly on resources.
 
-The Admin Ingestion (explain below) is the recommended way to execute PowerBI ingestion as this ingestion can extract most of the metadata.
+The Admin Ingestion (explained below) is the recommended way to execute PowerBI ingestion as this ingestion can extract most of the metadata.
 
 
 ### Admin Ingestion: Service Principal As Admin in Tenant Setting and Added as Member In Workspace
@@ -142,8 +145,9 @@ PowerBI Source would be able to ingest below listed metadata of that particular
   - Endorsement as tag
   - Dashboards 
   - Reports 
-  - Dashboard's Tiles
-  - Report's Pages
+  - Dashboard Tiles
+  - Report Pages
+  - App
 
 If you don't want to add a service principal as a member in your workspace, then you can enable the `admin_apis_only: true` in recipe to use PowerBI Admin API only. 
 
@@ -154,7 +158,7 @@ Caveats of setting `admin_apis_only` to `true`:
 
 
 ### Basic Ingestion: Service Principal As Member In Workspace 
-If you have added service principal as `member` in workspace then PowerBI Source would be able ingest below metadata of that particular workspace 
+If you have added service principal as `member` in workspace then PowerBI Source would be able to ingest below metadata of that particular workspace 
 
   - Dashboards 
   - Reports 

metadata-ingestion/src/datahub/ingestion/source/common/subtypes.py

@@ -70,6 +70,7 @@ class BIAssetSubTypes(StrEnum):
     # PowerBI
     POWERBI_TILE = "PowerBI Tile"
     POWERBI_PAGE = "PowerBI Page"
+    POWERBI_APP = "App"
 
     # Mode
     MODE_REPORT = "Report"

metadata-ingestion/src/datahub/ingestion/source/powerbi/config.py

@@ -473,6 +473,11 @@ class PowerBiDashboardSourceConfig(
         "To maintain backward compatibility, this is set to False.",
     )
 
+    extract_app: bool = pydantic.Field(
+        default=False,
+        description="Whether to ingest workspace app. Default to false",
+    )
+
     @root_validator(skip_on_failure=True)
     def validate_extract_column_level_lineage(cls, values: Dict) -> Dict:
         flags = [

metadata-ingestion/src/datahub/ingestion/source/powerbi/powerbi.py

@@ -4,6 +4,7 @@
 #
 #########################################################
 import logging
+from datetime import datetime
 from typing import Iterable, List, Optional, Tuple, Union
 
 import datahub.emitter.mce_builder as builder
@@ -59,6 +60,7 @@
     FineGrainedLineageUpstreamType,
 )
 from datahub.metadata.schema_classes import (
+    AuditStampClass,
     BrowsePathsClass,
     ChangeTypeClass,
     ChartInfoClass,
@@ -70,6 +72,7 @@
     DatasetLineageTypeClass,
     DatasetProfileClass,
     DatasetPropertiesClass,
+    EdgeClass,
     GlobalTagsClass,
     OtherSchemaClass,
     OwnerClass,
@@ -1006,7 +1009,9 @@ def to_chart_mcps(
             )
 
             # Browse path
-            browse_path = BrowsePathsClass(paths=[f"/powerbi/{workspace.name}"])
+            browse_path = BrowsePathsClass(
+                paths=[f"/{Constant.PLATFORM_NAME}/{workspace.name}"]
+            )
             browse_path_mcp = self.new_mcp(
                 entity_urn=chart_urn,
                 aspect=browse_path,
@@ -1306,6 +1311,102 @@ def extract_independent_datasets(
                 )
             )
 
+    def emit_app(
+        self, workspace: powerbi_data_classes.Workspace
+    ) -> Iterable[MetadataWorkUnit]:
+        if workspace.app is None:
+            return
+
+        if not self.source_config.extract_app:
+            self.reporter.info(
+                title="App Ingestion Is Disabled",
+                message="You are missing workspace app metadata. Please set flag `extract_app` to `true` in recipe to ingest workspace app.",
+                context=f"workspace-name={workspace.name}, app-name = {workspace.app.name}",
+            )
+
+        edges: List[EdgeClass] = [
+            EdgeClass(
+                destinationUrn=builder.make_dashboard_urn(
+                    platform=self.source_config.platform_name,
+                    platform_instance=self.source_config.platform_instance,
+                    name=powerbi_data_classes.Dashboard.get_urn_part_by_id(
+                        app_dashboard.original_dashboard_id
+                    ),
+                )
+            )
+            for app_dashboard in workspace.app.dashboards
+        ]
+
+        edges.extend(
+            [
+                EdgeClass(
+                    destinationUrn=builder.make_dashboard_urn(
+                        platform=self.source_config.platform_name,
+                        platform_instance=self.source_config.platform_instance,
+                        name=powerbi_data_classes.Report.get_urn_part_by_id(
+                            app_report.original_report_id
+                        ),
+                    )
+                )
+                for app_report in workspace.app.reports
+            ]
+        )
+
+        if edges:
+            logger.debug(
+                f"Emitting metadata-workunits for app {workspace.app.name}({workspace.app.id})"
+            )
+            dashboard_info: DashboardInfoClass = DashboardInfoClass(
+                title=workspace.app.name,
+                description=workspace.app.description
+                if workspace.app.description
+                else workspace.app.name,
+                # lastModified=workspace.app.last_update,
+                lastModified=ChangeAuditStamps(
+                    lastModified=AuditStampClass(
+                        actor="urn:li:corpuser:unknown",
+                        time=int(
+                            datetime.strptime(
+                                workspace.app.last_update, "%Y-%m-%dT%H:%M:%S.%fZ"
+                            ).timestamp()
+                        ),
+                    )
+                    if workspace.app.last_update
+                    else None
+                ),
+                dashboards=edges,
+            )
+
+            dashboard_urn: str = builder.make_dashboard_urn(
+                platform=self.source_config.platform_name,
+                platform_instance=self.source_config.platform_instance,
+                name=powerbi_data_classes.App.get_urn_part_by_id(workspace.app.id),
+            )
+
+            # Browse path
+            browse_path: BrowsePathsClass = BrowsePathsClass(
+                paths=[f"/powerbi/{workspace.name}"]
+            )
+
+            yield MetadataChangeProposalWrapper(
+                entityUrn=dashboard_urn,
+                aspect=dashboard_info,
+            ).as_workunit()
+
+            yield MetadataChangeProposalWrapper(
+                entityUrn=dashboard_urn,
+                aspect=browse_path,
+            ).as_workunit()
+
+            yield MetadataChangeProposalWrapper(
+                entityUrn=dashboard_urn, aspect=StatusClass(removed=False)
+            ).as_workunit()
+
+            yield MetadataChangeProposalWrapper(
+                entityUrn=dashboard_urn,
+                aspect=SubTypesClass(typeNames=[BIAssetSubTypes.POWERBI_APP]),
+            ).as_workunit()
+
     def get_workspace_workunit(
         self, workspace: powerbi_data_classes.Workspace
     ) -> Iterable[MetadataWorkUnit]:
@@ -1318,6 +1419,8 @@ def get_workspace_workunit(
                 # Return workunit to a Datahub Ingestion framework
                 yield workunit
 
+        yield from self.emit_app(workspace=workspace)
+
         for dashboard in workspace.dashboards:
             try:
                 # Fetch PowerBi users for dashboards

metadata-ingestion/src/datahub/ingestion/source/powerbi/rest_api_wrapper/data_classes.py

@@ -37,6 +37,35 @@ class DatasetKey(ContainerKey):
     dataset: str
 
 
+@dataclass
+class AppDashboard:
+    id: str
+    original_dashboard_id: str
+
+
+@dataclass
+class AppReport:
+    id: str
+    original_report_id: str
+
+
+@dataclass
+class App:
+    id: str
+    name: str
+    description: Optional[str]
+    last_update: Optional[str]
+    dashboards: List["AppDashboard"]
+    reports: List["AppReport"]
+
+    def get_urn_part(self):
+        return App.get_urn_part_by_id(self.id)
+
+    @staticmethod
+    def get_urn_part_by_id(id_: str) -> str:
+        return f"apps.{id_}"
+
+
 @dataclass
 class Workspace:
     id: str
@@ -49,6 +78,7 @@ class Workspace:
     dashboard_endorsements: Dict[str, List[str]]
     scan_result: dict
     independent_datasets: List["PowerBIDataset"]
+    app: Optional["App"]
 
     def get_urn_part(self, workspace_id_as_urn_part: Optional[bool] = False) -> str:
         # shouldn't use workspace name, as they can be the same?
@@ -233,9 +263,14 @@ class Report:
     pages: List["Page"]
     users: List["User"]
     tags: List[str]
+    app_reference: Optional["App"]
 
     def get_urn_part(self):
-        return f"reports.{self.id}"
+        return Report.get_urn_part_by_id(self.id)
+
+    @staticmethod
+    def get_urn_part_by_id(id_: str) -> str:
+        return f"reports.{id_}"
 
 
 @dataclass
@@ -271,9 +306,14 @@ class Dashboard:
     users: List["User"]
     tags: List[str]
     webUrl: Optional[str]
+    app_reference: Optional["App"]
 
     def get_urn_part(self):
-        return f"dashboards.{self.id}"
+        return Dashboard.get_urn_part_by_id(self.id)
+
+    @staticmethod
+    def get_urn_part_by_id(id_: str) -> str:
+        return f"dashboards.{id_}"
 
     def __members(self):
         return (self.id,)