Split dataset namespace & name (#346)

Signed-off-by: Мартынов Максим Сергеевич <[email protected]>
OpenLineage · Jul 19, 2024 · 9491d7c · 9491d7c
1 parent cb22bd0
commit 9491d7c
Showing 1 changed file with 27 additions and 26 deletions.
diff --git a/docs/spec/naming.md b/docs/spec/naming.md
@@ -12,39 +12,40 @@ Jobs and Datasets have their own namespaces, job namespaces being derived from s
 
 A dataset, or `table`, is organized according to a producer, namespace, database and (optionally) schema.
 
-| Data Store | Type | Namespace | Name | Format | 
-|:-----------|:-----------|:---------------------|:-----------------|:-----------|
-| Athena | Warehouse | Host | Catalog, Database, Table | awsathena://athena.{region_name}.amazonaws.com/{catalog}.{database}.{table} |
-| Azure Cosmos DB | Warehouse | Host, Database | Schema, Table | azurecosmos://{host}/dbs/{database}/colls/{table} |
-| Azure Data Explorer | Warehouse | Host | Database, Table | azurekusto://{host}.kusto.windows.net/{database}/{table} | 
-| Azure Synapse | Warehouse | Host, Port, Database | Schema, Table | sqlserver://{host}:{port};database={database}/{schema}.{table} |
-| BigQuery | Warehouse | bigquery | Project ID, dataset, table | bigquery://{project id}.{dataset name}.{table name} |
-| Cassandra | Warehouse | Host, Port | Keyspace, Table | cassandra://{host}:{port}/{keyspace}.{table} |
-| MySQL | Warehouse | Host, Port | Database, Table | mysql://{host}:{port}/{database}.{table} | 
-| Postgres | Warehouse | Host, Port | Database, Schema, Table | postgres://{host}:{port}/{database}.{schema}.{table} |
-| Redshift | Warehouse | Host, Port | Database, Schema, Table | redshift://{cluster_identifier}.{region_name}:{port}/{database}.{schema}.{table} |
-| Snowflake | Warehouse | account identifier (composite of organization name and account name) | Database, Schema, Table | snowflake://{organization name}-{account name}/{database}.{schema}.{table} | 
-| Trino | Warehouse | Host, Port | Catalog, Schema, Table | trino://{host}:{port}/{catalog}.{schema}.{table} |
-| ABFSS (Azure Data Lake Gen2) | Data lake | container, service | path | abfss://{container name}@{service name}/{path} |
-| DBFS (Databricks File System) | Distributed file system | workspace | path | hdfs://{workspace name}/{path} |
-| GCS | Blob storage | bucket | path | gs://{bucket name}/{path} |
-| HDFS | Distributed file system | Namenode host and port | path | hdfs://{namenode host}:{namenode port}/{path} |
-| Kafka | distributed event streaming platform | bootstrap server host and port | topic | kafka://{bootstrap server host}:{port}/{topic name} |
-| Local file system | File system | Host | Path | file://{host}/{path} |
-| S3 | Blob Storage | bucket name | path | s3://{bucket name}/{path} |
-| WASBS (Azure Blob Storage) | Blob Storage | container, service | path | wasbs://{container name}@{service name}/{path} |
+| Data Store | Type | Namespace | Name |
+|:-----------|:-----------|:---------------------|:-----------------|
+| Athena | Warehouse | awsathena://athena.{region_name}.amazonaws.com | {catalog}.{database}.{table} |
+| Azure Cosmos DB | Warehouse | azurecosmos://{host}/dbs/{database} | colls/{table} |
+| Azure Data Explorer | Warehouse | azurekusto://{host}.kusto.windows.net | {database}/{table} |
+| Azure Synapse | Warehouse | sqlserver://{host}:{port} | {schema}.{table} |
+| BigQuery | Warehouse | bigquery:// | {project id}.{dataset name}.{table name} |
+| Cassandra | Warehouse | cassandra://{host}:{port} | {keyspace}.{table} |
+| MySQL | Warehouse | mysql://{host}:{port} | {database}.{table} |
+| Postgres | Warehouse | postgres://{host}:{port} | {database}.{schema}.{table} |
+| Redshift | Warehouse | redshift://{cluster_identifier}.{region_name}:{port} | {database}.{schema}.{table} |
+| Snowflake | Warehouse | snowflake://{organization name}-{account name} | {database}.{schema}.{table} |
+| Trino | Warehouse | trino://{host}:{port} | {catalog}.{schema}.{table} |
+| ABFSS (Azure Data Lake Gen2) | Data lake | abfss://{container name}@{service name} | {path} |
+| DBFS (Databricks File System) | Distributed file system | hdfs://{workspace name} | {path} |
+| GCS | Blob storage | gs://{bucket name} | {object key} |
+| HDFS | Distributed file system | hdfs://{namenode host}:{namenode port} | {path} |
+| Kafka | distributed event streaming platform | kafka://{bootstrap server host}:{port} | {topic} |
+| Local file system | File system | file://{host} | {path} |
+| S3 | Blob Storage | s3://{bucket name} | {object key} |
+| WASBS (Azure Blob Storage) | Blob Storage | wasbs://{container name}@{service name} | {object key} |
 
 ## Job Naming
 
 A `Job` is a recurring data transformation with inputs and outputs. Each execution is captured as a `Run` with corresponding metadata.
 A `Run` event identifies the `Job` it instances by providing the job’s unique identifier.
-The `Job` identifier is composed of a `Namespace` and `Name`. The job name is unique within its namespace.
+The `Job` identifier is composed of a `Namespace` and `Name`. The job namespace is usually set in OpenLineage client config. The job name is unique within its namespace.
 
 
-| Producer | Formula | Example |
-| -------- | ------- | ------- |
-| Airflow | namespace + DAG + task | airflow-staging.orders_etl.count_orders |
-| SQL | namespace + name | gx.validate_datasets |
+| Job type | Name | Example |
+| :------- | :------ | :------ |
+| Airflow task | {dag_id}.{task_id} | orders_etl.count_orders |
+| Spark job | {appName}.{command}.{table} | my_awesome_app.execute_insert_into_hive_table.mydb_mytable |
+| SQL | {schema}.{table} | gx.validate_datasets |
 
 ## Run Naming