From 5e91fda469d095326203e5c0e26887b668e1753c Mon Sep 17 00:00:00 2001 From: renardeinside Date: Mon, 13 Jan 2025 22:18:37 +0100 Subject: [PATCH] minor fixes --- .../ucx/docs/installation/cross_workspace.mdx | 16 ++++---- .../reference/commands/code_migration.mdx | 10 ++--- .../ucx/docs/reference/commands/metastore.mdx | 6 +-- docs/ucx/docs/reference/commands/table.mdx | 38 +++++++++---------- docs/ucx/docs/reference/common_challenges.mdx | 16 ++++---- docs/ucx/docs/reference/linter_codes.mdx | 38 +++++++++---------- docs/ucx/docs/reference/utilities.mdx | 18 ++++----- 7 files changed, 71 insertions(+), 71 deletions(-) diff --git a/docs/ucx/docs/installation/cross_workspace.mdx b/docs/ucx/docs/installation/cross_workspace.mdx index d8cd35e258..0bfbaf4ab9 100644 --- a/docs/ucx/docs/installation/cross_workspace.mdx +++ b/docs/ucx/docs/installation/cross_workspace.mdx @@ -11,7 +11,7 @@ Ask your Databricks Account admin to run the [`sync-workspace-info` command](#sy workspace information with the UCX installations. Once the workspace information is synced, you can run the [`create-table-mapping` command](#create-table-mapping-command) to align your tables with the Unity Catalog. -[[back to top](#databricks-labs-ucx)] + ## `sync-workspace-info` command @@ -37,7 +37,7 @@ If you cannot get account administrator privileges in reasonable time, you can t run [`manual-workspace-info` command](#manual-workspace-info-command) to enter Databricks Workspace IDs and Databricks Workspace names. -[[back to top](#databricks-labs-ucx)] + ## `manual-workspace-info` command @@ -61,7 +61,7 @@ run. It prompts the user to enter the required information manually and creates useful for workspace administrators who are unable to use the `sync-workspace-info` command, because they are not Databricks Account Administrators. It can also be used to manually create the workspace info in a new workspace. -[[back to top](#databricks-labs-ucx)] + ## `create-account-groups` command @@ -83,7 +83,7 @@ This command is useful for the setups, that don't have SCIM provisioning in plac Once you're done with this command, proceed to the [group migration workflow](#group-migration-workflow). -[[back to top](#databricks-labs-ucx)] + ## `validate-groups-membership` command @@ -103,7 +103,7 @@ used to debug issues related to group membership. See [group migration](docs/loc Valid group membership is important to ensure users has correct access after legacy table ACL is migrated in [table migration process](#Table-Migration) -[[back to top](#databricks-labs-ucx)] + ## `validate-table-locations` command @@ -134,7 +134,7 @@ Considerations when resolving tables with overlapping locations are: - Tags - ACLs -[[back to top](#databricks-labs-ucx)] + ## `cluster-remap` command @@ -159,7 +159,7 @@ to revert the cluster remapping. You can revert the cluster remapping using the [`revert-cluster-remap` command](#revert-cluster-remap-command). -[[back to top](#databricks-labs-ucx)] + ## `revert-cluster-remap` command @@ -176,7 +176,7 @@ its configuration from UC to original one.It will iterate through the list of cl cluster configurations to original one.This will also ask the user to provide the list of clusters that has to be reverted as a prompt. By default, it will revert all the clusters present in the backup folder -[[back to top](#databricks-labs-ucx)] + ## `upload` command diff --git a/docs/ucx/docs/reference/commands/code_migration.mdx b/docs/ucx/docs/reference/commands/code_migration.mdx index 57de6c56a9..33ed92a6a5 100644 --- a/docs/ucx/docs/reference/commands/code_migration.mdx +++ b/docs/ucx/docs/reference/commands/code_migration.mdx @@ -7,7 +7,7 @@ After you're done with the [table migration](#Table-Migration), you can proceed Once you're done with the code migration, you can run the [`cluster-remap` command](#cluster-remap-command) to remap the clusters to be UC compatible. -[[back to top](#databricks-labs-ucx)] + ## `lint-local-code` command @@ -26,7 +26,7 @@ When run from an IDE terminal, this command generates output as follows: ![img.png](/img/lint-local-code-output.png) With modern IDEs, clicking on the file link opens the file at the problematic line -[[back to top](#databricks-labs-ucx)] + ## `migrate-local-code` command @@ -39,7 +39,7 @@ migrate all python and SQL files in the current working directory. This command at the moment only supports Python and SQL files and discards code comments and formatting during the automated transformation process. -[[back to top](#databricks-labs-ucx)] + ## `migrate-dbsql-dashboards` command @@ -58,7 +58,7 @@ This command can be run with `--dashboard-id` flag to migrate a specific dashboa This command is incremental and can be run multiple times to migrate new dashboards. -[[back to top](#databricks-labs-ucx)] + ## `revert-dbsql-dashboards` command @@ -71,4 +71,4 @@ databricks labs ucx revert-dbsql-dashboards [--dashboard-id ] This command can be run with `--dashboard-id` flag to migrate a specific dashboard. -[[back to top](#databricks-labs-ucx)] + diff --git a/docs/ucx/docs/reference/commands/metastore.mdx b/docs/ucx/docs/reference/commands/metastore.mdx index 829fd6c52a..e63422a527 100644 --- a/docs/ucx/docs/reference/commands/metastore.mdx +++ b/docs/ucx/docs/reference/commands/metastore.mdx @@ -3,7 +3,7 @@ These commands are used to assign a Unity Catalog metastore to a workspace. The metastore assignment is a pre-requisite for any further migration steps. -[[back to top](#databricks-labs-ucx)] + ## `show-all-metastores` command @@ -15,7 +15,7 @@ This command lists all the metastores available to be assigned to a workspace. I all the metastores available in the account. This command is useful when there are multiple metastores available within a region, and you want to see which ones are available for assignment. -[[back to top](#databricks-labs-ucx)] + ## `assign-metastore` command @@ -27,7 +27,7 @@ This command assigns a metastore to a workspace with `--workspace-id`. If there workspace region, the command automatically assigns that metastore to the workspace. If there are multiple metastores available, the command prompts for specification of the metastore (id) you want to assign to the workspace. -[[back to top](#databricks-labs-ucx)] + ## `create-ucx-catalog` command diff --git a/docs/ucx/docs/reference/commands/table.mdx b/docs/ucx/docs/reference/commands/table.mdx index 543d4aa0cd..9798b26569 100644 --- a/docs/ucx/docs/reference/commands/table.mdx +++ b/docs/ucx/docs/reference/commands/table.mdx @@ -27,7 +27,7 @@ using the [`skip` command](#skip-command). Once you're done with the table migration, proceed to the [code migration](#code-migration-commands). -[[back to top](#databricks-labs-ucx)] + ## `principal-prefix-access` command @@ -42,7 +42,7 @@ Once you're done running this command, proceed to the [`migrate-credentials` com The "prefix" refers to the start - i.e. prefix - of table locations that point to the cloud storage location. -[[back to top](#databricks-labs-ucx)] + ### Access for AWS S3 Buckets @@ -64,7 +64,7 @@ It has the following format: Once done, proceed to the [`migrate-credentials` command](#migrate-credentials-command). -[[back to top](#databricks-labs-ucx)] + ### Access for Azure Storage Accounts @@ -82,7 +82,7 @@ will be skipped. Once done, proceed to the [`migrate-credentials` command](#migrate-credentials-command). -[[back to top](#databricks-labs-ucx)] + ## `create-missing-principals` command (AWS Only) ```bash @@ -97,7 +97,7 @@ Two optional parameter are available for this command: `--role-name` - This parameter is used to set the prefix for the role name. The default value is `UCX-ROLE`. `--role-policy` - This parameter is used to set the prefix for the role policy name. The default value is `UCX-POLICY`. -[[back to top](#databricks-labs-ucx)] + ## `delete-missing-principals` command (AWS Only) ```bash @@ -107,7 +107,7 @@ This command helps to delete the IAM role created by UCX. It lists all the IAM R command and allows user to select multiple roles to delete. It also checks if selected roles are mapped to any storage credentials and asks for confirmation from user. Once confirmed, it deletes the role and its associated inline policy. -[[back to top](#databricks-labs-ucx)] + ## `create-uber-principal` command @@ -127,7 +127,7 @@ Azure Resource Manager APIs. This command is one of prerequisites for the [table migration process](#table-migration). -[[back to top](#databricks-labs-ucx)] + ## `migrate-credentials` command @@ -157,7 +157,7 @@ Please review the file and delete the Instance Profiles you do not want to be mi Once you're done with this command, run [`validate-external-locations` command](#validate-external-locations-command) after this one. -[[back to top](#databricks-labs-ucx)] + ## `validate-external-locations` command @@ -172,7 +172,7 @@ This command validates and provides mapping to external tables to external locat Once you're done with this command, proceed to the [`migrate-locations` command](#migrate-locations-command). -[[back to top](#databricks-labs-ucx)] + ## `migrate-locations` command @@ -195,7 +195,7 @@ or SQL warehouse Once you're done with this command, proceed to the [`create-table-mapping` command](#create-table-mapping-command). -[[back to top](#databricks-labs-ucx)] + ## `create-table-mapping` command @@ -230,7 +230,7 @@ This command is one of prerequisites for the [table migration process](#Table-Mi Once you're done with table migration, proceed to the [code migration](#code-migration-commands). -[[back to top](#databricks-labs-ucx)] + ## `skip` command @@ -247,7 +247,7 @@ only be used exclusively. This command is useful to temporarily disable migratio Once you're done with table migration, proceed to the [code migration](#code-migration-commands). -[[back to top](#databricks-labs-ucx)] + ## `unskip` command @@ -257,7 +257,7 @@ databricks labs ucx unskip --schema X [--table Y] [--view Z] This command removes the mark set by the [`skip` command](#skip-command) on the given schema, table or view. -[[back to top](#databricks-labs-ucx)] + ## `create-catalogs-schemas` command @@ -274,7 +274,7 @@ the schema and catalog if at least one such table is migrated to it. For AWS, it checks any instance profiles mapped to the interactive cluster or sql warehouse. It checks the mapping of instance profiles to the bucket. It then maps the bucket to the tables which has external location on those bucket created and grants `USAGE` access to the schema and catalog if at least one such table is migrated to it. -[[back to top](#databricks-labs-ucx)] + ## `assign-owner-group` command @@ -288,7 +288,7 @@ The principal running the command and later, the migration workflows, is require The command will list all the groups the principal is a member of and allow the selection of the owner group. It sets the default_owner_group property in the config.yml file. -[[back to top](#databricks-labs-ucx)] + ## `migrate-tables` command @@ -310,7 +310,7 @@ For AWS, it checks any instance profiles mapped to the interactive cluster or sq to the bucket. It then maps the bucket to the tables which has external location on those bucket created and grants either `SELECT` permission if the instance profile only has read access on the bucket and `ALL_PRIVILEGES` if the instance profile has write access on the bucket. -[[back to top](#databricks-labs-ucx)] + ## `revert-migrated-tables` command @@ -326,7 +326,7 @@ to debug issues related to table migration. Go back to the [`create-table-mapping` command](#create-table-mapping-command) after you're done with this command. -[[back to top](#databricks-labs-ucx)] + ## `move` command @@ -348,7 +348,7 @@ This is due to Unity Catalog not supporting multiple tables with overlapping pat This command supports moving multiple tables at once, by specifying `*` as the table name. -[[back to top](#databricks-labs-ucx)] + ## `alias` command @@ -361,4 +361,4 @@ It takes a `WorkspaceClient` object and `from` and `to` parameters as parameters the `TableMove` class. This command is useful for developers and administrators who want to create an alias for a table. It can also be used to debug issues related to table aliasing. -[[back to top](#databricks-labs-ucx)] + diff --git a/docs/ucx/docs/reference/common_challenges.mdx b/docs/ucx/docs/reference/common_challenges.mdx index ab22a8d7b6..3cd9edea40 100644 --- a/docs/ucx/docs/reference/common_challenges.mdx +++ b/docs/ucx/docs/reference/common_challenges.mdx @@ -1,7 +1,7 @@ # Common Challenges and the Solutions Users might encounter some challenges while installing and executing UCX. Please find the listing of some common challenges and the solutions below. -[[back to top](#databricks-labs-ucx)] + ### Network Connectivity Issues @@ -69,7 +69,7 @@ ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verif [to force requests library to set verify=False](https://github.com/psf/requests/blob/8c211a96cdbe9fe320d63d9e1ae15c5c07e179f8/requests/sessions.py#L718) as well as set `SSL_CERT_DIR` env var pointing to the proxy CA cert for the urllib3 library. -[[back to top](#databricks-labs-ucx)] + ### Insufficient Privileges @@ -111,7 +111,7 @@ Admin privileges required for commands: | [sync-workspace-info](#sync-workspace-info-command) | Account Admin | | [manual-workspace-info](#manual-workspace-info-command) | Workspace Admin | -[[back to top](#databricks-labs-ucx)] + ### Version Issues @@ -140,7 +140,7 @@ specific version of UCX, you can do it using the command `databricks labs install ucx@\`, for example, `databricks labs install ucx@v0.21.0`. -[[back to top](#databricks-labs-ucx)] + ### Authentication Issues @@ -165,7 +165,7 @@ authentication](https://docs.databricks.com/en/dev-tools/cli/authentication. [Basic authentication (legacy)](https://docs.databricks.com/en/dev-tools/cli/authentication.html#basic-auth). -[[back to top](#databricks-labs-ucx)] + ### Multiple Profiles in Databricks CLI @@ -189,7 +189,7 @@ profile: `databricks labs install ucx -p prod`. **Solution:** The installation command `databricks labs install ucx` will provide an option to select one account profile. -[[back to top](#databricks-labs-ucx)] + ### Workspace has an external Hive Metastore (HMS) @@ -219,7 +219,7 @@ admin settings do not have the configurations already set. For example, add *spark.hadoop.javax.jdo.option.ConnectionURL \* under Data Access Configuration of SQL Warehouse Admin Settings. -[[back to top](#databricks-labs-ucx)] + ### Verify the Installation @@ -236,4 +236,4 @@ successfully, the installation can be verified with the following steps: external HMS, verify from the results that the assessment has analyzed the external HMS tables. -[[back to top](#databricks-labs-ucx)] + diff --git a/docs/ucx/docs/reference/linter_codes.mdx b/docs/ucx/docs/reference/linter_codes.mdx index 113b699d63..23cde86806 100644 --- a/docs/ucx/docs/reference/linter_codes.mdx +++ b/docs/ucx/docs/reference/linter_codes.mdx @@ -42,7 +42,7 @@ for query in ["SELECT * FROM old.things", f"SELECT * FROM {table_name}"]: spark.sql(query).collect() ``` -[[back to top](#databricks-labs-ucx)] + ## `catalog-api-in-shared-clusters` @@ -50,7 +50,7 @@ for query in ["SELECT * FROM old.things", f"SELECT * FROM {table_name}"]: mode, so of your code has `spark.catalog.tableExists("table")` or `spark.catalog.listDatabases()`, you need to ensure that your cluster is running the correct runtime version and data security mode. -[[back to top](#databricks-labs-ucx)] + ## `changed-result-format-in-uc` @@ -64,7 +64,7 @@ for table in spark.catalog.listTables(): you need to make sure that `do_stuff_with_table` can handle the new format. -[[back to top](#databricks-labs-ucx)] + ## `direct-filesystem-access-in-sql-query` @@ -77,7 +77,7 @@ df = spark.sql("SELECT * FROM parquet.`/mnt/foo/path/to/parquet.file`") you need to change it to use UC tables. -[[back to top](#databricks-labs-ucx)] + ## `direct-filesystem-access` @@ -96,7 +96,7 @@ display(spark.read.csv('s3://bucket/folder/data.csv')) You need to change it to use UC tables or UC volumes. -[[back to top](#databricks-labs-ucx)] + ## `dependency-not-found` @@ -104,7 +104,7 @@ This message indicates that the linter has found a dependency, like Python sourc available in the workspace. The user must ensure that the dependency is available in the workspace. This usually means an error in the user code. -[[back to top](#databricks-labs-ucx)] + ## `jvm-access-in-shared-clusters` @@ -123,7 +123,7 @@ LOGGER = log4jLogger.LogManager.getLogger(__name__) you need to change it to use Python equivalents. -[[back to top](#databricks-labs-ucx)] + ## `legacy-context-in-shared-clusters` @@ -140,13 +140,13 @@ or this: sc.parallelize([1, 2, 3]) ``` -[[back to top](#databricks-labs-ucx)] + ## `not-supported` Installing eggs is no longer supported on Databricks 14.0 or higher. -[[back to top](#databricks-labs-ucx)] + ## `notebook-run-cannot-compute-value` @@ -165,14 +165,14 @@ a = "./leaf1.py" dbutils.notebook.run(a) ``` -[[back to top](#databricks-labs-ucx)] + ## `python-parse-error` This is a generic message indicating that the Python code could not be parsed. The user must manually check the Python code. -[[back to top](#databricks-labs-ucx)] + ## `python-udf-in-shared-clusters` @@ -197,7 +197,7 @@ It is not possible to register Java UDF from Python code on Unity Catalog cluste spark.udf.registerJavaFunction("func", "org.example.func", IntegerType()) ``` -[[back to top](#databricks-labs-ucx)] + ## `rdd-in-shared-clusters` @@ -207,7 +207,7 @@ RDD APIs are not supported on Unity Catalog clusters in Shared access mode. Use df.rdd.mapPartitions(myUdf) ``` -[[back to top](#databricks-labs-ucx)] + ## `spark-logging-in-shared-clusters` @@ -232,27 +232,27 @@ or sc._jvm.org.apache.log4j.LogManager.getLogger(__name__).info("test") ``` -[[back to top](#databricks-labs-ucx)] + ## `sql-parse-error` This is a generic message indicating that the SQL query could not be parsed. The user must manually check the SQL query. -[[back to top](#databricks-labs-ucx)] + ## `sys-path-cannot-compute-value` Path for `sys.path.append` cannot be computed and requires adjusting the path. It is not clear for automated code analysis where the path is located. -[[back to top](#databricks-labs-ucx)] + ## `table-migrated-to-uc` This message indicates that the linter has found a table that has been migrated to Unity Catalog. The user must ensure that the table is available in Unity Catalog. -[[back to top](#databricks-labs-ucx)] + ## `to-json-in-shared-clusters` @@ -263,10 +263,10 @@ above to get a subset of command context information. Example code that triggers dbutils.notebook.entry_point.getDbutils().notebook().getContext().toSafeJson() ``` -[[back to top](#databricks-labs-ucx)] + ## `unsupported-magic-line` This message indicates the code that could not be analysed by UCX. User must check the code manually. -[[back to top](#databricks-labs-ucx)] + diff --git a/docs/ucx/docs/reference/utilities.mdx b/docs/ucx/docs/reference/utilities.mdx index 8aa8cda226..96a4c80ba6 100644 --- a/docs/ucx/docs/reference/utilities.mdx +++ b/docs/ucx/docs/reference/utilities.mdx @@ -12,7 +12,7 @@ check the logs of the last run of a workflow and ensure that it was executed as debugging purposes when a workflow is not behaving as expected. By default, only `INFO`, `WARNING`, and `ERROR` logs are displayed. To display `DEBUG` logs, use the `--debug` flag. -[[back to top](#databricks-labs-ucx)] + ## `ensure-assessment-run` command @@ -25,7 +25,7 @@ This command will block until job finishes. Failed workflows can be fixed with the [`repair-run` command](#repair-run-command). Workflows and their status can be listed with the [`workflows` command](#workflows-command). -[[back to top](#databricks-labs-ucx)] + ## `update-migration-progress` command @@ -41,7 +41,7 @@ it to complete. Workflows and their status can be listed with the [`workflows` command](#workflows-commandr), while failed workflows can be fixed with the [`repair-run` command](#repair-run-command). -[[back to top](#databricks-labs-ucx)] + ## `repair-run` command @@ -54,7 +54,7 @@ want to repair a failed job. It can also be used to debug issues related to job done via [user interface](https://docs.databricks.com/en/workflows/jobs/repair-job-failures.html). Workflows and their status can be listed with the [`workflows` command](#workflows-command). -[[back to top](#databricks-labs-ucx)] + ## `workflows` command @@ -76,7 +76,7 @@ who want to check the status of UCX workflows and ensure that they have been exe for debugging purposes when a workflow is not behaving as expected. Failed workflows can be fixed with the [`repair-run` command](#repair-run-command). -[[back to top](#databricks-labs-ucx)] + ## `open-remote-config` command @@ -112,7 +112,7 @@ access the configuration file from the command line. Here's the description of c * `policy_id`: An optional string representing the ID of the cluster policy. * `include_databases`: An optional list of strings representing the names of databases to include for migration. -[[back to top](#databricks-labs-ucx)] + ## `installations` command @@ -135,7 +135,7 @@ the installations where the `ucx` package is installed and prints their details for administrators who want to see which users have installed `ucx` and where. It can also be used to debug issues related to multiple installations of `ucx` on the same workspace. -[[back to top](#databricks-labs-ucx)] + ## `report-account-compatibility` command @@ -159,7 +159,7 @@ databricks labs ucx report-account-compatibility --profile labs-azure-account 12:56:21 INFO [d.l.u.account.aggregate] Non-DELTA format: UNKNOWN: 5 objects ``` -[[back to top](#databricks-labs-ucx)] + ## `export-assessment` command ```commandline @@ -177,4 +177,4 @@ The export-assessment command is used to export UCX assessment results to a spec - `main` - **Default:** `main` -[[back to top](#databricks-labs-ucx)] +