✨ introduce docs (#3458)

New docs incoming!

Tech:
- Docusaurus
- TailwindCSS

For details on authoring the docs, please check:
docs/ucx/docs/dev/docs_authoring.mdx

For docs deployment pipeline please check:

.github/workflows/docs-release.yml


This PR doesn't change most of the content, rather just forms it into 4
categories:
- Installation 
- Process 
- Reference (includes commands, codes, etc.) 


Feature wise:
- 100% Docusaurus and MDX
- Auto-link checking, checks both links and anchors!
- Cool landing page
- Clickable images 

To setup the env and run/edit docs locally, please use instructions in
`docs/ucx/docs/dev/docs_authoring.mdx`.

Target URL for docs after deployment is
https://databrickslabs.github.io/ucx/


Some screenshots:

![image](https://github.com/user-attachments/assets/b9b3d208-87c5-4984-8e69-a8e09b50b8b6)

![image](https://github.com/user-attachments/assets/06fb48f3-c8fc-422e-90df-841a4a4f4b4b)

---------

Co-authored-by: Cor <[email protected]>
Co-authored-by: Guenia Izquierdo Delgado <[email protected]>

.github/workflows/docs-release.yml

@@ -0,0 +1,59 @@
+name: Docs Release
+
+on:
+  push:
+    tags:
+      - 'v*'
+  workflow_dispatch: # Enables manual triggering of the workflow
+
+jobs:
+  build:
+    name: Build Docusaurus
+    runs-on: 
+      group: databrickslabs-protected-runner-group
+      labels: linux-ubuntu-latest
+    defaults:
+      run:
+        working-directory: docs/ucx
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          cache: yarn
+          cache-dependency-path: docs/ucx/yarn.lock # need to put the lockfile path explicitly
+
+      - name: Install dependencies
+        run: yarn install --frozen-lockfile
+      - name: Build website
+        run: yarn build
+
+      - name: Upload Build Artifact
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: docs/ucx/build
+
+  deploy:
+    name: Deploy to GitHub Pages
+    needs: build
+    runs-on:
+      group: databrickslabs-protected-runner-group
+      labels: linux-ubuntu-latest
+
+    # Grant GITHUB_TOKEN the permissions required to make a Pages deployment
+    permissions:
+      pages: write # to deploy to Pages
+      id-token: write # to verify the deployment originates from an appropriate source
+
+    # Deploy to the github-pages environment
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+
+
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@v4

Makefile

@@ -32,3 +32,15 @@ solacc:
 	hatch run python tests/integration/source_code/solacc.py
 
 .PHONY: all clean dev lint fmt test integration coverage known solacc
+
+docs-install:
+	yarn --cwd docs/ucx install
+
+docs-serve-dev:
+	yarn --cwd docs/ucx start
+
+docs-build:
+	yarn --cwd docs/ucx build
+
+docs-serve: docs-build
+	yarn --cwd docs/ucx serve

docs/ucx/.gitignore

@@ -0,0 +1,20 @@
+# Dependencies
+/node_modules
+
+# Production
+/build
+
+# Generated files
+.docusaurus
+.cache-loader
+
+# Misc
+.DS_Store
+.env.local
+.env.development.local
+.env.test.local
+.env.production.local
+
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*

CONTRIBUTING.md → docs/ucx/docs/dev/contributing.mdx

@@ -1,4 +1,7 @@
-# Contributing
+
+import Admonition from '@theme/Admonition';
+
+# Developer Guide
 
 ## First Principles
 
@@ -27,7 +30,7 @@ or specialized functionality unavailable in standard libraries.
 
 ## Adding entries to `known.json` file
 
-When adding a new entry to the [`known.json` file](src/databricks/labs/ucx/source_code/known.json), ensure that the entry is unique and follows the correct format. The `known.json` file
+When adding a new entry to the [`known.json` file](https://github.com/databrickslabs/ucx/blob/main/src/databricks/labs/ucx/source_code/known.json), ensure that the entry is unique and follows the correct format. The `known.json` file
 stores information about known (in)compatibilities with Unity Catalog. The file speeds up static code analysis and prevents false positives.
 
 1. load into virtual environment
@@ -214,7 +217,7 @@ with the [Python plugin (Community Edition)](https://plugins.jetbrains.com/plugi
 IntelliJ CE, then it would work in PyCharm. Debugging capabilities are essential for troubleshooting and diagnosing issues during
 development. Please make sure that your test setup allows for easy debugging by following best practices.
 
-![debugging tests](docs/debugging-tests.gif)
+![debugging tests](/img/debugging-tests.gif)
 
 Adhering to these guidelines ensures that our integration tests are robust, efficient, and easily maintainable. This,
 in turn, contributes to the overall reliability and quality of our software.
@@ -263,7 +266,7 @@ HATCH_PYTHON="$(which python3.10)" make clean dev test
 ```
 
 Configure your IDE to use `.venv/bin/python` from the virtual environment when developing the project:
-![IDE Setup](docs/hatch-intellij.gif)
+![IDE Setup](/img/hatch-intellij.gif)
 
 
 Verify installation with
@@ -403,6 +406,11 @@ $ python3.10 -m pip install hatch
 $ make dev
 $ make test
 ```
-Note: Before performing a clean installation, deactivate the virtual environment and follow the commands given above.
 
-Note: The initial `hatch env show` is just to list the environments managed by Hatch and is not needed.
+<Admonition type="info" title="Deactivating the Virtual Environment">
+  Before performing a clean installation, deactivate the virtual environment and follow the commands given above.
+</Admonition>
+
+<Admonition type="note" title="Env command">
+  The initial `hatch env show` is just to list the environments managed by Hatch and is not needed.
+</Admonition>

docs/ucx/docs/dev/docs_authoring.mdx

@@ -0,0 +1,138 @@
+import Admonition from '@theme/Admonition';
+
+# Authoring Documentation
+
+This document provides guidelines for writing documentation for the UCX project.
+
+
+## Tech Stack
+
+The UCX documentation is built using [Docusaurus](https://docusaurus.io/), a modern static site generator. Docusaurus is a project of Facebook Open Source and is used by many open-source projects to build their documentation websites.
+We also use [MDX](https://mdxjs.com/) to write markdown files that include JSX components. This allows us to write markdown files with embedded React components.
+For styling, we use [Tailwind CSS](https://tailwindcss.com/), a utility-first CSS framework for rapidly building custom designs.
+
+## Writing Documentation
+
+Most of the documentation is written in markdown files with the `.mdx` extension. 
+The markdown files are located in the `docs` directory of the UCX project.
+
+## Prerequisites
+
+Before you start writing documentation, make sure you have the following tools installed on your machine:
+- [Node.js](https://nodejs.org/en/)
+- [Yarn](https://yarnpkg.com/)
+
+On macOS, you can install Node.js and Yarn using [Homebrew](https://brew.sh/):
+
+```bash
+brew install node
+npm install --global yarn
+```
+
+
+## Setup 
+
+To set up the documentation locally, follow these steps:
+
+1. Clone the UCX repository
+2. Run:
+
+```bash
+make docs-install
+```
+
+## Running the Documentation Locally
+
+To run the documentation locally, use the following command:
+
+```bash
+make docs-serve-dev
+```
+## Checking search functionality
+
+<Admonition type="tip" title="Tip" emoji="💡">
+We're using local search, and it won't be available in the development server.
+</Admonition>
+
+To check the search functionality, run the following command:
+
+```bash
+make docs-serve
+```
+
+## Adding images
+
+To add images to your documentation, place the image files in the `static/img` directory.
+
+To include an image in your markdown file, use the following syntax:
+
+```markdown
+![Alt text](/img/image-name.png)
+```
+
+<Admonition type="tip" title="Tip" emoji="💡">
+Images support zooming features out of the box.
+</Admonition>
+
+## Linking pages
+
+It is **strongly** recommended to make all links absolute. 
+By doing so we ensure that it's easy to move files without losing links inside them. 
+
+To add an absolute link, use this syntax:
+
+```md
+[link text](/docs/folder/file_name)
+```
+
+Always start with `/docs`. The file extension `.md` or `.mdx` can be omitted. 
+
+To add an anchor to a specific heading, use this syntax:
+
+```md
+[link with anchor](/docs/folder/file_name#anchor)
+```
+
+After writing docs, run this command:
+
+```
+make docs-build
+```
+
+It will throw an error on any unresolved link or anchor.
+
+
+## Content alignment and structure of folders
+
+When writing documentation, make sure to align the content with the existing documentation.
+
+The rule of thumb is:
+
+<div className="text-red-700 p-4 dark:text-red-400 flex justify-center items-center">
+    <div className='max-w-screen-sm text-center text-balance'>
+        <div className="text-2xl font-bold font-mono text-balance">
+            Do not put any technical details in the main documentation.
+        </div>
+        <div className="text-lg font-mono">
+        All technical details should be kept in <code>/docs/dev/</code> section.
+        </div>
+    </div>
+</div>
+
+No need for:
+- Source code links 
+- Deep technical details
+- Implementation details
+
+Or any other details that are not necessary for the **end-user**.
+
+## Deployment
+
+The UCX documentation is hosted on GitHub Pages, and the deployment process is automated using GitHub Actions. 
+
+To configure the deployment process, you need to have admin permissions on the repo:
+1. go to the repository settings -> Pages
+2. In the "Build and deployment" section select the `Github Actions` source. 
+
+The deployment process is triggered automatically when a release is created.
+Also, the deployment process can be triggered manually from the `Actions` tab in the GitHub repository by privileged users.

docs/ucx/docs/dev/index.mdx

@@ -0,0 +1,7 @@
+---
+sidebar_position: 4
+---
+
+# Contributing to UCX
+
+This section is for contributors to the UCX toolkit. It contains information on how to contribute to the toolkit, including how to submit issues, pull requests, and how to contribute to the documentation.