doc: make developer guide great again (by making it a book) (#17521)

Signed-off-by: xxchan <[email protected]>
Co-authored-by: Noel Kwan <[email protected]>

.github/workflows/doc.yml

@@ -25,28 +25,28 @@ jobs:
           remove-docker-images: 'true'
           root-reserve-mb: 10240
           temp-reserve-mb: 10240
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
       - name: Setup Rust toolchain
         run: rustup show
-      - name: Install dependencies
+      - name: Install dependencies for compiling RisingWave
         run: sudo apt-get update && sudo apt-get install -y make build-essential cmake protobuf-compiler curl openssl libssl-dev libsasl2-dev libcurl4-openssl-dev pkg-config postgresql-client tmux lld
       - name: Run sccache-cache
-        uses: mozilla-actions/[email protected].3
+        uses: mozilla-actions/[email protected].5
         with:
           version: "v0.5.2"
       - name: build rustdocs
         run: |
           cargo doc --workspace --no-deps --document-private-items
           cp docs/rustdoc/rust.css target/doc/rust.css
 
-          mkdir artifact
-          cp -R target/doc/* artifact
+          mkdir -p artifact/rustdoc
+          cp -R target/doc/* artifact/rustdoc
       - name: Show available storage
         run: df -h
-      - name: Install cargo-docset
+      - name: Install tools for building docs
         uses: taiki-e/install-action@v2
         with:
-          tool: cargo-docset
+          tool: cargo-docset,mdbook,mdbook-toc,mdbook-linkcheck
       - name: build docsets
         run: |
           cargo docset --no-clean --docset-name RisingWave
@@ -58,8 +58,13 @@ jobs:
           </entry>" > RisingWave.xml
 
           cp -t artifact "risingwave.docset.tgz" "RisingWave.xml"
+      - name: build developer doc
+        run: |
+          cd docs/dev
+          mdbook build
+          cp -R book/html/* ../../artifact
       - name: Upload artifacts
-        uses: actions/upload-pages-artifact@v1
+        uses: actions/upload-pages-artifact@v3
         with:
           path: artifact
       - name: Show available storage
@@ -76,4 +81,4 @@ jobs:
     steps:
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v1
+        uses: actions/deploy-pages@v4

.gitignore

@@ -83,3 +83,6 @@ e2e_test/iceberg/spark-*-bin*
 *.slt.temp
 
 .direnv/
+
+# mdbook
+book

CONTRIBUTING.md

@@ -1,72 +1,16 @@
-# Contribution guidelines
+# Contributing to RisingWave
 
-Thanks for your interest in contributing to RisingWave! We welcome and appreciate contributions.
+Contributing documentation has moved to the **[RisingWave Developer Guide](https://risingwavelabs.github.io/risingwave/)**.
 
-This document describes how to submit your code changes. To learn about the development process, see the [developer guide](docs/developer-guide.md). To understand the design and implementation of RisingWave, refer to the design docs listed in [docs/README.md](docs/README.md).
 
-If you have questions, you can search for existing discussions or start a new discussion in the [Discussions forum of RisingWave](https://github.com/risingwavelabs/risingwave/discussions), or ask in the RisingWave Community channel on Slack. Please use the [invitation link](https://risingwave.com/slack) to join the channel.
+## Before hacking on RisingWave
 
-To report bugs, create a [GitHub issue](https://github.com/risingwavelabs/risingwave/issues/new/choose).
+We encourage people to discuss their design before hacking on code. Typically,
+you [file an issue] or start a thread on the [Community Slack] before submitting
+a pull request.
 
+Please read [the process] of how to submit your change to RisingWave.
 
-## Table of contents
-
-- [Contribution guidelines](#contribution-guidelines)
-  - [Table of contents](#table-of-contents)
-  - [Tests and miscellaneous checks](#tests-and-miscellaneous-checks)
-  - [Submit a PR](#submit-a-pr)
-    - [Pull Request title](#pull-request-title)
-    - [Pull Request description](#pull-request-description)
-    - [Sign the CLA](#sign-the-cla)
-    - [Cherry pick the commit to release candidate branch](#cherry-pick-the-commit-to-release-candidate-branch)
-
-## Tests and miscellaneous checks
-
-Before submitting your code changes, ensure you fully test them and perform necessary checks. The testing instructions and necessary checks are detailed in the [developer guide](docs/developer-guide.md#test-your-code-changes).
-
-## Submit a PR
-
-### Pull Request title
-
-As described in [here](https://github.com/commitizen/conventional-commit-types/blob/master/index.json), a valid PR title should begin with one of the following prefixes:
-
-- `feat`: A new feature
-- `fix`: A bug fix
-- `doc`: Documentation only changes
-- `refactor`: A code change that neither fixes a bug nor adds a feature
-- `style`: A refactoring that improves code style
-- `perf`: A code change that improves performance
-- `test`: Adding missing tests or correcting existing tests
-- `build`: Changes that affect the build system or external dependencies (example scopes: `.config`, `.cargo`, `Cargo.toml`)
-- `ci`: Changes to RisingWave CI configuration files and scripts (example scopes: `.github`, `ci` (Buildkite))
-- `chore`: Other changes that don't modify src or test files
-- `revert`: Reverts a previous commit
-
-For example, a PR title could be:
-
-- `refactor: modify executor protobuf package path`
-- `feat(execution): enable comparison between nullable data arrays`, where `(execution)` means that this PR mainly focuses on the execution component.
-
-You may also check out previous PRs in the [PR list](https://github.com/risingwavelabs/risingwave/pulls).
-
-### Pull Request description
-
-- If your PR is small (such as a typo fix), you can go brief.
-- If it is large and you have changed a lot, it's better to write more details.
-
-### Sign the CLA
-
-Contributors will need to sign RisingWave Labs' CLA.
-
-### Cherry pick the commit to release candidate branch
-We have a GitHub Action to help cherry-pick commits from `main` branch to a `release candidate` branch, such as `v*.*.*-rc` where `*` is a number.
-
-Checkout details at: https://github.com/risingwavelabs/risingwave/blob/main/.github/workflows/cherry-pick-to-release-branch.yml
-
-To trigger the action, we give a correct label to the PR on `main` branch :
-https://github.com/risingwavelabs/risingwave/blob/main/.github/workflows/cherry-pick-to-release-branch.yml#L10
-
-It will act when the PR on `main` branch merged:
-- If `git cherry-pick` does not find any conflicts, it will open a PR to the `release candidate` branch, and assign the original author as the reviewer.
-
-- If there is a conflict, it will open an issue and make the original author the assignee.
+[Community Slack]: https://risingwave.com/slack
+[file an issue]: https://github.com/risingwavelabs/risingwave/issues/new/choose
+[the process]: https://risingwavelabs.github.io/risingwave/contribution

docs/README.md

@@ -4,7 +4,7 @@ This directory contains RisingWave design documents that are intended to be used
 
 ## Developer guide
 
-After you learn about the basics of RisingWave, take a look at our [developer guide](developer-guide.md). It'll help you up to speed with the development process.
+After you learn about the basics of RisingWave, take a look at our [developer guide](https://risingwavelabs.github.io/risingwave/) to get up to speed with the development process.
 
 ## Table of Contents
 

docs/dev/README.md

@@ -0,0 +1,30 @@
+# RisingWave Developer guide
+
+This is the source of the RisingWave Developer guide, published at
+<https://risingwavelabs.github.io/risingwave/> (via GitHub Actions to GitHub Pages).
+(Note: the url was previously for the crate rustdocs, and now they are moved to <https://risingwavelabs.github.io/risingwave/rustdoc>)
+
+The book is written in Markdown, using the [mdbook](https://rust-lang.github.io/mdBook/) tool to convert to HTML.
+
+Edit `SUMMARY.md` to add new chapters.
+
+## Building the book
+
+```sh
+# install tools
+> cargo binstall mdbook mdbook-linkcheck mdbook-toc
+# start a web server on localhost that you can visit to view the book,
+# and it will automatically reload each time you edit a page.
+> mdbook serve --open
+```
+
+## Table of Contents
+
+We use `mdbook-toc` to auto-generate TOCs for long sections. You can invoke the preprocessor by
+including the `<!-- toc -->` marker at the place where you want the TOC.
+
+
+## Link Validations
+
+We use `mdbook-linkcheck` to validate URLs included in our documentation.
+`linkcheck` will be run automatically when you build with the instructions in the section above.

docs/dev/book.toml

@@ -0,0 +1,15 @@
+[book]
+authors = ["xxchan"]
+title = "RisingWave Developer Guide"
+
+[preprocessor.toc]
+command = "mdbook-toc"
+renderer = ["html"]
+
+[output.html]
+smart-punctuation = true
+git-repository-url = "https://github.com/risingwavelabs/risingwave/tree/main/docs/dev/src"
+edit-url-template = "https://github.com/risingwavelabs/risingwave/edit/main/docs/dev/{path}"
+search.use-boolean-and = true
+
+[output.linkcheck]

docs/dev/src/SUMMARY.md

@@ -0,0 +1,41 @@
+# Summary
+
+[Introduction](./intro.md)
+[Contribution Guidelines](./contributing.md)
+
+---
+
+# Building and debugging RisingWave
+
+- [Building and Running](./build-and-run/intro.md)
+- [Testing](./tests/intro.md)
+- [Debugging](./debugging.md)
+- [Observability](./observability.md)
+
+---
+
+# Benchmarking and Profiling
+
+- [CPU Profiling](./benchmark-and-profile/cpu-profiling.md)
+- [Memory (Heap) Profiling](./benchmark-and-profile/memory-profiling.md)
+- [Microbench](./benchmark-and-profile/microbenchmarks.md)
+
+---
+
+# Specialized topics
+
+- [Continuous Integration](./ci.md)
+
+<!--
+
+TODO:
+
+- [RiseDev](./risedev.md)
+- [Error Handling](./error-handling.md)
+- [Develop Connector]()
+    - [Connector e2e tests]()
+    - [integration tests]()
+- [Compile time]()
+    - [Crate organization]()
+    - [Optimize for compile time]()
+- [Adding dependencies]

docs/memory-profiling.md → docs/dev/src/benchmark-and-profile/memory-profiling.md

@@ -203,6 +203,3 @@ jeprof --collapsed target/release/risingwave compute-node.10404.2466.i2466.heap
 ```
 
 </details>
-
-
-

docs/microbenchmarks.md → docs/dev/src/benchmark-and-profile/microbenchmarks.md

@@ -49,4 +49,4 @@ cargo bench json_parser
     ```
 
     You can take a look at [Criterion Docs](https://bheisler.github.io/criterion.rs/book/user_guide/command_line_options.html)
-    for more information.
+    for more information.