Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a bonsai guide page to the SLEAP website #1785

Open
wants to merge 13 commits into
base: develop
Choose a base branch
from
Binary file added docs/_static/bonsai-connection.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/bonsai-filecapture.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/bonsai-predictcentroids.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/bonsai-predictposeidentities.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/bonsai-predictposes.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/_static/bonsai-workflow.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
77 changes: 77 additions & 0 deletions docs/guides/bonsai.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
(bonsai)=

# Using Bonsai with SLEAP

Bonsai is a visual language for reactive programming and currently supports SLEAP models.

:::{note}
Currently Bonsai supports only single instance, top-down and top-down-id SLEAP models.
:::

### Exporting a SLEAP trained model
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ensure heading levels increment by one level at a time.

- ### Exporting a SLEAP trained model
+ ## Exporting a SLEAP trained model
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
### Exporting a SLEAP trained model
## Exporting a SLEAP trained model
Tools
Markdownlint

11-11: Expected: h2; Actual: h3 (MD001, heading-increment)
Heading levels should only increment by one level at a time


Before we can import a trained model into Bonsai, we need to use the {code}`sleap-export` command to convert the model to a format supported by Bonsai. For example, to export a top-down-id model, the command is as follows:

```bash
sleap-export -m centroid/model/folder/path -m top_down_id/model/folder/path -e exported/model/path
```

Please refer to the {ref}`sleap-export` docs for more details on using the command.

This will generate the necessary `.pb` file and other information files required by Bonsai. In this example, these files were saved to the specified `exported/model/path` folder.

The `exported/model/path` folder will have a structure like the following:

```
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specify the language for fenced code blocks.

- ```
+ ```plaintext
Tools
Markdownlint

25-25: null (MD040, fenced-code-language)
Fenced code blocks should have a language specified

exported/model/path
├── centroid_config.json
├── confmap_config.json
├── frozen_graph.pb
└── info.json
```

### Installing Bonsai and necessary packages

1. Install Bonsai. See the [Bonsai installation instructions](https://bonsai-rx.org/docs/articles/installation.html).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove trailing spaces.

- 1. Install Bonsai. See the [Bonsai installation instructions](https://bonsai-rx.org/docs/articles/installation.html). 
+ 1. Install Bonsai. See the [Bonsai installation instructions](https://bonsai-rx.org/docs/articles/installation.html).

- Right click on the `FileCapture` module and select **Create Connection**. Now click on the required SLEAP module to complete the connection. 
+ Right click on the `FileCapture` module and select **Create Connection**. Now click on the required SLEAP module to complete the connection.

Also applies to: 67-67

Committable suggestion was skipped due to low confidence.

Tools
Markdownlint

35-35: Expected: 0 or 2; Actual: 1 (MD009, no-trailing-spaces)
Trailing spaces


2. Download and add the necessary packages for Bonsai to run with SLEAP. See the official [Bonsai SLEAP documentation](https://github.com/bonsai-rx/sleap?tab=readme-ov-file#bonsai---sleap) for more information.

### Using Bonsai SLEAP modules

Once you have Bonsai installed with the required packages, you should be able to open the Bonsai application.

The workflow must have a source module `FileCapture` which can be found in the toolbox search in the workflow editor. Provide the path to the video that was used to train the SLEAP model in the `FileName` field of the module.

![Bonsai FileCapture module](../_static/bonsai-filecapture.jpg)

#### Top-down model
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ensure headings are surrounded by blank lines.

- The top-down model requires both the `PredictCentroids` and the `PredictPoses` modules.
+ 
+ The top-down model requires both the `PredictCentroids` and the `PredictPoses` modules.
+ 
- The `PredictPoseIdentities` module will predict the instances with identities.
+ 
+ The `PredictPoseIdentities` module will predict the instances with identities.
+ 
- The `PredictSinglePose` module will predict the poses for single instance models.
+ 
+ The `PredictSinglePose` module will predict the poses for single instance models.
+ 
- Right click on the `FileCapture` module and select **Create Connection**. Now click on the required SLEAP module to complete the connection.
+ 
+ Right click on the `FileCapture` module and select **Create Connection**. Now click on the required SLEAP module to complete the connection.
+ 

Also applies to: 58-58, 63-63, 66-66

Committable suggestion was skipped due to low confidence.

Tools
Markdownlint

47-47: Expected: 1; Actual: 0; Below (MD022, blanks-around-headings)
Headings should be surrounded by blank lines

The top-down model requires both the `PredictCentroids` and the `PredictPoses` modules.

The `PredictCentroids` module will predict the centroids of detections. There are two fields inside the `PredictCentroids` module: the `ModelFileName` field and the `TrainingConfig` field. The `TrainingConfig` field expects the path to the training config JSON file for the centroid model. The `ModelFileName` field expects the path to the `frozen_graph.pb` file in the `exported/model/path` folder.

![Bonsai PredictCentroids module](../_static/bonsai-predictcentroids.jpg)

The `PredictPoses` module will predict the instances of detections. Similar to the `PredictCentroid` module, there are two fields inside the `PredictPoses` module: the `ModelFileName` field and the `TrainingConfig` field. The `TrainingConfig` field expects the path to the training config JSON file for the centered instance model. The `ModelFileName` field expects the path to the `frozen_graph.pb` file in the `exported/model/path` folder.

![Bonsai PredictPoses module](../_static/bonsai-predictposes.jpg)

#### Top-Down-ID model
The `PredictPoseIdentities` module will predict the instances with identities. This module has two fields: the `ModelFileName` field and the `TrainingConfig` field. The `TrainingConfig` field expects the path to the training config JSON file for the top-down-id model. The `ModelFileName` field expects the path to the `frozen_graph.pb` file in the `exported/model/path` folder.

![Bonsai PredictPoseIdentities module](../_static/bonsai-predictposeidentities.jpg)

#### Single instance model
The `PredictSinglePose` module will predict the poses for single instance models. This module also has two fields: the `ModelFileName` field and the `TrainingConfig` field. The `TrainingConfig` field expects the path to the training config JSON file for the single instance model. The `ModelFileName` field expects the path to the `frozen_graph.pb` file in the `exported/model/path` folder.

### Connecting the modules
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct the spelling of "Right-click".

- Right click on the `FileCapture` module and select **Create Connection**.
+ Right-click on the `FileCapture` module and select **Create Connection**.
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
### Connecting the modules
### Connecting the modules
Right-click on the `FileCapture` module and select **Create Connection**.
Tools
LanguageTool

[grammar] ~66-~66: The verb “Right-click” is spelled with a hyphen. (CLICK_HYPHEN)
Context: ...thfolder. ### Connecting the modules Right click on theFileCapture` module and select ...

Markdownlint

66-66: Expected: 1; Actual: 0; Below (MD022, blanks-around-headings)
Headings should be surrounded by blank lines

Right click on the `FileCapture` module and select **Create Connection**. Now click on the required SLEAP module to complete the connection.

![Bonsai module connection ](../_static/bonsai-connection.jpg)

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Avoid multiple consecutive blank lines.

-

Committable suggestion was skipped due to low confidence.

Tools
Markdownlint

48-48: Expected: 1; Actual: 2 (MD012, no-multiple-blanks)
Multiple consecutive blank lines

Once it is done, the workflow in Bonsai will look something like the following:

![Bonsai.SLEAP workflow](../_static/bonsai-workflow.jpg)

Now you can click the green start button to run the workflow and you can add more modules to analyze and visualize the results in Bonsai.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a comma before 'and' in a compound sentence.

- Now you can click the green start button to run the workflow and you can add more modules to analyze and visualize the results in Bonsai.
+ Now you can click the green start button to run the workflow, and you can add more modules to analyze and visualize the results in Bonsai.
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
Now you can click the green start button to run the workflow and you can add more modules to analyze and visualize the results in Bonsai.
Now you can click the green start button to run the workflow, and you can add more modules to analyze and visualize the results in Bonsai.
Tools
LanguageTool

[uncategorized] ~75-~75: Use a comma before ‘and’ if it connects two independent clauses (unless they are closely connected and short). (COMMA_COMPOUND_SENTENCE)
Context: ...e green start button to run the workflow and you can add more modules to analyze and...


For more documentation on various modules and workflows, please refer to the [official Bonsai docs](https://bonsai-rx.org/docs/articles/editor.html).
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ensure the file ends with a single newline character.

+ 
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.

Suggested change
For more documentation on various modules and workflows, please refer to the [official Bonsai docs](https://bonsai-rx.org/docs/articles/editor.html).
For more documentation on various modules and workflows, please refer to the [official Bonsai docs](https://bonsai-rx.org/docs/articles/editor.html).
Tools
Markdownlint

77-77: null (MD047, single-trailing-newline)
Files should end with a single newline character

5 changes: 5 additions & 0 deletions docs/guides/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@

{ref}`remote-inference` when you trained models and you want to run inference on a different machine using a **command-line interface**.

## SLEAP with Bonsai

{ref}`bonsai` when you want to analyze the trained SLEAP model to visualize the poses, centroids and identities for further visual analysis.

```{toctree}
:hidden: true
:maxdepth: 2
Expand All @@ -44,4 +48,5 @@ proofreading
colab
custom-training
remote
bonsai
```