Added middleware configs and some scripts to ease CI jobs for benchmarking and improve user experience

docs/how_to_run.md

@@ -1,3 +1,84 @@
+## Run All Benchmarks
+If you haven't taken a look at [How To Install](how_to_install.md) section, please firstly start with [How To Install](how_to_install.md) Because it's assumed that you applied [How To Install](how_to_install.md) section in this section.
+
+To run all benchmarks, just select your middleware implementation and go ahead with `run_all_benchmarks.sh` bash script. This command initially will run the script you chose. In this example, it's used the default config scripts which the necessary middleware configuration is applied. This property serves the functionality for the users to apply some custom configurations freely.
+
+```shell
+# go to workspace this repository is built
+cd ws
+source /opt/ros/rolling/setup.sh
+source install/setup.sh
+# go to moveit_middleware_benchmark package's directory
+cd src/moveit_middleware_benchmark
+# conduct all benchmarks
+sh src/moveit_middleware_benchmark/scripts/run_all_benchmarks.sh -i ./src/moveit_middleware_benchmark/middleware_configurations/rmw_cyclonedds/config.sh -d /benchmark_results
+```
+
+Let's explain all operations at `run_all_benchmarks.sh`.
+
+#### selection of initial script
+```
+-i ./src/moveit_middleware_benchmark/middleware_configurations/rmw_cyclonedds/config.sh
+```
+
+This argument is for selecting the initial scripts to be run. In this repository, These initial scripts is used for configuring middleware so that middleware is used more effectively. For example, you can use the initial script to configure TCP settings for rmw_zenoh like below.
+
+```shell
+echo "The configurations for rmw_zenoh_cpp is started!"
+export RMW_IMPLEMENTATION=rmw_zenoh_cpp
+sudo sysctl -w "net.ipv4.tcp_rmem=4096 4096 4096"
+sudo sysctl -w "net.ipv4.tcp_wmem=4096 4096 4096"
+sudo sysctl -w "net.ipv4.tcp_mem=4096 4096 4096"
+echo "The configurations for rmw_zenoh_cpp is finished!"
+```
+
+#### directory selection to save benchmark results
+```
+-d /benchmark_results
+```
+
+This argument indicates where the benchmark results are saved. For scenario_perception_benchmark and scenario_basic_service_client, the results of these scenarios are written in the shape of the below directory tree. It should be added that `run_all_benchmarks.sh` script uses json format to save the benchmark results.
+
+```
+benchmark_results/
+├── scenario_basic_service_client
+│   ├── rmw_cyclonedds_cpp.json
+│   ├── rmw_fastrtps_cpp.json
+│   └── rmw_zenoh_cpp.json
+└── scenario_perception_pipeline
+    ├── rmw_cyclonedds_cpp.json
+    ├── rmw_fastrtps_cpp.json
+    └── rmw_zenoh_cpp.json
+```
+
+## Plot Visualization of Benchmark Results
+
+After running `run_all_benchmarks.sh`, you can also visualize the box plots of benchmark results. Suppose that you have some benchmark results stored in below directory and the directory named `benchmark_results` is located in `ws` directory which this repository is built.
+
+
+```
+benchmark_results/
+├── scenario_basic_service_client
+│   ├── rmw_cyclonedds_cpp.json
+│   ├── rmw_fastrtps_cpp.json
+│   └── rmw_zenoh_cpp.json
+└── scenario_perception_pipeline
+    ├── rmw_cyclonedds_cpp.json
+    ├── rmw_fastrtps_cpp.json
+    └── rmw_zenoh_cpp.json
+```
+
+Just give the directory of benchmark results as argument and then visualize the benchmark results in plot.
+```shell
+cd ws
+python3 src/moveit_middleware_benchmark/scripts/box_plot_visualizer.py benchmark_results
+
+```
+
+**NOTE THAT THE BELOW PICTURE DOESN'T PRESENT REAL RESULTS. IT'S JUST FOR SHOWCASE**
+
+![](./pictures/box_plot_example.png)
+
 ## Scenarios
 
 ### [Perception Pipeline Benchmark](scenarios/perception_pipeline_benchmark.md)

middleware_configurations/rmw_cyclonedds/config.sh

@@ -0,0 +1,3 @@
+echo "The configurations for rmw_cyclonedds_cpp is started!"
+export RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
+echo "The configurations for rmw_cyclonedds_cpp is finished!"

middleware_configurations/rmw_fastrtps/config.sh

@@ -0,0 +1,3 @@
+echo "The configurations for rmw_fastrtps_cpp is started!"
+export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
+echo "The configurations for rmw_fastrtps_cpp is finished!"

middleware_configurations/rmw_zenoh/config.sh

@@ -0,0 +1,6 @@
+echo "The configurations for rmw_zenoh_cpp is started!"
+export RMW_IMPLEMENTATION=rmw_zenoh_cpp
+sudo sysctl -w "net.ipv4.tcp_rmem=4096 4096 4096"
+sudo sysctl -w "net.ipv4.tcp_wmem=4096 4096 4096"
+sudo sysctl -w "net.ipv4.tcp_mem=4096 4096 4096"
+echo "The configurations for rmw_zenoh_cpp is finished!"

scripts/box_plot_visualizer.py

@@ -0,0 +1,77 @@
+import matplotlib.pyplot as plt
+import numpy as np
+import json
+import os
+import sys
+
+BENCHMARK_RESULTS_DIR = sys.argv[1]
+
+middleware_colors = {
+    "rmw_zenoh_cpp": "orange",
+    "rmw_cyclonedds_cpp": "peachpuff",
+    "rmw_fastrtps_cpp": "tomato",
+}
+
+middleware_list = ["rmw_zenoh_cpp", "rmw_cyclonedds_cpp", "rmw_fastrtps_cpp"]
+
+
+def read_benchmark_json(file_name):
+    benchmark_json_data = None
+    with open(file_name) as f:
+        benchmark_json_data = json.load(f)
+    return benchmark_json_data
+
+
+def get_real_time_list_from_benchmark_json(benchmark_json_data):
+    real_time_list = []
+    for benchmark_info in benchmark_json_data["benchmarks"]:
+        if benchmark_info["run_type"] == "iteration":
+            real_time_list.append(benchmark_info["real_time"])
+    return real_time_list
+
+
+def get_middleware_dataset_for_scenario(scenario_name):
+    middleware_datasets = []
+    for middleware_name in middleware_list:
+        file_name = os.path.join(
+            BENCHMARK_RESULTS_DIR, scenario_name, f"{middleware_name}.json"
+        )
+        benchmark_json_data = read_benchmark_json(file_name)
+        dataset = get_real_time_list_from_benchmark_json(benchmark_json_data)
+        middleware_datasets.append(
+            {
+                "name": middleware_name,
+                "dataset": dataset,
+                "color": middleware_colors[middleware_name],
+            }
+        )
+    return middleware_datasets
+
+
+def plot_dataset_of_scenario(plt, scenario_name, middleware_datasets):
+    labels = []
+    colors = []
+    datasets = []
+
+    for x in middleware_datasets:
+        labels.append(x["name"])
+        colors.append(x["color"])
+        datasets.append(x["dataset"])
+
+    fig, ax = plt.subplots()
+    ax.set_title(scenario_name)
+    ax.set_ylabel("real time (ns)")
+    ax.set_xlabel("middlewares")
+
+    bplot = ax.boxplot(datasets, patch_artist=True, tick_labels=labels)
+
+    # fill with colors
+    for patch, color in zip(bplot["boxes"], colors):
+        patch.set_facecolor(color)
+
+
+for scenario_name in os.listdir(BENCHMARK_RESULTS_DIR):
+    middleware_datasets = get_middleware_dataset_for_scenario(scenario_name)
+    plot_dataset_of_scenario(plt, scenario_name, middleware_datasets)
+
+plt.show()

scripts/run_all_benchmarks.sh

@@ -0,0 +1,40 @@
+# Inspired from https://unix.stackexchange.com/questions/31414/how-can-i-pass-a-command-line-argument-into-a-shell-script
+helpFunction()
+{
+   echo ""
+   echo "Usage: $0 -i initial_script -d benchmark_results_directory"
+   echo -i "\t-i initial_script to run once all benchmarks are started"
+   echo -d "\t-d the directory the benchmark results is saved"
+   exit 1 # Exit script after printing help
+}
+
+while getopts "i:m:d:" opt
+do
+   case "$opt" in
+      i ) initial_script="$OPTARG" ;;
+      d ) benchmark_results_directory="$OPTARG" ;;
+      ? ) helpFunction ;; # Print helpFunction in case parameter is non-existent
+   esac
+done
+
+# Print helpFunction in case parameters are empty
+if [ -z "$initial_script" ] || [ -z "$benchmark_results_directory" ]
+then
+   echo "Some or all of the parameters are empty";
+   helpFunction
+fi
+
+echo "benchmark results directory is $benchmark_results_directory"
+
+echo "Benchmarking is starting!"
+echo "Starting initial scripts before benchmarks run!"
+. "$initial_script"
+echo "Initial script has finished! Now starting to benchmark middleware with scenarios!"
+
+mkdir ${benchmark_results_directory}/scenario_basic_service_client -p
+ros2 daemon stop
+ros2 launch moveit_middleware_benchmark scenario_basic_service_client_benchmark.launch.py benchmark_command_args:="--benchmark_out=${benchmark_results_directory}/scenario_basic_service_client/${RMW_IMPLEMENTATION}.json --benchmark_out_format=json --benchmark_repetitions=6"
+
+mkdir ${benchmark_results_directory}/scenario_perception_pipeline -p
+ros2 daemon stop
+ros2 launch moveit_middleware_benchmark scenario_perception_pipeline_benchmark.launch.py benchmark_command_args:="--benchmark_out=${benchmark_results_directory}/scenario_perception_pipeline/${RMW_IMPLEMENTATION}.json --benchmark_out_format=json --benchmark_repetitions=6"