Finish user-guide

README.md

@@ -1,6 +1,18 @@
 # Pythia rust project
 
-## About
+- [Pythia rust project](#pythia-rust-project)
+  - [About \& License](#about--license)
+  - [Code base](#code-base)
+  - [Installation instructions](#installation-instructions)
+  - [Usage](#usage)
+    - [In CloudLab, using pythia with Openstack](#in-cloudlab-using-pythia-with-openstack)
+    - [In docker image](#in-docker-image)
+    - [Getting Documentation of Pythia code base](#getting-documentation-of-pythia-code-base)
+  - [Pythia repositories](#pythia-repositories)
+  - [Forks of openstack projects](#forks-of-openstack-projects)
+
+
+## About & License
 
 This is the opensourced code base of **[DOCC Lab at Tufts University](https://docclab.cs.tufts.edu/)** and **[Peac Lab at Boston University](https://www.bu.edu/peaclab/)**'s full paper **[VAIF](https://dl.acm.org/doi/abs/10.1145/3544497.3544504?casa_token=rjolR5C9q9wAAAAA:7iVtHXCABI3JMd7OZ4OOqrs4U1EptpCHWNyqrLUDfUoVfYYHe65WLGeEBd2csmYe_p3eWqoIR2VJ3w) on SoCC21** (pervious short paper **[Pythia](https://dl.acm.org/doi/abs/10.1145/3357223.3362704) on SoCC19**)
 
@@ -37,8 +49,8 @@ Please follow [this user guide](https://github.com/docc-lab/pythia/blob/master/u
   - How to create an experiment with shared profile
   - How to setup initial search space for Pythia
   - How to execute Pythia
-  - How to injection different problems to OpenStack
-  - How to grab result of Pythia's analysis
+  - How to injection problem to OpenStack
+  - How to use Pythia to analyze injected problems
   - How to do troubleshooting
 
 ### In docker image

user-guide.md

@@ -1,10 +1,31 @@
-#VAIF User Guide
+# VAIF(Pythia) CloudLab Profile User Guide
+
 *(If any problem occurs, please refer to the troubleshooting section at the end of this guide)*
 
 To provide the best user experience, we decided to open-source VAIF with its codebase + a cloudlab profile that helps set up VAIF and the corresponding experiment's environment.
 
+- [VAIF(Pythia) CloudLab Profile User Guide](#vaifpythia-cloudlab-profile-user-guide)
+  - [Prerequisite](#prerequisite)
+  - [Experiment creation with the shared profile](#experiment-creation-with-the-shared-profile)
+  - [(Optional) Change number of compute node](#optional-change-number-of-compute-node)
+  - [Offline Profiling: Set search space for Pythia](#offline-profiling-set-search-space-for-pythia)
+  - [Start and Stop pythia](#start-and-stop-pythia)
+      - [Start continuous workload](#start-continuous-workload)
+      - [Start Pythia (push-the-button)](#start-pythia-push-the-button)
+      - [Stop pythia continous loop and continuous workload](#stop-pythia-continous-loop-and-continuous-workload)
+  - [Problem Injection](#problem-injection)
+      - [Recommended method](#recommended-method)
+      - [Alternative problem injection](#alternative-problem-injection)
+  - [Get Results of Analysis (a simple case study)](#get-results-of-analysis-a-simple-case-study)
+  - [Troubleshooting](#troubleshooting)
+      - [Check node stats](#check-node-stats)
+      - [check pythia agents \&\& restart pythia agenet if necessary](#check-pythia-agents--restart-pythia-agenet-if-necessary)
+      - [When you update pythia server](#when-you-update-pythia-server)
+      - [Pythia not compliling](#pythia-not-compliling)
+
+
 ## Prerequisite
-- CloudLab](https://www.cloudlab.us/) account
+- [CloudLab](https://www.cloudlab.us/) account
 - Basic knowledge of Linux
 - Basic understanding of [VAIF](https://dl.acm.org/doi/abs/10.1145/3472883.3487000 )
 
@@ -24,7 +45,7 @@ To provide the best user experience, we decided to open-source VAIF with its cod
      - One will get another email when it completes.
      - Then it will be ready to use
 
-## [Optional] Config Pythia (change number of compute node in openstack)
+## (Optional) Change number of compute node
 
 - If one wants more than more compute node in openstack and chages “number of compute nodes” during experiment creation to >1, please follow the following steps
   - If not, skip this section.
@@ -33,7 +54,7 @@ To provide the best user experience, we decided to open-source VAIF with its cod
 - One should SSH into ctl node after experiment creation and auto-setup finished
 Check the config /etc/pythia/controller.toml -- it should reflect the actual ctl and cp nodes. If one has 1 compute node, fix config (delete cp-2 and cp-3).
 
-## Set search space for Pythia (Offline Profiling)
+## Offline Profiling: Set search space for Pythia 
 
 - SSH into ctl node
 - Sudo su geniuser
@@ -42,23 +63,72 @@ Check the config /etc/pythia/controller.toml -- it should reflect the actual ctl
   - It takes around an hour to run
 - Go to pythia directory and run `pythia manifest ~/offline_traces.txt`
 
-## Start and Stop Continuous Loop:
-- Go to pythia directory and run pythia’s continuous loop
-`sudo RUST_BACKTRACE=1 cargo run --bin pythia_controller pythia_out 2>&1 | tee pythia_logs`
+## Start and Stop pythia
+#### Start continuous workload
 - Run continuous workload(an example workload) while pythia continuous loop is up
   - Open a new terminal window and ssh into the controller instance
   - Go to geniuser’s directory and run `~/pythia/workloads/continuous_workload.sh`
-- To stop pythia continous loop and continuous workload
+#### Start Pythia (push-the-button)
+- Go to pythia directory and run pythia’s continuous loop
+`sudo RUST_BACKTRACE=1 cargo run --bin pythia_controller pythia_out 2>&1 | tee pythia_logs`
+#### Stop pythia continous loop and continuous workload
   - Pythia: simply exit
   - Continous workload: ` kill $(ps aux | grep workload | awk '{print $2}')`
 
-## Problem Injection:
-
-
-## Get Results of Analysis:
-
-
-
-## Troubleshooting:
-
-
+## Problem Injection
+#### Recommended method
+- Ready problems for NOVA (note that when you apply the patch below, you need to choose one injected problem and change its sleep from 1-> 20 
+  - e.g., time.sleep (random.randint(0,20))
+- See the git diff file in the drive (problem_injections_nova_diff)
+- Apply this to your instance’s nova repo
+- Then do `pip_install` under /local/nova
+  - `sudo systemctl restart nova-compute.service`
+- Create the dummy dir 
+  - `sudo mkdir /users/output`
+  - `sudo chmod ugo+rwx /users/output/`
+- Do above steps for all nodes 
+
+#### Alternative problem injection
+- Take a look at the trace (e.g., server_create), and determine the tracepoint to inject latency (e.g., /local/nova/nova/virt/libvirt/imagebackend.py:355)
+- Inject with 
+  ```
+  import random
+  import time
+  time.sleep(random.randint(0,20))
+  ```
+- Do this for all nodes (i.e., ctl, cp-1 .,..)
+- Create the dummy dir in all nodes
+  - sudo mkdir /users/output
+  - sudo chmod ugo+rwx /users/output/
+- In all nodes, run `pip_install` then `sudo systemctl restart nova-compute.service`
+  - Alternatively `restart_openstack_ctl` or `restart_openstack_compute` according to the controller or compute instance
+- Then execute a workload
+
+
+## Get Results of Analysis (a simple case study)
+
+**Max_concurrent_builds**: Too low limit on simultaneous server creations throttles performance (Problem 3 from the paper
+- change `max_concurrent_builds` option to a low number (e.g., 2)
+- To do this so, go to nova.conf file (/etc/nova/nova.conf)
+- Then comment-in the option  `max_concurrent_builds` and set it to 2.
+- Finally, restart all the services (including nova)
+- Pythia will output results
+
+## Troubleshooting
+#### Check node stats 
+- `curl --data-binary '{"jsonrpc":"2.0","id":"curltext","method":"read_node_stats","params":[]}' -H 'content-type:application/json' http://cp-1:3030`
+#### check pythia agents && restart pythia agenet if necessary
+- `systemctl --type=service | grep pythia`
+- `sudo journalctl -u pythia.service`
+- `sudo systemctl restart pythia `
+  - Or stop and start
+
+#### When you update pythia server
+- `cargo install --path /local/reconstruction/pythia_server`
+- Then `sudo systemctl stop pythia`
+- `sudo systemctl start pythia`
+
+#### Pythia not compliling
+- Do `cargo run --help`, it compiles pythia and runs it with help arg. 
+- Consequently binary file of pythia is generated under  target/release. 
+- Then simply copy that bin ( target/release/pythia ) to /users/geniuser/.cargo/bin/, then it is fixed..