Merge branch 'develop'

Author: joe-at-startupmedia

README.md

@@ -2,8 +2,6 @@
 [![Testing](https://github.com/joe-at-startupmedia/pmon3/actions/workflows/testing.yml/badge.svg)](https://github.com/joe-at-startupmedia/pmon3/actions/workflows/testing.yml)
 
 
-
-
 `pmon3` is a process manager for Golang applications. It allows you to keep applications alive forever and to reload them without downtime.
 
 <img width="537" alt="pmon3_ls" src="https://github.com/joe-at-startupmedia/pmon3/assets/13522698/5d79ad53-664d-4ee7-bfac-f3fc94c2b316">
@@ -41,7 +39,7 @@ The systemd installation process entails the following steps:
 1. create the bash completion profile (requires the bash-completion package)
 1. enable and start the `pmond` system service
 
-```shell
+```bash
 #build the project
 make build
 #install on systemd-based system
@@ -52,14 +50,14 @@ make systemd_install
 ### Using Release Installer
 
 ```bash
-wget -O - https://raw.githubusercontent.com/joe-at-startupmedia/pmon3/master/release-installer.bash | bash -s 1.14.13
+wget -O - https://raw.githubusercontent.com/joe-at-startupmedia/pmon3/master/release-installer.bash | bash -s 1.15.0
 ```
 
 :exclamation::exclamation: Note :exclamation::exclamation:
 
 After installing `pmon3` for the first time, both installation methods provided above should automatically enable and start the service. if the `pmond` service does not start automatically, you need to manually start the service.
 
-```shell
+```bash
 sudo systemctl start pmond
 
 # Others
@@ -69,7 +67,7 @@ sudo /usr/local/pmon3/bin/pmond &
 <a name="section_commands"></a>
 ## Commands
 
-#### Help
+### Help
 
 ```
 Usage:
@@ -79,29 +77,30 @@ Available Commands:
   completion  Generate completion script
   del         Delete process by id or name
   desc        Show process extended details
+  dgraph      Show the process queue order
   drop        Delete all processes
   exec        Spawn a new process
   help        Help about any command
-  init        Restart all stopped processes
+  init        initialize all stopped processes
   kill        Terminate all processes
   log         Display process logs by id or name
   logf        Tail process logs by id or name
   ls          List all processes
-  restart     Restart a process by id or name
+  restart     (re)start a process by id or name
   stop        Stop a process by id or name
   topn        Shows processes using the native top
   version
 
 Flags:
   -h, --help   help for pmon3
 
-Use "pmon3 [command] --help" for more information about a command
+Use "pmon3 [command] --help" for more information about a command.
 ```
 
 <a name="pmon3_exec"></a>
-#### Running process [run/exec]
+### Running process [run/exec]
 
-```
+```bash
 pmon3 exec [application_binary] [flags]
 ```
 
@@ -113,7 +112,7 @@ The starting process accepts several parameters. The parameter descriptions are
 --name
 
 // Where to store logs. It will override the confuration files `logs_dir` property
---log-dir  -d
+--log-dir
 
 // The absolute path of a custom log file
 --log  -l
@@ -129,49 +128,52 @@ The starting process accepts several parameters. The parameter descriptions are
 
 // Do not restart automatically. It will automatically restart by default.
 --no-autorestart  -n
+
+// Provide a list of process names that this process will depend on
+--dependency parent-process-name [--dependency parent-process-name2]...
 ```
 
 #### Example：
 
-```
+```bash
 pmon3 exec ./bin/gin --args "-prjHome=`pwd`" --user ntt360
 ```
 
 :exclamation::exclamation: Note :exclamation::exclamation:
 
 Parameter arguments need to use the absolute path.
 
-#### View List  [ list/ls ]
+### View List  [ list/ls ]
 
-```
+```bash
 pmon3 ls
 ```
 
-#### Top Native [ topn ]
+### Top Native [ topn ]
 
 This will output the resource utilization of all processes using the native `top` command that is pre-installed on most unix-based operating systems. It will only show those processes managed by (and including) the `pmond` process. The output is updated every few seconds until the process is terminated using Ctrl+C.
 
-```
+```bash
 pmon3 topn
 ```
 <img width="559" alt="pmon3_topn" src="https://github.com/joe-at-startupmedia/pmon3/assets/13522698/a77cce0f-55b0-479f-8489-d6aaf9fcdd6b">
 
-#### (re)start the process [ restart/start ]
+### (re)start the process [ restart/start ]
 
-```
+```bash
 pmon3 restart [id or name]
 ```
 
 <a name="pmon3_stop"></a>
-#### Stop the process  [ stop ]
+### Stop the process  [ stop ]
 
-```
+```bash
 pmon3 stop [id or name]
 ```
 
-#### Process logging
+### Process logging
 
-```
+```bash
 # view logs of the process specified
 pmon3 log [id or name]
 
@@ -182,38 +184,55 @@ pmon3 log -a [id or name]
 pmon3 logf [id or name]
 ```
 
-#### Delete the process  [ del/delete ]
+### Delete the process  [ del/delete ]
 
-```
+```bash
 pmon3 del [id or name]
 ```
 
-#### View details [ show/desc ]
+### View details [ show/desc ]
 
-```
+```bash
 pmon3 show [id or name]
 ```
 
 <img width="475" alt="pmon3_show" src="https://github.com/joe-at-startupmedia/pmon3/assets/13522698/6b564a1c-0e26-468c-bd01-6dabce0c7620">
 
 <a name="pmon3_kill"></a>
-#### Terminate all running process [ kill ]
-```
+### Terminate all running process [ kill ]
+```bash
 pmon3 kill [--force]
 ```
 
 <a name="pmon3_init"></a>
-#### Restart all stopped process or start processes specified in the app config [ init ]
-```
+### (re)start all stopped process [ init ]
+```bash
+#(re)start processes specified in the Apps Config only
+pmon3 init --apps-config-only
+
+#(re)start processes specified in the Apps Config and those which already exist in the database
 pmon3 init
 ```
 
 <a name="pmon3_drop"></a>
-#### Terminate and delete all processes [drop]
-```
+### Terminate and delete all processes [drop]
+```bash
 pmon3 drop [--force]
 ```
 
+<a name="pmon3_dgraph"></a>
+### Display the dependency graph [dgraph/order]
+
+This command is useful to debug dependency resolution without (re)starting processes
+
+```bash
+#processes specified in the Apps Config only
+pmon3 dgraph --apps-config-only
+
+#processes specified in the Apps Config and the database
+pmon3 dgraph
+```
+
 <a name="section_config"></a>
 ## Configuration
 The default path of the configuration file is `/etc/pmon3/config/config.yml`. This value can be overridden with the `PMON3_CONF` environment variable. 
@@ -233,6 +252,8 @@ The following configuration options are available:
 #cmd_exec_response_wait: 1500
 # wait [n] milliseconds after connecting to IPC client before issuing commands
 #ipc_connection_wait: 0
+# wait [n] milliseconds after enqueueing a dependent process
+#dependent_process_enqueued_wait: 2000
 # a script to execute when a process is restarted which accepts the process details json as the first argument
 #on_process_restart_exec:
 # a script to execute when a process fails (--no-autorestart) which accepts the process details json as the first argument
@@ -249,6 +270,8 @@ The following configuration options are available:
 #apps_config_file: /etc/pmon3/config/app.config.json
 ```
 
+All configuration changes are effective when the next command is issued - restarting pmond is unnecessary.
+
 The configuration values can be overridden using environment variables:
 
 
@@ -258,6 +281,7 @@ The configuration values can be overridden using environment variables:
 * `CONFIGOR_HANDLEINTERRUPTS`
 * `CONFIGOR_CMDEXECRESPONSEWAIT`
 * `CONFIGOR_IPCCONNECTIONWAIT`
+* `CONFIGOR_DEPENDENTPROCESSENQUEUEDWAIT`
 * `CONFIGOR_ONPROCESSRESTARTEXEC`
 * `CONFIGOR_ONPROCESSFAILUREEXEC`
 * `CONFIGOR_POSIXMESSAGEQUEUEDIR`
@@ -270,15 +294,9 @@ The configuration values can be overridden using environment variables:
 ## Application(s) Config
 
 By default, when `pmond` is restarted from a previously stopped state, it will load all processes in the database that were previously running, have been marked as stopped as a result of pmond closing and have `--no-autorestart` set to false (default value).
+If applications are specified in the Apps Config, they will overwrite matching processes which already exist in the database.
 
-#### Criteria
- When an application config is provided `pmond` will instead refer to the `apps` array specified based on the following criteria:
-1. When `pmond` is starting from a fresh install
-2. When `pmon3` successfully executes a [drop](#pmon3_drop) command followed by running an [init](#pmon3_init) command.
-
-If [init](#pmon3_init) is ran while there are still stopped process in the database (resulting from `pmond` daemon restart or [kill](#pmon3_kill), the Application config will *NOT* be used, and instead the previously-stopped process will be restarted.
-
-#### Configuration
+### Configuration
 
 #### /etc/pmon3/config/config.yml
 ```yaml
@@ -297,6 +315,7 @@ apps_config_file: /etc/pmon3/config/apps.config.json
         "args": "-h startup-patroni-1.node.consul -p 5555 -r 5000",
         "user": "vagrant",
         "log_dir": "/var/log/custom/",
+        "dependencies": ["happac2"]
       }
     },
     {
@@ -305,7 +324,7 @@ apps_config_file: /etc/pmon3/config/apps.config.json
         "name": "happac2",
         "log": "/var/log/happac2.log",
         "args": "-h startup-patroni-1.node.consul -p 5556 -r 5001",
-        "user": "vagrant"
+        "user": "vagrant",
         "no_auto_restart": true
       }
     },
@@ -314,14 +333,21 @@ apps_config_file: /etc/pmon3/config/apps.config.json
       "flags": {
         "name": "metabase-api",
         "args": "/var/www/vhosts/metabase-api/index.js",
-        "env_vars": "NODE_ENV=prod"
+        "env_vars": "NODE_ENV=prod",
         "user": "dw_user"
       }
     }
   ]
 }
 ```
 
+### Dependencies
+
+Depenencies can be provided as a json array and determine the order in which the processes are booted. They are sorted using a directed acyclic graph meaning that there cannot be cyclical dependencies between processes (for obvious reasons). Dependecy resolution can be debugged using the [dgraph](#pmon3_dgraph) command. Parent processes can wait [n] amount of seconds between spawning dependent processes by utilziing the `dependent_process_enqueued_wait` configuration variable which currently defaults to 2 seconds.
+
+
+### Flags
+
 All possible `flags` values matching those specified in the [exec](#exec_flags) command:
 
 * user
@@ -331,6 +357,7 @@ All possible `flags` values matching those specified in the [exec](#exec_flags)
 * args
 * env_vars
 * name
+* dependencies
 
 <a name="section_events"></a>
 ## Event Handling With Custom Scripts
@@ -342,13 +369,13 @@ on_process_restart_exec: ""
 on_process_failure_exec: ""
 ```
 
-#### 1. Specify the executable script to run for the `on_process_restart_exec` value. `pmond` will pass a json-escaped string of the process details as the first argument.
+### 1. Specify the executable script to run for the `on_process_restart_exec` value. `pmond` will pass a json-escaped string of the process details as the first argument.
 #### /etc/pmond/config/config.yml
 ```yaml
 on_process_restart_exec: "/etc/pmon3/bin/on_restart.bash"
 ```
 
-#### 2. create the script to accept the json-escaped process details:
+### 2. create the script to accept the json-escaped process details:
 #### /etc/pmon3/bin/on_restart.bash
 ```bash
 PROCESS_JSON="$1"
@@ -357,15 +384,15 @@ PROCESS_NAME=$(echo "${PROCESS_JSON}" | jq '.name')
 echo "process restarted: ${PROCESS_ID} - ${PROCESS_NAME}" >> /var/log/pmond/output.log
 ```
 
-#### 3. start pmond in debug mode
+### 3. start pmond in debug mode
 ```bash 
 $ PMON3_DEBUG=true pmond
 INFO/vagrant/go_src/pmon3/pmond/observer/observer.go:29 pmon3/pmond/observer.HandleEvent() Received event: &{restarted 0xc0001da630}
 WARN/vagrant/go_src/pmon3/pmond/observer/observer.go:47 pmon3/pmond/observer.onRestartEvent() restarting process: happac3 (3)
 DEBU/vagrant/go_src/pmon3/pmond/observer/observer.go:70 pmon3/pmond/observer.onEventExec() Attempting event executor(restarted): /etc/pmon3/bin/on_restart.bash "{\"id\":3,\"created_at\":\"2024-05-03T05:44:25.114957302Z\",\"updated_at\":\"2024-05-03T06:09:18.71222185Z\",\"pid\":4952,\"log\":\"/var/log/pmond/acf3f83.log\",\"name\":\"happac3\",\"process_file\":\"/usr/local/bin/happac\",\"args\":\"-h startup-patroni-1.node.consul -p 5557 -r 5002\",\"status\":2,\"auto_restart\":true,\"uid\":1000,\"username\":\"vagrant\",\"gid\":1000}"
 ```
 
-#### 4. confirm the script executed successfully
+### 4. confirm the script executed successfully
 ```bash
 $ tail /var/log/pmond/output.log
 process restarted: 4 - "happac4"
@@ -468,12 +495,12 @@ If there is a path in the parameters you pass, please use the absolute path. The
 
 `pmon3` provides Bash automation. If you find that the command cannot be automatically provided, please install `bash-completion` and exit the terminal to re-enter:
 
-```shell
+```bash
 sudo yum install -y bash-completion
 ```
 
 #### Using ZSH instead of Bash
-```shell
+```bash
 autoload -U +X compinit && compinit
 autoload -U +X bashcompinit && bashcompinit
 sudo sh -c "pmon3 completion zsh > /etc/profile.d/pmon3.sh"
@@ -484,7 +511,7 @@ source /etc/profile.d/pmon3.sh
 
 If you encounter the error above, make sure the `pmond` service has started successfully.
 
-```shell
+```bash
 sudo systemctl start pmond
 ```
 
@@ -493,13 +520,13 @@ sudo systemctl start pmond
 You should only use `sudo` to start the `pmond` process which requires superuser privileges due to the required process forking commands. However, the `pmon3` cli should be used *without* `sudo` to ensure that the spawned processes are attached to the correct parent pid. When using `sudo`, the processes will be attached to ppid 1 and as a result, will become orphaned if the `pmond` process exits prematurely. Using `sudo` also prevents non-root users from being able to access the log files. The following Makefile command applies the adequate non-root permissions to the log files.
 
 #### Applying permissions
-```shell
+```bash
 #This is automatically called by make systemd_install
 make systemd_permissions
 ```
 
 #### Spawn a new process as the root user without using `sudo`
-```
+```bash
 pmon3 exec /usr/local/bin/happac --user root
 ```