Sync SURF prodcution updates (#110)

* New SCL release with enhancements and bug fixes * update last changes * some update munge, templateing * must add emplates/ssh/known_hosts.mustache * as suggested by Jaap
basvandervlies · May 12, 2023 · c15fc0a · c15fc0a
1 parent 4a48cdb
commit c15fc0a
Show file tree

Hide file tree

Showing 30 changed files with 602 additions and 315 deletions.
diff --git a/Changelog.md b/Changelog.md
@@ -1,5 +1,6 @@
 <!-- vim-markdown-toc GFM -->
 
+* [Version: 1.5.0 (2023-05-11)](#version-150-2023-05-11)
 * [Version: 1.4.1 (2022-07-12)](#version-141-2022-07-12)
 * [Version: 1.4.0 (2022-06-06)](#version-140-2022-06-06)
 * [Version: 1.3.0 (2022-05-04)](#version-130-2022-05-04)
@@ -29,6 +30,66 @@
 * [Version: 0.9.0 (2018-08-24)](#version-090-2018-08-24)
 
 <!-- vim-markdown-toc -->
+# Version: 1.5.0 (2023-05-11)
+
+SCL enhancements:
+ * drop support for CFengine 3.7 and 3.10 Code is removed
+ * Added `run_class` option to `copy_files`
+ * `scl_mustach_copy` can now handle debug output for string and list
+ * `scl_service_rotate_files` logic rewrite so that it works as expected
+ * `scl_copy_files` and `scl_service_install_tarballs` fix for `run_bundle` did not work must use class `<file>_repaired`
+ * added new scl library bodies:
+     * `scl_cmd_kept`:  Only set class `{value}_succeded` when command exits with value `0`
+ * The json filenames that must be loaded can now contain variable names, eg:
+```
+"ssh": {
+   "json_files": [
+                "soil_$(def.cluster_role).json"
+   ]
+```
+ *  New method for copy/expanding mustache templates `scl_mustache_service_autorun`, Each bundle can define templats to be used, eg:
+     * `scl_mustache_service_autorun("resolv", "")`, uses `resolv.template_2_destination`
+     * `scl_mustache_service_autorun("resolv", "resolv_other_bundle")`, uses `resolv_other_bundle.template_2_destination`
+
+These services have bug fixes or new features:
+ * jupyterhub:
+    * Added `scl_service_copy_dirs` functionality
+ * munge:
+    * Option to specify the uid/gid for the munge user/group. Controlled via class MUNGE_CHECK_UID_GID
+ * node_status:
+    * Must use directory mode bits for cfengine 3.20 and higher
+ * pam
+    * Added generation of /etc/security/limits.sh
+ * rootfiles
+    * New `user_ssh_keys_dir` variable for copying user private/public keys to `/root/.ssh`, needed for git repo's
+ * ssh
+    * remove debian_8 setup
+    * Added `scl_service_copy_dirs` functionality
+    * added new class `SSH_HOST_CERTIFICATE`
+      * will generate `$(ssh.config_dir)/ssh_known_hosts2` file with the aid of json variable `scl.ssh.cert_authorities`
+      * [ssh host certificate setup]( https://berndbausch.medium.com/ssh-certificates-a45bdcdfac39)
+```
+"ssh": {
+   "classes": }
+        "HOST_CERTIFICATE": "any"
+   },
+   "cert_authorities": [
+        {
+        "servers": "*",
+        "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA9mwksJWzluDF8ZungY2TiOTYVb6TmuTNi8AsG5+hJa",
+        "comment": "CA-host@clustercomputing"
+        }
+    ]
+```
+ * slurm:
+    * Added slurm major version as class based on `current_version` definition, eg: `SLURM_21_08`
+    * When class `SLURM_CONFIGLESS_CONF_LINKS` is unset, remove the create symbolic links
+    * use `mode_file` and `mode_dir` to be compatible with cfengine 3,20 and higher
+    * SLURM version 22.05 and higher do not need file `cgroup_allowed_devices_file.conf`
+    * `interactive_step_options` added as json parameter. How start interactive job via salloc
+    * `launch_parameters` set launch parameters for job launch plugin, default: `use_interactive_step`
+    * `powersave_section` dictionary added to mustache/json options.
+
 # Version: 1.4.1 (2022-07-12)
 
 SCL enhancements:

diff --git a/doc/library.md b/doc/library.md
@@ -10,6 +10,7 @@
  * [packages](library/packages.md)
  * [paths](library/paths.md)
  * [processes](library/processes.md)
+ * [promises](library/promises.md)
  * [services](library/services.md)
  * [stdlib](library/stdlib.md)
  * [storage](library/storage.md)

diff --git a/doc/library/files.md b/doc/library/files.md
@@ -50,7 +50,8 @@ This bundle will read the json attribute `copy_files` from the specified `bundle
 attribute can be specified:
  * dest: destination
  * mog: mode/owner/group of the file
- * run_bundle: The name of the bundle to run when something is repaired, default not set.
+ * run_bundle: The name of the bundle to run when something is repaired, default not set
+ * run_class: When set only copy the file is class condition is met, default is `any`
  * secure_cp from mpf is used to copy the files
  * source: source
 
@@ -59,9 +60,16 @@ The `copy_files` can specified by json data eg:
 "slurm": {
     "copy_files": [
         {
-            "dest": "/opt/slurm/etc//job_submit.lua",
+            "dest": "/opt/slurm/etc/job_submit.lua",
             "source": "data/slurm/lisa/job_submit.lua",
-            "mog": ["0644", "slurm", "slurm"]
+            "mog": [
+                "0644",
+                "slurm",
+                "slurm"
+            ],
+            "run_class": [
+                "SLURM_SERVER"
+            ]
         }
     ]
 }

diff --git a/doc/library/promises.md b/doc/library/promises.md
@@ -0,0 +1,2 @@
+Source: [stdlib.cf](/masterfiles/lib/scl/promises.cf)
+
diff --git a/doc/library/templates.md b/doc/library/templates.md
@@ -77,7 +77,14 @@ The bundle is called by:
  * scl_data_autorun or scl_mustache_autorun
 ## scl_mustache_autorun(bundle_name)
 
-This  bundle will take of copying the mustache template file(s) and expanding the template(s)
+This bundle will take of copying the mustache template file(s) and expanding the template(s)
+## scl_mustache_service_autorun(bundle_name)
+
+This bundle will take of copying the mustache template file(s) and expanding the template(s). The
+parameter required is the `service_name` and optional we can specify form which bundle we need
+to get the `template_2_destination` json definition from, eg:
+ * `scl_mustache_service_autorun("resolv", "")`, uses `resolv.template_2_destination`
+ * `scl_mustache_service_autorun("resolv", "resolv_define")`, uses `resolv_define.template_2_destination`
 ## scl_mustache_copy(bundle_name, files)
 
 The bundle is internally and will take care of copying the template file(s) from the bundle data directory
@@ -88,21 +95,22 @@ on the policy hub to a the local node directory. The bundle is called by:
 This bundle is used internally and will expand the mustache template file(s) with the json data. The
 bundle is called by:
  * scl_mustache_autorun
-## scl_mustache_cf_data_2_file(bundle_name, template_file, destination , data_section)
+ * scl_mustache_service_autorun
+## scl_mustache_cf_data_2_file(service_name, template_file, destination , data_section)
 
 With this bundle you can generate a file from a template with cfengine internal json data. There are 2
 options:
  1. json data section variable passed as argument. Then the section will be merged from the
     cfengine internal json data as toplevel. So variables in mustache file must be referenced
-    without the bundle name, eg `<variable_name>` instead of `vars.scl.$(bundle_name).<variable_name>`
+    without the service name, eg `<variable_name>` instead of `vars.scl.$(service_name).<variable_name>`
  1. cfengine internal json data if you do not pass your own json data section variable
 
 The template is fetch from the local node direcrory:
- * `$(def.node_template_dir)/$(bundle_name)/$(template_file)`
+ * `$(def.node_template_dir)/$(service_name)/$(template_file)`
 
 When json data section variable is specified. You must specify which date section you want to use. The json
 files must be merged via scl, as we only can merge data from this bundle, eg:
- * `scl.<bundle_name>[$(data_section)]`
+ * `scl.<service_name>[$(data_section)]`
 
 This data will then be used as toplevel for the mustache template. You can just use the variables
 name(s): `<variable_name>`
diff --git a/doc/services/jupyterhub.md b/doc/services/jupyterhub.md
@@ -9,7 +9,8 @@ node. Each jupyterhub has its own parameters for submitting jobs. Apache is used
 see [apache.cf](/services/apache.cf). This service bundle will also generate the apache configuration
 file for you.
 
-The following json variables can be set in def.cf/json to  invoke files bundles:
+The following json variables can be set in def.cf/json to invoke files bundles:
+ * copy_dirs: See [files.cf](/masterfiles/lib/scl/files.cf)
  * install_tarballs: See [files.cf](/masterfiles/lib/scl/files.cf)
 
 The following clases can be set via def.cf/json:

diff --git a/doc/services/rootfiles.md b/doc/services/rootfiles.md
@@ -41,7 +41,7 @@ and extra json file(s) can be specified via:
 ```
 vars:
     any::
-        "rootfiles_json_files" slist => { "surfsara.json" };
+        "rootfiles_json_files" slist => { "surf.json" };
 ```
 
 The variable must be `root_files_json_files` and with this setup 1 extra json file will be merged.
@@ -92,7 +92,7 @@ if there are changes, eg:
 
 ### COPY_FILES
 
-With this variable is set it will copy the specified file(s) to the
+When this variable is set it will copy the specified file(s) to the
 specified destination, example:
 ```
 copy_files: [
@@ -119,3 +119,11 @@ ssh_keys: {
     "keys": [ "id_ed25519", "id_rsa" ]
 }
 ```
+
+### user_ssh_keys_dir
+
+When this variable is set we copy user private/public keys to the `/root/.ssh` directory. These are user keys for an user that can become root
+but want to use their own git repositories. Use `ssh-add -t 4h /root/.ssh/root-bas` in combo with `ssh-agent`, eg:
+```
+"user_ssh_keys_dir": "data/rootfiles/ssh_keys/git_user"
+```
diff --git a/doc/services/ssh.md b/doc/services/ssh.md
@@ -85,25 +85,5 @@ vars:
  * override server setting in def.cf
 ```
 vars:
-    "ssh" data => parsejson( '{ "X11Forwarding":  "no"  }' );
+     "ssh" data => parsejson( '{ "X11Forwarding":  "no"  }' );
 ```
-
-### COPY_FILES
-
-When this variable is set it will copy the specified file to the `ssh.config_dir`, eg:
-```json
-"copy_files": [
-    {
-        "dest": "$(ssh.config_dir)/shosts.equiv",
-        "source": "data/ssh/lisa/shosts.equiv",
-        "mog": [ "0644", "root", "root" ]
-    },
-    {
-        "dest": "$(ssh.config_dir)/ssh_known_hosts2",
-        "source": "data/ssh/lisa/ssh_known_hosts2",
-        "mog": [ "0644", "root", "root" ],
-        "run_bundle": "ssh_daemons_restart"
-    }
-]
-```
-where `data` is a ''cf-serverd shortcut''.
diff --git a/masterfiles/lib/scl/common.cf b/masterfiles/lib/scl/common.cf
@@ -27,6 +27,12 @@ body classes scl_cmd_repair(return_code,class_prefix)
     promise_kept     => { "$(class_prefix)_succeeded" };
 }
 
+body classes scl_cmd_kept(class_prefix)
+{
+      kept_returncodes => { "0" };
+      promise_kept     => { "$(class_prefix)_succeeded" };
+}
+
 body action scl_report(level)
 {
     report_level => $(level);

diff --git a/masterfiles/lib/scl/files.cf b/masterfiles/lib/scl/files.cf
@@ -56,7 +56,8 @@ This bundle will read the json attribute `copy_files` from the specified `bundle
 attribute can be specified:
  * dest: destination
  * mog: mode/owner/group of the file
- * run_bundle: The name of the bundle to run when something is repaired, default not set.
+ * run_bundle: The name of the bundle to run when something is repaired, default not set
+ * run_class: When set only copy the file is class condition is met, default is `any`
  * secure_cp from mpf is used to copy the files
  * source: source
 
@@ -65,9 +66,16 @@ The `copy_files` can specified by json data eg:
 "slurm": {
     "copy_files": [
         {
-            "dest": "/opt/slurm/etc//job_submit.lua",
+            "dest": "/opt/slurm/etc/job_submit.lua",
             "source": "data/slurm/lisa/job_submit.lua",
-            "mog": ["0644", "slurm", "slurm"]
+            "mog": [
+                "0644",
+                "slurm",
+                "slurm"
+            ],
+            "run_class": [
+                "SLURM_SERVER"
+            ]
         }
     ]
 }
@@ -171,7 +179,7 @@ body file_select scl_days_old_by_atime(days)
     #atime       => irange(ago(3,0,0,0,0,0),ago(0,0,"$(days)",0,0,0));
     # this select all files that have not access for 'days'
     atime => irange(0,ago(0,0,"$(days)",0,0,0));
-    file_result => "$(file_time)";
+    file_result => "atime";
 }
 
 body link_from scl_relative_ln_s(x)
@@ -314,40 +322,43 @@ bundle agent scl_service_rotate_files(bundle_name, data)
             "index" slist => getindices("rotate_files_data");
 
             "run_class_var_$(index)" slist => getvalues("rotate_files_data[$(index)][run_class]"),
-                handle => "$(this.bundle):$(bundle_name):$(index)",
                 comment => "Only run package action if this class is true";
 
-            "run_class_var_str_$(index)" string => ifelse(
-                    "run_class_set_$(index)", format("%s", "$(run_class_var_$(index))"),
-                    "any"
-                ),
-                handle => "$(this.bundle):$(bundle_name):$(index)",
-                if => "DEBUG|DEBUG_$(bundle_name)|DEBUG_$(this.bundle)";
-
 classes:
     any::
         "json_rotate_files_specified" expression => isvariable("scl.$(bundle_name)[rotate_files]"),
             comment => "Do we override the bundle services rotate_files  with def.json settings";
 
-        "run_class_set_$(index)" expression => classify("$(rotate_files_data[$(index)][run_class])"),
-            comment => "if not set assume 'any' class";
+        "run_class_set" expression => reglist("@(run_class_var_$(index))", ".+"),
+            comment => "Service has specified classes. so check if there are met, if not set assume 'any' class";
+
+        "run_class" expression => "$(run_class_var_$(index))",
+            comment => "Run service if run_class definitions are met",
+            if => "run_class_set";
 
 files:
     any::
         "$(rotate_files_data[$(index)][log_file])"
             comment => "rotate_files: $(rotate_files_data[$(index)][log_file]), number of backups: $(rotate_files_data[$(index)][number_of_backups])",
             rename  => rotate("$(rotate_files_data[$(index)][number_of_backups])"),
-            if => "run_class_set_$(index)";
+            if => or(
+                "run_class",
+                "!run_class_set"
+            );
 
         "$(rotate_files_data[$(index)][log_file]).1"
             comment     => "rotate_files: gzip $(rotate_files_data[$(index)][log_file]).1",
             transformer => "$(paths.path[gzip]) $(this.promiser)",
-            if => "run_class_set_$(index)";
+            if => or(
+                "run_class",
+                "!run_class_set"
+            );
 
-reports:
+methods:
     any::
-        "$(this.bundle):$(bundle_name): index: $(index) file: $(rotate_files_data[$(index)][log_file]) #rotates: $(rotate_files_data[$(index)][number_of_backups]) class: $(rotate_files_data[$(index)][run_class]) run_class_var_str: $(run_class_var_str_$(index))"
-            if => "DEBUG|DEBUG_$(bundle_name)|DEBUG_$(this.bundle)";
+        "" usebundle => scl_show_data("$(this.bundle)", "rotate_files_data[$(index)]"),
+            handle => "$(this.bundle):$(bundle_name):$(index)",
+            if => "DEBGU|DEBUG_$(bundle_name)|DEBUG_$(this.bundle)";
 }
 
 bundle agent scl_service_copy_dirs(bundle_name)
@@ -501,6 +512,17 @@ vars:
     any::
         "index" slist => getindices("scl.$(bundle_name)[$(section)]");
 
+        "run_class_var_$(index)" slist => getvalues("scl.$(bundle_name)[$(section)][$(index)][run_class]"),
+            handle => "$(this.bundle):$(bundle_name):$(section):$(index)",
+            comment => "Only copy files if this class is true";
+
+        "run_class_var_str_$(index)" string => ifelse(
+                "run_class_set_$(index)", format("%s", "$(run_class_var_$(index))"),
+                "any"
+            ),
+            handle => "$(this.bundle):$(bundle_name):$(section):$(index)",
+            if => "DEBUG|DEBUG_$(bundle_name)|DEBUG_$(this.bundle)";
+
 classes:
     any::
         "$(bundle_name)_$(section)_$(index)_run_bundle" expression => isvariable("scl.$(bundle_name)[$(section)][$(index)][run_bundle]"),
@@ -512,29 +534,43 @@ classes:
                 canonify("$(bundle_name)_$(section)$(scl.$(bundle_name)[$(section)][$(index)][dest])_repaired")
                 );
 
+        "run_class_$(index)" expression => classify("$(scl.$(bundle_name)[$(section)][$(index)][run_class])"),
+            comment => "Run service if run_class definitions are met";
+
+        "run_class_set_$(index)" expression => reglist("@(run_class_var_$(index))", ".+"),
+            comment => "Service has specified classes. so check if thetre are met, if not set assume 'any' class";
+
 
 files:
     any::
         "$(scl.$(bundle_name)[$(section)][$(index)][dest])"
             comment => "Copy bundle files and set class if its get copied",
             copy_from => secure_cp("$(scl.$(bundle_name)[$(section)][$(index)][source])", "$(sys.policy_hub)"),
             #classes => results("namespace", "$(scl.$(bundle_name)[$(section)][$(index)][dest])");
-            classes => results("namespace", "$(bundle_name)_$(section)$(scl.$(bundle_name)[$(section)][$(index)][dest])");
+            classes => results("namespace", "$(bundle_name)_$(section)$(scl.$(bundle_name)[$(section)][$(index)][dest])"),
+            if => or(
+                "run_class_$(index)",
+                "!run_class_set_$(index)"
+                );
 
         "$(scl.$(bundle_name)[$(section)][$(index)][dest])"
             comment => "Check permission",
             perms => mog(
                 "$(scl.$(bundle_name)[$(section)][$(index)][mog][0])",
                 "$(scl.$(bundle_name)[$(section)][$(index)][mog][1])",
                 "$(scl.$(bundle_name)[$(section)][$(index)][mog][2])"
-            );
+            ),
+            if => or(
+                "run_class_$(index)",
+                "!run_class_set_$(index)"
+                );
 
 methods:
     any::
                 #canonify("$(scl.$(bundle_name)[$(section)][$(index)][dest])"),
         "" usebundle => "$(scl.$(bundle_name)[$(section)][$(index)][run_bundle])",
             if => and(
-                canonify("$(bundle_name)_$(section)$(scl.$(bundle_name)[$(section)][$(index)][dest])"),
+                canonify("$(bundle_name)_$(section)$(scl.$(bundle_name)[$(section)][$(index)][dest])_repaired"),
                 canonify("$(bundle_name)_$(section)_$(index)_run_bundle")
                 );