Merge pull request #4 from mlcommons/dev

updated docs

README.md

@@ -1,4 +1,4 @@
-# CM repository with cross-platform scripts for DevOps, MLOps, AIOps and MLPerf
+# Cross-platform CM interface for DevOps, MLOps, AIOps and MLPerf
 
 [![License](https://img.shields.io/badge/License-Apache%202.0-green)](LICENSE.md)
 [![Powered by CM](https://img.shields.io/badge/Powered_by-MLCommons%20CM-blue)](https://github.com/mlcommons/ck).

automation/script/README-extra.md

@@ -69,17 +69,18 @@ while automatically adapting to a given environment!
 In order to reuse some CM scripts embedded into shared projects, 
 you need to install these projects via the CM interface.
 
-For example, to use automation scripts developed by the [MLCommons task force on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/list_of_scripts.md)
+For example, to use automation scripts developed by the 
+[MLCommons task force on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)
 and shared via GitHub, you just need to pull this repository via CM:
 
 ```bash
-cm pull repo --url=https://github.com/mlcommons/ck
+cm pull repo --url=https://github.com/mlcommons/cm4mlops --checkout=dev
 ```
 
 or
 
 ```bash
-cm pull repo mlcommons@ck
+cm pull repo mlcommons@cm4mlops --checkout=dev
 ```
 
 You can now see all available CM scripts in your system as follows:

automation/script/module.py

@@ -3686,7 +3686,7 @@ def doc(self, i):
                                     [ (artifact alias, artifact UID) ] or
                                     [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]
 
-          (repos) (str): list of repositories to search for automations (internal & mlcommons@ck by default)
+          (repos) (str): list of repositories to search for automations
 
           (output_dir) (str): output directory (../docs by default)
 
@@ -3751,7 +3751,7 @@ def dockerfile(self, i):
                                     [ (artifact alias, artifact UID) ] or
                                     [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]
 
-          (repos) (str): list of repositories to search for automations (internal & mlcommons@ck by default)
+          (repos) (str): list of repositories to search for automations
 
           (output_dir) (str): output directory (./ by default)
 
@@ -3779,7 +3779,7 @@ def docker(self, i):
                                     [ (artifact alias, artifact UID) ] or
                                     [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]
 
-          (repos) (str): list of repositories to search for automations (internal & mlcommons@ck by default)
+          (repos) (str): list of repositories to search for automations
 
           (output_dir) (str): output directory (./ by default)
 

automation/script/module_misc.py

@@ -106,7 +106,7 @@ def doc(i):
                                 [ (artifact alias, artifact UID) ] or
                                 [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]
 
-      (repos) (str): list of repositories to search for automations (internal & mlcommons@ck by default)
+      (repos) (str): list of repositories to search for automations
 
       (output_dir) (str): output directory (../docs by default)
 
@@ -149,6 +149,7 @@ def doc(i):
 
     # Search for automations in repos
     lst = []
+
     for repo in list_of_repos:
         parsed_artifact[1] = ('',repo) if utils.is_cm_uid(repo) else (repo,'')
         ii['parsed_artifact'] = parsed_artifact
@@ -611,8 +612,8 @@ def doc(i):
                              '',
                              '```cmr "cm gui" --script="'+','.join(tags)+'"```',
                              '',
-                             'Use this [online GUI](https://cKnowledge.org/cm-gui/?tags={}) to generate CM CMD.'.format(','.join(tags)),
-                             '',
+#                             'Use this [online GUI](https://cKnowledge.org/cm-gui/?tags={}) to generate CM CMD.'.format(','.join(tags)),
+#                             '',
                              '#### '+x4,
                              '',
                              '{}'.format(cli_all_tags_alternative_docker),
@@ -1339,7 +1340,7 @@ def dockerfile(i):
                                 [ (artifact alias, artifact UID) ] or
                                 [ (artifact alias, artifact UID), (artifact repo alias, artifact repo UID) ]
 
-      (repos) (str): list of repositories to search for automations (internal & mlcommons@ck by default)
+      (repos) (str): list of repositories to search for automations
 
       (output_dir) (str): output directory (./ by default)
 
@@ -1368,7 +1369,7 @@ def dockerfile(i):
 
     console = i.get('out') == 'con'
 
-    cm_repo = i.get('docker_cm_repo', 'mlcommons@ck')
+    cm_repo = i.get('docker_cm_repo', 'mlcommons@cm4mlops')
     cm_repo_flags = i.get('docker_cm_repo_flags', '')
 
     # Search for script(s)
@@ -1842,7 +1843,7 @@ def docker(i):
                 dockerfilename_suffix = dockerfilename_suffix[len(dockerfilename_suffix) - 1]
 
 
-        cm_repo=i.get('docker_cm_repo', 'mlcommons@ck')
+        cm_repo=i.get('docker_cm_repo', 'mlcommons@cm4mlops')
 
         docker_path = i.get('docker_path', '').strip()
         if docker_path == '': 

script/activate-python-venv/README.md

@@ -12,8 +12,8 @@ Developers: [Grigori Fursin](https://cKnowledge.org/gfursin)
 ---
 #### Summary
 
-* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/dev/cm-mlops)*
-* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv)*
+* CM GitHub repository: *[mlcommons@cm4mlops](https://github.com/mlcommons/cm4mlops/tree/dev)*
+* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv)*
 * CM meta description for this script: *[_cm.json](_cm.json)*
 * All CM tags to find and reuse this script (see in above meta description): *activate,python-venv*
 * Output cached? *False*
@@ -30,7 +30,7 @@ Developers: [Grigori Fursin](https://cKnowledge.org/gfursin)
 
 #### Pull CM repository with this automation recipe (CM script)
 
-```cm pull repo mlcommons@ck```
+```cm pull repo mlcommons@cm4mlops```
 
 #### Print CM help from the command line
 
@@ -79,8 +79,6 @@ if r['return']>0:
 
 ```cmr "cm gui" --script="activate,python-venv"```
 
-Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=activate,python-venv) to generate CM CMD.
-
 #### Run this script via Docker (beta)
 
 `cm docker script "activate python-venv" `
@@ -102,18 +100,18 @@ ___
 ### Dependencies on other CM scripts
 
 
-  1. Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/_cm.json)
-  1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/customize.py)***
-  1. ***Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/_cm.json)***
+  1. Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/_cm.json)
+  1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/customize.py)***
+  1. ***Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/_cm.json)***
      * install,python-venv
        * CM names: `--adr.['python-venv']...`
-       - CM script: [install-python-venv](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/install-python-venv)
+       - CM script: [install-python-venv](https://github.com/mlcommons/cm4mlops/tree/master/script/install-python-venv)
   1. ***Run native script if exists***
-     * [run.bat](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/run.bat)
-     * [run.sh](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/run.sh)
-  1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/_cm.json)
+     * [run.bat](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/run.bat)
+     * [run.sh](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/run.sh)
+  1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/_cm.json)
   1. Run "postrocess" function from customize.py
-  1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/activate-python-venv/_cm.json)
+  1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/activate-python-venv/_cm.json)
 
 ___
 ### Script output

script/add-custom-nvidia-system/README.md

@@ -12,8 +12,8 @@ Maintainers: [Public MLCommons Task Force on Automation and Reproducibility](htt
 ---
 #### Summary
 
-* CM GitHub repository: *[mlcommons@ck](https://github.com/mlcommons/ck/tree/dev/cm-mlops)*
-* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system)*
+* CM GitHub repository: *[mlcommons@cm4mlops](https://github.com/mlcommons/cm4mlops/tree/dev)*
+* GitHub directory for this script: *[GitHub](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system)*
 * CM meta description for this script: *[_cm.yaml](_cm.yaml)*
 * All CM tags to find and reuse this script (see in above meta description): *add,custom,system,nvidia*
 * Output cached? *True*
@@ -30,7 +30,7 @@ Maintainers: [Public MLCommons Task Force on Automation and Reproducibility](htt
 
 #### Pull CM repository with this automation recipe (CM script)
 
-```cm pull repo mlcommons@ck```
+```cm pull repo mlcommons@cm4mlops```
 
 #### Print CM help from the command line
 
@@ -81,8 +81,6 @@ if r['return']>0:
 
 ```cmr "cm gui" --script="add,custom,system,nvidia"```
 
-Use this [online GUI](https://cKnowledge.org/cm-gui/?tags=add,custom,system,nvidia) to generate CM CMD.
-
 #### Run this script via Docker (beta)
 
 `cm docker script "add custom system nvidia[variations]" `
@@ -126,48 +124,48 @@ ___
 ### Dependencies on other CM scripts
 
 
-  1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/_cm.yaml)***
+  1. ***Read "deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/_cm.yaml)***
      * detect,os
-       - CM script: [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
+       - CM script: [detect-os](https://github.com/mlcommons/cm4mlops/tree/master/script/detect-os)
      * detect,cpu
-       - CM script: [detect-cpu](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-cpu)
+       - CM script: [detect-cpu](https://github.com/mlcommons/cm4mlops/tree/master/script/detect-cpu)
      * get,sys-utils-cm
-       - CM script: [get-sys-utils-cm](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-sys-utils-cm)
+       - CM script: [get-sys-utils-cm](https://github.com/mlcommons/cm4mlops/tree/master/script/get-sys-utils-cm)
      * get,python3
        * CM names: `--adr.['python', 'python3']...`
-       - CM script: [get-python3](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-python3)
+       - CM script: [get-python3](https://github.com/mlcommons/cm4mlops/tree/master/script/get-python3)
      * get,cuda,_cudnn
-       - CM script: [get-cuda](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cuda)
+       - CM script: [get-cuda](https://github.com/mlcommons/cm4mlops/tree/master/script/get-cuda)
      * get,tensorrt
-       - CM script: [get-tensorrt](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-tensorrt)
+       - CM script: [get-tensorrt](https://github.com/mlcommons/cm4mlops/tree/master/script/get-tensorrt)
      * get,cmake
-       - CM script: [get-cmake](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-cmake)
+       - CM script: [get-cmake](https://github.com/mlcommons/cm4mlops/tree/master/script/get-cmake)
      * get,generic-python-lib,_requests
-       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-python-lib)
      * get,generic,sys-util,_glog-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,generic,sys-util,_gflags-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,generic,sys-util,_libre2-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,generic,sys-util,_libnuma-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,generic,sys-util,_libboost-all-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,generic,sys-util,_rapidjson-dev
-       - CM script: [get-generic-sys-util](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-sys-util)
+       - CM script: [get-generic-sys-util](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-sys-util)
      * get,nvidia,mlperf,inference,common-code
        * CM names: `--adr.['nvidia-inference-common-code']...`
-       - CM script: [get-mlperf-inference-nvidia-common-code](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-mlperf-inference-nvidia-common-code)
+       - CM script: [get-mlperf-inference-nvidia-common-code](https://github.com/mlcommons/cm4mlops/tree/master/script/get-mlperf-inference-nvidia-common-code)
      * get,generic-python-lib,_pycuda
-       - CM script: [get-generic-python-lib](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/get-generic-python-lib)
-  1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/customize.py)***
-  1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/_cm.yaml)
+       - CM script: [get-generic-python-lib](https://github.com/mlcommons/cm4mlops/tree/master/script/get-generic-python-lib)
+  1. ***Run "preprocess" function from [customize.py](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/customize.py)***
+  1. Read "prehook_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/_cm.yaml)
   1. ***Run native script if exists***
-     * [run.sh](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/run.sh)
-  1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/_cm.yaml)
-  1. ***Run "postrocess" function from [customize.py](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/customize.py)***
-  1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/ck/tree/dev/cm-mlops/script/add-custom-nvidia-system/_cm.yaml)
+     * [run.sh](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/run.sh)
+  1. Read "posthook_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/_cm.yaml)
+  1. ***Run "postrocess" function from [customize.py](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/customize.py)***
+  1. Read "post_deps" on other CM scripts from [meta](https://github.com/mlcommons/cm4mlops/tree/dev/script/add-custom-nvidia-system/_cm.yaml)
 
 ___
 ### Script output