Merge pull request #246 from grycap/change-doc

Update documentation

docs/additional-config.md

@@ -0,0 +1,20 @@
+# Additional configuration
+
+To give the administrator a more personalized cluster configuration, the OSCAR manager searches for a config map on the cluster with the additional properties to apply. Since this is still a work in progress, the only configurable property currently is the container images' origin. As seen in the following ConfigMap definition, you can set a list of "prefixes" that you consider secure repositories, so images that do not come from one of these are restricted.
+
+``` yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: additional-oscar-config
+  namespace: oscar-svc
+data:
+  config.yaml: |
+    images:
+      allowed_prefixes:
+      - ghcr.io
+```
+
+Additionally, this property can be added when creating an OSCAR cluster through the IM, which will automatically create the ConfigMap.
+
+![allowed-prefixes](images/im-dashboard/im-additional-config.png)

docs/exposed-services.md

@@ -38,7 +38,6 @@ Once the service is deployed, you can check if it was created correctly by makin
 
 ``` bash
 https://{oscar_endpoint}/system/services/{service_name}/exposed/{path_resource} 
-
 ```
 
 Notice that if you get a `502 Bad Gateway` error, it is most likely because the specified port on the service does not match the API port.
@@ -128,7 +127,7 @@ functions:
      expose:
       min_scale: 2 
       max_scale: 10 
-      port: 80  
+      api_port: 80  
       cpu_threshold: 50 
 ```
 

docs/fdl.md

@@ -86,7 +86,7 @@ storage_providers:
 | `name` </br> *string*                                             | The name of the service                                                                                                                                                                                                                                      |
 | `cluster_id` </br> *string*                                       | Identifier for the current cluster, used to specify the cluster's StorageProvider in job delegations. OSCAR-CLI sets it using the _cluster_id_ from the FDL. Optional. (default: "")                                                                            |
 | `image` </br> *string*                                            | Docker image for the service                                                                                                    |
-| `vo` </br> *string*                                               | Virtual Organization (VO) in which the user creating the service is enrolled. Optional (default: "")                                                             |
+| `vo` </br> *string*                                               | Virtual Organization (VO) in which the user creating the service is enrolled. (Required for multitenancy)                                                             |
 | `allowed_users` </br> *string array*                    | Array of EGI UIDs to grant specific user permissions on the service. If empty, the service is considered as accesible to all the users with access to the OSCAR cluster. (Enabled since OSCAR version v3.0.0).                                                                                                                                                                |
 | `alpine` </br> *boolean*                                          | Set if the Docker image is based on Alpine. If `true`, a custom release of the [faas-supervisor](https://github.com/grycap/faas-supervisor) will be used. Optional (default: false)                                                                                                                   |
 | `script` </br> *string*                                           | Local path to the user script to be executed inside the container created out of the service invocation                                                                                                                                                                                        |

docs/mount.md

@@ -0,0 +1,20 @@
+# Mounting external storage on service volumes
+
+This feature enables the mounting of a folder from a storage provider, such as MinIO or dCache, into the service container. As illustrated in the following diagram, the folder is placed inside the /mnt directory on the container volume, thereby making it accessible to the service. This functionality can be utilized with exposed services, such as those using a Jupyter Notebook, to make the content of the storage bucket accessible directly within the Notebook.
+
+![mount-diagram](images/mount.png)
+
+As OSCAR has the credentials of the default MinIO instance internally, if you want to use a different one or a different storage provider, you need to set these credentials on the service [FDL](/fdl). Currently, the storage providers supported on this functionality are:
+
+ - [MinIO provider](/fdl/#minioprovider)
+ - [WebDav provider](/fdl/#webdavprovider)
+
+Let's explore these with an FDL example:
+
+```
+mount:
+  storage_provider: minio.default
+  path: /body-pose-detection-async
+```
+
+The example above means that OSCAR mounts the `body-pose-detection-async` bucket of the default MinIO inside the OSCAR services. So, the content of the `body-pose-detection-async` bucket will be found in `/mnt/body-pose-detection-async` folder inside the execution of OSCAR services.

examples/dog-breed-detector/dog-breed.md

@@ -0,0 +1,33 @@
+# Dog breed detector module
+
+This example uses a pre-trained classification model, which can be found on the DeePaaS marketplace [link]. This example uses a generic script implemented to work with all models that have installed the deepaas-cli command line interface with a version <=2.3.1 instead of a specifically defined script. Moreover, it can be used both with asynchronous and synchronous invocations.
+
+The steps to use this are the same as other examples, first you need to create the service, either using oscar-cli with the following command
+
+``` sh
+oscar-cli apply dog-breed-detector.yaml
+```
+
+or through the graphical interface of your cluster.
+
+Usually, DeePaaS models need some given parameters to be defined alongside the input of the inference invocation. To solve this, the service receives a JSON type in the following format where you can define, on the one hand, the key of the JSON the name and value of the parameter to be used on the command line and the other hand, inside the array 'oscar-files' each of the files codified as a base64 string, and the extension of it.
+
+``` json
+{
+    'network': 'Resnet50',
+    'oscar-files': [
+        {
+            'key': 'files',
+            'file_format': 'jpg',
+            'data': [BASE_64_ENCODED_FILE],
+        }
+    ]
+}
+```
+As you can see, this example needs to set the parameter `network` and receives a `jpg` file. So, to invoke the service synchronous you should make a call like the following 
+
+``` sh
+oscar-cli service run dog-breed-detector --input inputparams.json
+```
+
+or uploading the file asynchronous to the MinIO bucket.

examples/dog-breed-detector/dog-breed.yaml

@@ -0,0 +1,17 @@
+functions:
+  oscar:
+  - oscar-cluster:
+      name: dog-breed-detector
+      memory: 2Gi
+      cpu: '1.0'
+      image: ai4oshub/dogs-breed-detector
+      log_level: CRITICAL
+      vo: [vo of the cluster] # If you are using clusters with EGI authentication
+      script: script.sh
+      allowed_users: [] # Public service
+      input:
+      - storage_provider: minio.default
+        path: dogbreed/input
+      output:
+      - storage_provider: minio.default
+        path: dogbreed/output

examples/dog-breed-detector/script.sh

@@ -0,0 +1,67 @@
+python - << EOF
+import json
+import base64
+import string
+import random
+import os
+from subprocess import run, PIPE
+
+# Read input file
+FILE_PATH=os.getenv("INPUT_FILE_PATH")
+if [[ "$FILE_PATH" != *.json ]]; then
+	run(["mv", "$INPUT_FILE_PATH", "$INPUT_FILE_PATH.json"])
+	FILE_PATH="$FILE_PATH"+".json"
+fi
+
+DEEPAAS_CLI_VERSION=[2,3,1]
+STR_VERSION='2.3.1'
+DEEPAAS_CLI_VCOMMAND=['deepaas-cli', '--version']
+
+DEEPAAS_CLI_COMMAND=['deepaas-cli', 'predict']
+OSCAR_FILES="oscar-files"
+
+def check_deepaas_version():
+    v = run(DEEPAAS_CLI_VCOMMAND, stdout=PIPE)
+    version = v.stdout.decode("utf-8").strip('\n').split(".")
+    if len(version) > 3: version.pop()
+    for i, ver in enumerate(version):
+        if int(ver) < DEEPAAS_CLI_VERSION[i]:
+            print(f"Error: 'deepaas-cli' version must be >={STR_VERSION}. Current version is: {print_version(version)}")
+            exit(1)
+
+def print_version(version):
+    return '.'.join(version)
+
+def add_arg(key, value):
+   DEEPAAS_CLI_COMMAND.append("--"+key)
+   DEEPAAS_CLI_COMMAND.append(value)
+
+def decode_b64(filename, data):
+    with open(filename, "wb") as f:
+        f.write(base64.b64decode(data))
+
+def parse_files(files):
+        for file in files:
+                rnd_str = ''.join(random.choice(string.ascii_lowercase) for i in range(5))
+                filename=''.join(["tmp-file-", rnd_str, ".", file["file_format"]])
+                add_arg(file["key"], filename)
+                decode_b64(filename, file["data"])
+
+# Check the deepaas-cli version
+check_deepaas_version()
+
+# Process input
+with open(FILE_PATH, "r") as f:
+ params = json.loads(f.read())
+
+for k, v in params.items():
+    # If param is 'oscar-files' decode the array of files
+    if k == "oscar-files":
+       parse_files(v)
+    else:
+        if isinstance(v, int): 
+            v = str(v)
+        add_arg(k, v)
+run(DEEPAAS_CLI_COMMAND)
+
+EOF

mkdocs.yml

@@ -40,6 +40,8 @@ nav:
 
 - Advanced Features:   
   - MinIO bucket replication: minio-bucket-replication.md
+  - Mount external volumes: mount.md
+  - Additional configuration: additional-config.md
 
 - Development:
   - Documentation: devel-docs.md