(Issue #3573) Container limits doc and RN

docs/md/manual/10_Manage_Tools/10.4._Edit_a_Tool.md

@@ -4,6 +4,9 @@
 - [Edit a Tool version](#edit-a-tool-version)
     - [Run/Delete a Tool version](#rundelete-a-tool-version)
     - [Commit a Tool](#commit-a-tool)
+        - [Committing features](#committing-features)
+            - [Pre/post-commit hooks](#prepost-commit-hooks)
+            - [Container size limits](#container-size-limits)
 - [Edit a Tool settings](#edit-a-tool-settings)
 - [Delete a Tool](#delete-a-tool)
 
@@ -59,6 +62,8 @@ When it is complete COMMITING status on the right side of the screen will change
 
 #### Committing features
 
+##### Pre/post-commit hooks
+
 In certain use-cases, extra steps shall be executed before/after running the commit command in the container. For example, to avoid warning messages about terminating the previous session (which was committed) of the tool application in a non-graceful manner. Some applications may require extra cleanup to be performed before the termination.
 
 To workaround such issues in the **Cloud Pipeline** an approach of "**pre/post-commit hooks**" is implemented. That allows to perform some graceful cleanup/restore before/after performing the commit itself.
@@ -97,6 +102,38 @@ Consider an example with **RStudio** tool, that **Cloud Pipeline** provides "out
     - (**2**) post-commit script is found at the specified path in the docker image - and it is being executed
     - (**3**) post-commit script was performed successfully
 
+##### Container size limits
+
+Docker images can be extremely large. Therefore, **Cloud Pipeline** supports a mechanism to warn/reject users from creation of such big images.
+
+To configure that mechanism, there is the special [system preference](../12_Manage_Settings/12.10._Manage_system-level_settings.md#commit) - **`commit.container.size.limits`**.  
+This preference has a format:
+
+```
+{
+    "soft": <soft_limit_size>,
+    "hard": <hard_limit_size>
+}
+```
+
+This preference defines "**soft**" and "**hard**" limits for a container size in bytes:
+
+- if user tries to perform a commit operation and the container size exceeds "soft" limit - user will get warning notification, but can proceed the commit at their own risk
+- if the container size exceeds "hard" limit - commit operation in this case will be unavailable
+- if the size of any limit is set as `0` - this means there is no limitation of that type
+
+Example of the preference configuration:  
+    ![CP_EditTool](attachments/EditTool_25.png)  
+In this example, "**soft**" limit is set as 1 GB and there is no "**hard**" limit.
+
+In case, when "**soft**" limit is set more than `0` and user tries to commit some tool which container exceeds this limit, the following warning will appear:  
+    ![CP_EditTool](attachments/EditTool_26.png)  
+At the same time, commit operation is available.
+
+In case, when "**hard**" limit is set more than `0` and user tries to commit some tool which container exceeds this limit, the following error will appear:  
+    ![CP_EditTool](attachments/EditTool_27.png)  
+Commit operation is unavailable.
+
 ## Edit a Tool settings
 
 Settings in this tab are applied to all Tool versions (i.e. these settings will be a default for all Tool version).

docs/md/manual/10_Manage_Tools/10.5._Launch_a_Tool.md

@@ -5,6 +5,8 @@
 - [Launch a Tool with "friendly" URL](#launch-a-tool-with-friendly-url)
 - [Launch a tool with "hosted" applications](#launch-a-tool-with-hosted-applications)
 - [Instance management](#instance-management)
+- [Restrictions](#restrictions)
+    - [Container size limits](#container-size-limits)
 
 > To launch a Tool you need to have **EXECUTE** permissions for it. For more information see [13. Permissions](../13_Permissions/13._Permissions.md).
 >
@@ -204,3 +206,37 @@ In CP platform next hierarchy is set for applying of inputted allowed instance t
 - Tool level (specified for a tool on "Instance management" panel) (see [above](#instance-management))
 - _(global)_ **`cluster.allowed.instance.types.docker`** (specified on "Cluster" tab in "Preferences" section of system-level settings) (see [v.0.14 - 12.10. Manage system-level settings](../12_Manage_Settings/12.10._Manage_system-level_settings.md#cluster))
 - _(global)_ **`cluster.allowed.instance.types`** (specified on "Cluster" tab in "Preferences" section of system-level settings) (see [v.0.14 - 12.10. Manage system-level settings](../12_Manage_Settings/12.10._Manage_system-level_settings.md#cluster))
+
+## Restrictions
+
+### Container size limits
+
+Docker images can be extremely large. Therefore, **Cloud Pipeline** supports a mechanism to warn/reject users from launching of such big images.
+
+To configure that mechanism, there is the special [system preference](../12_Manage_Settings/12.10._Manage_system-level_settings.md#launch) - **`launch.tool.size.limits`**.  
+This preference has a format:
+
+```
+{
+    "soft": <soft_limit_size>,
+    "hard": <hard_limit_size>
+}
+```
+
+This preference defines "**soft**" and "**hard**" limits for a Docker container size in bytes:
+
+- if user tries to launch a tool and the size of its container exceeds "soft" limit - user will get warning notification, but can proceed the tool launch at their own risk
+- if the size of a tool container exceeds "hard" limit - tool launch in this case will be unavailable
+- if the size of any limit is set as `0` - this means there is no limitation of that type
+
+Example of the preference configuration:  
+    ![CP_LaunchTool](attachments/LaunchTool_46.png)  
+In this example, there is no "**soft**" limit and "**hard**" limit is set as 1 GB.
+
+In case, when "**soft**" limit is set more than `0` and user tries to launch some tool which container exceeds this limit, the following warning will appear:  
+    ![CP_LaunchTool](attachments/LaunchTool_47.png)  
+At the same time, the launch operation is still available.
+
+In case, when "**hard**" limit is set more than `0` and user tries to launch some tool which container exceeds this limit, the following error will appear:  
+    ![CP_LaunchTool](attachments/LaunchTool_48.png)  
+Tool launch is unavailable.

docs/md/manual/12_Manage_Settings/12.10._Manage_system-level_settings.md

@@ -19,6 +19,7 @@
     - [Launch](#launch)
     - [Lustre FS](#lustre-fs)
     - [Miscellaneous](#miscellaneous)
+    - [Monitoring](#monitoring)
     - [Search](#search)
     - [System](#system)
     - [User Interface](#user-interface)
@@ -118,6 +119,7 @@ This tab contains various commit settings:
 | **`commit.deploy.key`** | Used to SSH for COMMIT. Key is stored in a DB |
 | **`commit.timeout`** | Commit will fail if exceeded (in seconds) |
 | **`commit.post.command.path`** | Specifies a path to a script within a docker image, that will be executed in a committed image, after docker commit occurs |
+| **`commit.container.size.limits`** | Defines "soft" and "hard" limits for a container size in bytes.<br>If during the commit operation container size exceeds "soft" limit, user will get warning notification.<br>If container size exceeds "hard" limit, commit operation for such tool will be unavailable |
 | **`pause.timeout`** |  |
 
 ### DTS
@@ -279,7 +281,7 @@ Settings in this tab contains default Launch parameters:
 | **`launch.container.cpu.resource`** |  |
 | **`launch.container.memory.resource.policy`** |  |
 | **`launch.container.memory.resource.request`** |  |
-| **`launch.insufficient.capacity.message`** | Defines the text displayed in the run logs in case when an instance requested by the user is missing due to `InsufficientInstanceCapacity` error (means insufficient capacity in the selected Cloud Region) |
+| **`launch.insufficient.capacity.message`** | Defines the text displayed in the run logs in case when an instance requested by the user is missing due to `InsufficientInstanceCapacity` error<br>(means insufficient capacity in the selected Cloud Region) |
 | **`launch.run.visibility`** | Allow to view foreign runs based on pipeline permissions (value `INHERIT`) or restrict visibility of all non-owner runs (value `OWNER`) |
 | **`launch.dind.enable`** | Enables Docker in Docker functionality |
 | **`launch.dind.container.vars`** | Allows to specify the variables, which will be passed to the DIND container (if they are set for the host environment) |
@@ -291,6 +293,7 @@ Settings in this tab contains default Launch parameters:
 | **`launch.serverless.stop.timeout`** |  |
 | **`launch.serverless.wait.count`** |  |
 | **`launch.system.parameters`** | System parameters, that are used when launching pipelines |
+| **`launch.tool.size.limits`** | Defines "soft" and "hard" limits for a container size in bytes.<br>If tool container size exceeds "soft" limit, user will get warning notification when trying to launch such a tool.<br>If container size exceeds "hard" limit, launch of such a tool will be prohibited |
 
 ### Lustre FS
 
@@ -312,6 +315,13 @@ These settings define Lustre FS parameters:
 | **`misc.max.tool.icon.size.kb`** | Sets maximum size (in Kb) of the uploaded icon for the tool |
 | **`system.events.confirmation.metadata.key`** | Sets the **KEY** for the user's attribute displaying information about "blocking" notifications confirmation |
 
+### Monitoring
+
+| Setting name | Description |
+|---|---|
+| **`monitoring.archive.runs.enable`** | Enables [archiving mechanism](../11_Manage_Runs/11.5._Archive_runs.md) for completed runs |
+| **`monitoring.archive.runs.delay`** | Manipulates the frequency of archiving operations (in milliseconds).<br>Operations of runs archiving are asynchronous and performed after each period of time specified via this preference |
+
 ### Search
 
 Settings in this tab contains Elasticsearch parameters:
@@ -338,9 +348,12 @@ The settings in this tab contain parameters and actions that are performed depen
 
 | Setting name | Description |
 |---|---|
+| **`system.archive.run.metadata.key`** | Defines the name of the metadata key that shall be specified for users/groups which runs shall be [archived](../11_Manage_Runs/11.5._Archive_runs.md) |
+| **`system.archive.run.owners.chunk.size`** | Defines the chunk size by which the data of completed runs to archive shall be divided. Data is divided by users count (runs owners count) |
+| **`system.archive.run.runs.chunk.size`** | Defines the chunk size by which the data of completed runs to archive shall be divided. Data is divided by master runs count |
 | **`system.ssh.default.root.user.enabled`** | Enables launch of the SSH terminal from the run under `root`-user access |
-| **`system.max.idle.timeout.minutes`** | Specifies a duration in minutes. If CPU utilization is below **`system.idle.cpu.threshold`** for this duration - notification will be sent to the user and the corresponding run will be marked by the "IDLE" label |
-| **`system.idle.action.timeout.minutes`** | Specifies a duration in minutes. If CPU utilization is below **`system.idle.cpu.threshold`** for this duration - an action, specified in **`system.idle.action`** will be performed |
+| **`system.max.idle.timeout.minutes`** | Specifies a duration in minutes.<br>If CPU utilization is below **`system.idle.cpu.threshold`** for this duration - notification will be sent to the user and the corresponding run will be marked by the "IDLE" label |
+| **`system.idle.action.timeout.minutes`** | Specifies a duration in minutes.<br>If CPU utilization is below **`system.idle.cpu.threshold`** for this duration - an action, specified in **`system.idle.action`** will be performed |
 | **`system.resource.monitoring.period`** | Specifies period (in seconds) between the users' instances scanning to collect the monitoring metrics |
 | **`system.monitoring.time.range`** | Specifies time delay (in sec) after which the notification "HIGH\_CONSUMED\_RESOURCES" would be sent again, if the problem is still in place |
 | **`system.disk.consume.threshold`** | Specifies disk threshold (in %) above which the notification "HIGH\_CONSUMED\_RESOURCES" would be sent and the corresponding run will be marked by the "PRESSURE" label |

docs/md/release_notes/v.0.15/v.0.15_-_Release_notes.md

@@ -288,7 +288,7 @@ Two preferences are introduced:
     - This hook can be used to perform any filesystem cleanup or other operations, that shall not affect the currently running processes.
 - If a corresponding pre/post script is not found in the docker image - it will not be executed.
 
-For more details see [here](../../manual/10_Manage_Tools/10.4._Edit_a_Tool.md#committing-features).
+For more details see [here](../../manual/10_Manage_Tools/10.4._Edit_a_Tool.md#prepost-commit-hooks).
 
 ## Restricting manual installation of the nvidia tools
 

docs/md/release_notes/v.0.17/v.0.17_-_Release_notes.md

@@ -44,6 +44,7 @@
 - [Cluster run usage](#cluster-run-usage)
 - [Cluster run estimation price](#cluster-run-estimation-price)
 - [Terminal view](#terminal-view)
+- [Container limits](#container-limits-for-commitlaunch-tool-operations)
 - [AWS: seamless authentication](#aws-seamless-authentication)
 - [AWS: transfer objects between AWS regions](#aws-transfer-objects-between-aws-regions-using-pipe-storage-cpmv-commands)
 - [AWS: switching of regions for launched jobs in case of insufficient capacity](#aws-switching-of-cloud-regions-for-launched-jobs-in-case-of-insufficient-capacity)
@@ -1571,6 +1572,25 @@ Required color schema can be configured in two ways:
 
 For details see [here](../../manual/15_Interactive_services/15.2._Using_Terminal_access.md#terminal-view).
 
+## Container limits for commit/launch tool operations
+
+Docker images can be extremely large.  
+Therefore in the current version, a mechanism to warn/reject users from using of too big images was implemented.
+
+There are new system preferences **`commit.container.size.limits`** and **`launch.tool.size.limits`** to manage that behavior:  
+    ![CP_v.0.17_ReleaseNotes](attachments/RN017_ContainerSize_1.png)
+
+Both preferences have the same format and define "**soft**" and "**hard**" limits for a container size in bytes:
+
+- **`commit.container.size.limits`** defines limits of the docker container that are taken into account during the tool commit operation:  
+    - if container size exceeds "soft" limit - user will get warning, but can proceed the commit at their own risk
+    - if the container size exceeds "hard" limit - tool commit operation will be prohibited
+- **`launch.tool.size.limits`** defines limits of the docker container that are taken into account during the tool launch operation:  
+    - if container size exceeds "soft" limit - user will get warning, but can launch the tool at their own risk
+    - if the container size exceeds "hard" limit - tool launch will be prohibited
+
+For more details see sections in [Tool commit](../../manual/10_Manage_Tools/10.4._Edit_a_Tool.md#container-size-limits) and [Tool launch](../../manual/10_Manage_Tools/10.5._Launch_a_Tool.md#container-size-limits).
+
 ## AWS: seamless authentication
 
 In some cases, users are faced with the following scenarios: