Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce hard/soft limits for commiting run and run tool/pipeline #3573

Open
SilinPavel opened this issue Jun 26, 2024 · 1 comment
Open
Labels
kind/enhancement New feature or request state/has-doc Issues that have documentation

Comments

@SilinPavel
Copy link
Member

SilinPavel commented Jun 26, 2024

Background
Docker images can be extremely big, so it is nice to have a mechanism to warn/reject user from creation or running such big images.

Approach

  1. Let's introduce 2 new System preference - commit.container.size.limits, launch.tool.size.limits (Object) with structure:

    {
      "soft": <size value in bytes, long>,
      "hard": <size value in bytes, long>
    }
    

    Provide validation for the values of the properties`

  2. Modify a method in PipelineRunController
    GET /run/{runId}/commit/check to provide more extended result like:

    {
        "enoughSpace": {
           "status": <ENUM (FAIL, WARN, OK)>,
           "message": <can be null, message in case of warning or fail>,
           "value": <raw value of the metric, for enoughSpace true/false>
        },
        "containerSize":  {
            "status": <ENUM (FAIL, WARN, OK)>,
            "message": <can be null, message in case of warning or fail>,
            "value": <raw value of the metric, for containerSize size in bytes>
        }
    }
    

    To do so, let's introduce new server method in DockerContainerOperationManager to be able to ask node which is host the run, current size of the run container.

    • This method should work similar as commit procedure (ssh to the node) and utilize command docker inspect ${CONTAINER_ID} -s --format "{{.SizeRw}}"
    • Timeout should be implemented for the method to not to hang much time
    • If something goes wrong (timeout or error) return -1 as containerSize
  3. On UI:

    • During commit change logic to verify not only enoughSpace but also check that containerSize check status
      • If WARN:
        • Warn user that container image will be too big, but let to commit if needed
      • If FAIL
        • Restrict to commit
    • During Tool/Pipeline run, check if tool version size < system preference launch.tool.size.limits soft/hard
      • If bigger:
        • For soft : Warn user that container image is too big, but let to run if needed
        • For hard : Restrict to run
@SilinPavel SilinPavel added the kind/enhancement New feature or request label Jun 26, 2024
NShaforostov added a commit that referenced this issue Sep 2, 2024
* (Issue #3619) 'Runs archiving' doc
* (Issue #3573) 'Container limits' doc
* (Issue #3568) 'Compose a Dockerfile' doc
* (Issue #3576) 'GPU statistics monitor' doc
* (Issue #3602) 'Pod network consumption alert and restriction' doc
@NShaforostov
Copy link
Collaborator

Docs added via #3669 and located here and here.

@NShaforostov NShaforostov added the state/has-doc Issues that have documentation label Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement New feature or request state/has-doc Issues that have documentation
Projects
None yet
Development

No branches or pull requests

2 participants