Discussion: Enhancing Docker Image Usability for Cookbook Development

[mosoriob]

Motivation

Creating or improving a cookbook requires several testing steps: modifying the LLM repository, pushing the changes, GitHub building the image, updating the cookbook app, creating a new job, and testing. In my opinion, these are too many steps. The number of steps is high due to the current approach selected in the Docker images.

Context

Our current approach starts a Docker container that installs Conda and other packages on the user's account. This method allows users to resume their sessions over time, as the packages remain in their account.

However, this approach has several disadvantages:

1. Limited Testing Environment: The paths depend on the TACC system, which means we cannot test the image on a local computer or with GitHub Actions. This limitation adds queue waiting time.
2. Reproducibility Issues: Each user account is unique. If a user's account is not clean (i.e., they have previously installed packages), it can lead to inconsistencies and potential issues.

Proposed Complementary Approach

We propose a complementary approach: creating Docker images with all necessary packages (Conda and system packages) pre-installed. This approach involves the following steps:

1. Modify the repository
2. Push the changes
3. GitHub builds the image
4. Run the image on your local computer

Note that if a user installs a package during a session, it will not be available in subsequent sessions. Users will need to add any required packages to the repository package list.

[mepearson]

Fully support.

[wmobley]

How does get around the unique port forwarding issues of tacc?

[mosoriob]

The run.sh file will run only for tapis jobs.

IF IS RUNNING ON TACC:
    # run port forwarding