Implement AiiDA control page

[yakutovicha]

fixes #155

[cpignedoli]

User-Friendly AiiDAlab App for Managing Main Workflows (MWs)

This AiiDAlab app aims to enhance the user experience when working with Main Workflows (MWs) in AiiDA. Currently, the default "verdi process list -a" command provides an overview of all workflows, but it lacks user-centric features and can be cumbersome to navigate, especially when dealing with numerous MWs.

MW is a workflow submitted by a user, usually with a description, that will typically have child workflows

Key Features:

1. Intuitive MW List: The app will present a user-friendly MW list, displaying the descriptions of each workflow along with their statuses. Users can quickly identify MWs, such as "PDOS DBBR on Au," "STM goblet," etc., based on the provided descriptions rather than the AiiDA workchain names.

2. Workflow Progress Visualization: Users can easily track the progress of a specific MW by clicking on it. The app will visually highlight completed steps and indicate which step is currently running, queued, or paused. This provides users with a clear understanding of where their MW stands in terms of execution.

3. Process Control: The app will empower users to interact with their MWs efficiently. Users can pause and resume processes as needed, allowing better control over long-running or resource-intensive calculations. Additionally, the app will provide a "kill" option to terminate unwanted or problematic MWs.

4. Efficient Workflow Retrieval: Searching for past MWs will be a breeze with the app's convenient filtering and sorting options. Users can effortlessly find a specific MW, like "DBBR Br removed on Au" from two months ago, and access its individual steps with ease. The app will allow searching based on date ranges and provide a smooth scrolling experience, avoiding disruptive automatic page refreshes.

5. Optimized Performance: Ensuring the app's responsiveness and speed is a priority. By optimizing database queries and utilizing efficient data retrieval techniques, we aim to create a seamless experience for users, even when dealing with a significant number of MWs.

The goal of this app is to make MW management in AiiDA more intuitive, efficient, and enjoyable for users who frequently submit and monitor complex workflows. Your feedback and contributions are highly appreciated as we strive to create a powerful tool that complements AiiDA's capabilities.

[unkcpz]

The control page dependent on the aiidalab-widgets-base, uses widgets from AWB. It is still an open question whether AWB should be the dependency of aiidalab-home.
A problem I can foresee is since app share the environment with all kernel packages, install/update app will potentially break the control page.

We can also make AWB the core package as defined in aiidalab and prevent user to override with incompatible version. But this brings restriction to app developers.

The ultimate solution could be for every app has its own Python environment created when using aiidalab install by create an independent conda environment. I had played with this idea a bit but stuck at the jupyter kernel as well as the python interpereter's should be set into notebook metadata in advance which makes no sense for app developement. (I was thinking that the kernel can be set such as passing kernel name to URL, but didn't find anything solution like this.)

[danielhollas]

@yakutovicha thanks for this. 🗜️ I like the overall approach and design. I would suggest two things so we can move with an initial version of the control page:

1. Vendor in AWB dependencies. This will allow faster iteration on the design, and in the meantime we can figure out whether / how to add AWB to base image.
2. It is clear that the Process page will require the most discussion. Therefore I would omit it in the first version, so that it can be discussed separately and we can have something basic in the meantime.

Couple minor things / ideas...

1. Perhaps it would make sense to merge the Status and Daemon pages? I would imagine most of the time I'd like to see both at the same time to get the overall state of the system and be able to control it.
2. Provide buttons to increase / decrease number of daemon workers
3. Daemon page: Start / Stop buttons should be deactivated when the daemon is (not) running.
4. Profile page: "Make default" button should not be active for default profile.
5. Get rid of the alert prompt when exiting the page using the same trick as in other pages.

[danielhollas]

It would be great to re-enable the tests in CI before this PR is merged, per #161, and add basic tests for the new functionality.

[danielhollas]

I've merged in main with updated ruff that now checks jupyter notebooks. It reports a lint violation here:

control.ipynb:cell 3:19:25: B006 Do not use mutable data structures for argument defaults
   |
19 | def update_output(value={"new": "Group"}):
   |                         ^^^^^^^^^^^^^^^^ B006
20 |     if value["new"] in correspondance:
21 |         with output:
   |
   = help: Replace with `None`; initialize within function

[danielhollas]

@yakutovicha btw: I am going to be on the coding retreat next month and I've heard you'll be there as well? Perhaps this PR is a good candidate to work on, WDYT?

[danielhollas]

The control page dependent on the aiidalab-widgets-base, uses widgets from AWB. It is still an open question whether AWB should be the dependency of aiidalab-home.
A problem I can foresee is since app share the environment with all kernel packages, install/update app will potentially break the control page.

Just to continue this discussion, I think aiidalab-home cannot depend on AWB. All the apps (including home app) currently share a python environment, which means only one AWB version at a time. But we need to allow application to install different versions of AWB according to their needs, which will always run a risk of breaking home app. Thus I don't see any other way.

If there are components that are currently needed in AWB and needed here, I think the simplest thing is to copy them. I think in general we don't want to expand the home app too much, so the number of such components should not be high.

@edan-bainglass @superstar54 FYI about the status of this PR, please have a look at my earlier comment. #156 (comment)

I don't think it would be too much work to finish this PR in some minimal form and keep iterating on it in subsequent PRs.

[edan-bainglass]

@edan-bainglass @superstar54 FYI about the status of this PR, please have a look at my earlier comment. #156 (comment)

I don't think it would be too much work to finish this PR in some minimal form and keep iterating on it in subsequent PRs.

Thanks @danielhollas. @yakutovicha can we set up a meeting soon (maybe before the break) to discuss the current status and define a roadmap to a minimal release? All are welcome to join.

[yakutovicha]

Thanks @danielhollas. @yakutovicha can we set up a meeting soon (maybe before the break) to discuss the current status and define a roadmap to a minimal release? All are welcome to join.

I'm happy to meet, but frankly, the only time that might work for me is this Wed between 16 and 17. All the remaining days are booked.