Refactor queue drivers

[sondreso]

Compute resource specifications:
Compute resource specifications should be model specific, and and we should translate the requirements to the appropriate settings for the currently used queue system.

In particular, the number of CPU cores and memory required is inherit to the model, not the specific queue system that is configured. We should translate these requirements/configurations to the specific queue system when submitting the jobs, allowing the queue system to make decisions about how to best distribute the jobs over the available compute resources. (This means that the Torque setting NUM_CPUS_PER_NODE should be deprecated and removed, as it attempts to do this resource allocation manually and leave it up to the user).

Improved status updating and tracking:
Currently every job (JobQueueNode) will interact with the queue system on its own, updating and tracking the status of a single job submitted to the queue system (It creates one thread per realizations, each doing its own polling). This scales poorly when increasing the number of realizations, to the point where we will ddos the queue system when asking for statuses. Mitigating actions exist both in the Torque driver (qstat_proxy.sh to cache the result of the command) and the LSF driver (Internal lookup table (caching) in the driver itself).

A better solution here would be to lift the abstraction to a sightly higher level, such that the "Queue manager" can query the status of all the jobs simultaneously. Something along the lines of this:

def get_torque_status(jobs: List[JobId]) -> Dict[JobId, JobStatus]:
    status_str = subprocess.call("qstat", qstat_args)
    statuses = parse_statuses(status_str)
    return {jobid: statuses[jobid] for jobid in jobs}

Move code to Python:
We should move the current C-code implementation to Python.

NOTE:
The queue driver is performance critical to ERT, and at no point during refactoring we should exceed a performance limit such that ERT will stop working. This means that extra testing to ensure scaling when increasing the number of realizations is needed.

List of issues:

We should make sure that all the new code comes with a benchmarking / performance tests.

---

Definition of Done:

• CPU and memory requirements can be specified per model and will be passed to all relevant queue systems. #8198
• No queue related code left in C
• No branching and subsequent caching when fetching the status of a job submitted to the cluster.
• Run asset model with 200 realizations (LSF, ESMDA) with memory options specified and make a report
• Run asset model with 1000 realizations (LSF, ES) with memory options specified and make a report
• equinor/scout#1045
• https://github.com/equinor/komodo-releases/issues/4884
• https://github.com/equinor/komodo-releases/issues/5126
• Write a summary of performance between running JobQueue and Scheduler on bigpoly #7342
• Test SLURM driver in production environment - get in touch with users
• equinor/everest#2168
• Make sure the queue documentation is up to date
• Make sure that all the drivers have substantial test coverage for both happy and sad paths

---

Migration to Python

Moving Scheduler to Main

• Make Scheduler default queue for LOCAL driver #6755
• Document the queue driver components currently in use #6505
• Make JobQueue and JobQueueNode implementation purely in Python #6506
• Make local queue handle realizations that crash #6674
• Implement and make Python LSF driver handle crashes #6590
• Implement support for MAX_RUNNING in local and LSF queue
• Implement support for MAX_RUNTIME in local and LSF queue driver
• Remove C source code from PR (and clean up CMakeLists) #6673
• Remove the JobStatus enum class, initialize the states directly. We probably don't need JobStatus class.
• Add additional unit tests for validation of the LocalDriver #6679
• Implement abstract driver class in Python
• Rewrite queue_options as dict instead of tuple #6724
• Are transitions in StateMachine queued?
• Mark failing tests with @pytest.mark.skip(reason="Needs reimplementation")
• Fix test_batch_sim.py
• Implement run_exit_callback
• Implement temporary framework to avoid queue system feature branch #6681
• test_queue_config either refactor or remove the tests. Validate through config_dict_generate if it is tested already #6716
• test_driver either refactor or remove the tests. #6715
• test_lsf_driver - fix test_run_mocked_lsf_queue test #6721
• test_torque_driver - fix skipped tests #6753
• Implement stop_long_running_jobs in the Scheduler #6710
• test_file_report - fix skipped tests #6696
• test_batch_sim - fix skipped tests (split into more issues if not straightforward) #6709
• test_simulation_context - fix skipped tests #6702
• test_tracking_integration - fix skipped tests #6694
• Move RETRY driver (submit & kill) functionality into the Scheduler #6708
• Rename JobQueue to Scheduler #6692
• test_job_queue - fix failing tests #6725

drivers

• Implement TORQUE driver class in Python #6675
• Implement SLURM driver class in Python #6676
• Implement LOCAL driver class in Python #6677
• Implement LSF driver class in Python #6678

---

Dask related:

• Prototype: Implement JobQueue using dask #6498
• Investigate whether we can use Dask to handle all drivers directly #6328
• Explore dask dashboard when running different queues #6497
• Explore whether dask and ert queue options are compatible #6500
• Explore dask communication layer tls providing encrypted communication #6534

Other Issues:

• Cleanup driver and job_queue with the unused code #6342
• Investigate usability of Linux C-groups #6295
• Study API for possibilities of individual drivers #6331
• Investigate usability of Prefect as JobQueue #6332
• Organize a session / demo of interacting with queues (LSF, TORQUE, SLURM) #6333
• Explore whether we can remove job_queue/RunStatus #6359
• Improve logging for started jobs on the cluster #6383
• Move status polling into JobQueue #6291
• Do status polling in Python #6503
• Split drivers code into Python and C #6504
• Add integration test to run polycase on SLURM #6296
• Implement TORQUE API for slurm for test code #6329
• Reimplement local queue in Python #6470
• Replace Cwrap with Pybind (Might not be needed in the end)
• equinor/scout#1054
• Remove Stage, Step and Job from ensemble_evaluator._builder #6218
• Investigate whether we can just use jobs.json as an input to the queue and let the driver handle execution automatically. It does not seem to be relevant any more as we rely jobs.json still.
• stderr from job_dispatch is lost when runpath is missing #3502
• Include a new SLURM QUEUE_OPTION (QUEUE_OPTION SLURM ACCOUNT xxx) #4357
• Failed conversion of queue driver option values are silently ignored #6391
• Job submitted to LSF are not taken down properly when using Terminate experiment #5955
• Reverse behaviour for KEEP_QSUB_OUTPUT #6437
• Remove "-x" from default qstat options #6435
• NUM_CPUS_PR_NODE vs FORWARD_MODEL specifying cpus #6054
• Different memory notations on different queues #6517
• Use pydentic model for QueueConfig instead of raw dict
• After clicking Kill realizations realizations with state Waiting are submitted before being stopped. #6134