Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Job manager routes #296

Merged
merged 37 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
673769d
job_manager --> jobs
ryuwd Sep 16, 2024
6b4895d
split jobs router into multiple jobs router modules; simplify status …
ryuwd Sep 16, 2024
5b26ffa
Get result correctly from tasks
ryuwd Sep 16, 2024
305b3bc
Moved to avoid clashing wrong route being matched
ryuwd Sep 16, 2024
3f8f8fe
Fixed some tests
ryuwd Sep 16, 2024
f959cdc
Update doc for remove bulk
ryuwd Sep 17, 2024
030fdf1
Rescheduling including TODO regarding job state machine bug (check th…
ryuwd Sep 17, 2024
420c138
Major refactoring of job rescheduling
ryuwd Sep 17, 2024
e99b3d1
Make reset_jobs optional
ryuwd Sep 17, 2024
e278df3
Improved job rescheduling and the test
ryuwd Sep 21, 2024
1071b7e
Add missing MaxRescheduling to with_config_repo
ryuwd Sep 21, 2024
8286072
Start refactoring things to use generally less queries (start with re…
ryuwd Sep 22, 2024
37adf07
Evolve set_job_status to do bulk operations
ryuwd Sep 23, 2024
e3b82cb
refactored job submission to reduce the number of statements executed…
ryuwd Sep 30, 2024
09dbdb8
use _bulk function
ryuwd Dec 13, 2024
a53ba2c
Fixing tests
ryuwd Dec 14, 2024
465b238
assume check_permissions does the right thing
ryuwd Dec 14, 2024
59d087f
JobException-->JobError
ryuwd Dec 14, 2024
0a15146
remove dependencies from DiracxRouter inst
ryuwd Dec 14, 2024
e790989
more fun with tests
ryuwd Dec 15, 2024
5871b2e
update cs for test and new MaxRescheduling location under JobScheduli…
ryuwd Dec 16, 2024
ad7e5e8
Cleanup insert_bulk - use taskgroups, and single row inserts for job …
ryuwd Dec 16, 2024
1fb0f3a
overwrite the correct method ...
ryuwd Dec 16, 2024
97d37c3
Fixed date truncation to avoid escaping issues in datetime formatting
ryuwd Dec 16, 2024
76e0fc3
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 16, 2024
83dad27
Remove stray comment
ryuwd Dec 16, 2024
4c0346d
Regenerate diracx client
ryuwd Dec 16, 2024
80388dc
Moved non-DB related job submission logic out of diracx-db to diracx-…
ryuwd Dec 17, 2024
f0291b0
Move job status functions from diracx-db to diracx-routers
ryuwd Dec 17, 2024
9b273c9
This is less than ideal
ryuwd Dec 17, 2024
1625a40
Skip tests.
ryuwd Dec 17, 2024
4df7c99
refix tests
ryuwd Dec 17, 2024
8016b6c
We agreed to move these functions back into diracx-db utils module
ryuwd Dec 17, 2024
828e8c2
Gubbins hiccups
ryuwd Dec 17, 2024
da1b717
cast to list
ryuwd Dec 17, 2024
f34c957
nearly..
ryuwd Dec 17, 2024
57e610c
I affirm that the tests will pass
ryuwd Dec 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions diracx-cli/tests/legacy/cs_sync/integration_test.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -1280,6 +1280,7 @@ Systems
{
#@@-prod - /C=ch/O=DIRAC/OU=DIRAC CI/CN=ciuser - 2023-10-02 12:36:08
RescheduleDelays = 0
MaxRescheduling = 3
}
}
}
Expand Down
1 change: 1 addition & 0 deletions diracx-cli/tests/legacy/cs_sync/integration_test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -801,6 +801,7 @@ Systems:
Optimizers:
JobScheduling:
RescheduleDelays: '0'
MaxRescheduling: '3'
FailoverURLs: {}
Services:
Matcher:
Expand Down
8 changes: 6 additions & 2 deletions diracx-cli/tests/test_jobs.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import json
import os
import tempfile
from io import StringIO

import pytest
from pytest import raises
Expand Down Expand Up @@ -51,8 +52,11 @@ async def test_submit(with_cli_login, jdl_file, capfd):
async def test_search(with_cli_login, jdl_file, capfd):
"""Test searching for jobs."""
# Submit 20 jobs
with open(jdl_file, "r") as temp_file:
await cli.jobs.submit([temp_file] * 20)
with open(jdl_file, "r") as x:
what_we_submit = x.read()
jdls = [StringIO(what_we_submit) for _ in range(20)]

await cli.jobs.submit(jdls)

cap = capfd.readouterr()

Expand Down
2 changes: 1 addition & 1 deletion diracx-client/src/diracx/client/generated/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------
# pylint: disable=wrong-import-position
Expand Down
8 changes: 7 additions & 1 deletion diracx-client/src/diracx/client/generated/_client.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand All @@ -19,6 +19,7 @@
AuthOperations,
ConfigOperations,
JobsOperations,
LollygagOperations,
ryuwd marked this conversation as resolved.
Show resolved Hide resolved
WellKnownOperations,
)

Expand All @@ -34,6 +35,8 @@ class Dirac: # pylint: disable=client-accepts-api-version-keyword
:vartype config: generated.operations.ConfigOperations
:ivar jobs: JobsOperations operations
:vartype jobs: generated.operations.JobsOperations
:ivar lollygag: LollygagOperations operations
:vartype lollygag: generated.operations.LollygagOperations
:keyword endpoint: Service URL. Required. Default value is "".
:paramtype endpoint: str
"""
Expand Down Expand Up @@ -85,6 +88,9 @@ def __init__( # pylint: disable=missing-client-constructor-parameter-credential
self.jobs = JobsOperations(
self._client, self._config, self._serialize, self._deserialize
)
self.lollygag = LollygagOperations(
self._client, self._config, self._serialize, self._deserialize
)

def send_request(
self, request: HttpRequest, *, stream: bool = False, **kwargs: Any
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand Down
6 changes: 3 additions & 3 deletions diracx-client/src/diracx/client/generated/_serialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -320,7 +320,7 @@ def _create_xml_node(tag, prefix=None, ns=None):
return ET.Element(tag)


class Model(object):
class Model:
"""Mixin for all client request body/response body models to support
serialization and deserialization.
"""
Expand Down Expand Up @@ -601,7 +601,7 @@ def _decode_attribute_map_key(key):
return key.replace("\\.", ".")


class Serializer(object): # pylint: disable=too-many-public-methods
class Serializer: # pylint: disable=too-many-public-methods
"""Request object model serializer."""

basic_types = {str: "str", int: "int", bool: "bool", float: "float"}
Expand Down Expand Up @@ -1536,7 +1536,7 @@ def xml_key_extractor(
return children[0]


class Deserializer(object):
class Deserializer:
"""Response object model deserializer.
:param dict classes: Class type dictionary for deserializing complex types.
Expand Down
2 changes: 1 addition & 1 deletion diracx-client/src/diracx/client/generated/_vendor.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion diracx-client/src/diracx/client/generated/aio/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------
# pylint: disable=wrong-import-position
Expand Down
8 changes: 7 additions & 1 deletion diracx-client/src/diracx/client/generated/aio/_client.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand All @@ -19,6 +19,7 @@
AuthOperations,
ConfigOperations,
JobsOperations,
LollygagOperations,
WellKnownOperations,
)

Expand All @@ -34,6 +35,8 @@ class Dirac: # pylint: disable=client-accepts-api-version-keyword
:vartype config: generated.aio.operations.ConfigOperations
:ivar jobs: JobsOperations operations
:vartype jobs: generated.aio.operations.JobsOperations
:ivar lollygag: LollygagOperations operations
:vartype lollygag: generated.aio.operations.LollygagOperations
:keyword endpoint: Service URL. Required. Default value is "".
:paramtype endpoint: str
"""
Expand Down Expand Up @@ -85,6 +88,9 @@ def __init__( # pylint: disable=missing-client-constructor-parameter-credential
self.jobs = JobsOperations(
self._client, self._config, self._serialize, self._deserialize
)
self.lollygag = LollygagOperations(
self._client, self._config, self._serialize, self._deserialize
)

def send_request(
self, request: HttpRequest, *, stream: bool = False, **kwargs: Any
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion diracx-client/src/diracx/client/generated/aio/_vendor.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# coding=utf-8
# --------------------------------------------------------------------------
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.26.5)
# Code generated by Microsoft (R) AutoRest Code Generator (autorest: 3.10.3, generator: @autorest/python@6.27.1)
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
# --------------------------------------------------------------------------
# pylint: disable=wrong-import-position
Expand All @@ -14,6 +14,7 @@
from ._operations import AuthOperations # type: ignore
from ._operations import ConfigOperations # type: ignore
from ._operations import JobsOperations # type: ignore
from ._operations import LollygagOperations # type: ignore

from ._patch import __all__ as _patch_all
from ._patch import *
Expand All @@ -24,6 +25,7 @@
"AuthOperations",
"ConfigOperations",
"JobsOperations",
"LollygagOperations",
]
__all__.extend([p for p in _patch_all if p not in __all__]) # pyright: ignore
_patch_sdk()
Loading
Loading