Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

⬆️ Maintenance: upgrades catalog service requirements #6846

Merged
merged 20 commits into from
Dec 9, 2024
Merged

⬆️ Maintenance: upgrades catalog service requirements #6846

merged 20 commits into from
Dec 9, 2024

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Nov 27, 2024 •
edited
Loading

What do these changes do?

  • Full upgrade of catalog services
  • We encounter a serialization issue with messages from one service to another via rabbit. The sender serialized with new version of pydantic and the receiver tried to deserialize with an older version which produced errors like
|   'exception_details': 'NEWOBJ class argument must be a type, not '
|                        '_AnnotatedAlias',
|   'exception_type': "<class '_pickle.UnpicklingError'>",
  • Question: How can we prevent this issue?
    Currently, we lack test coverage that fully addresses this scenario. To mitigate the problem, one potential solution is to enforce consistent versions of "essential libraries" across the repository, such as pydantic, fastapi, aiohttp, sqlalchemy, and others.

Overview

  • #packages before ~ 107
  • #packages after ~ 110
# name before after upgrade count packages
1 aio-pika 9.4.1 9.5.3 minor 1 catalog⬆️
2 aiocache 0.12.2 0.12.3 1 catalog⬆️
3 aiodocker 0.21.0 0.24.0 minor 1 catalog⬆️
4 aiofiles 23.2.1 24.1.0 MAJOR 1 catalog⬆️
5 aiohttp 3.9.3 3.11.10 minor 2 catalog⬆️🧪
6 aiormq 6.8.0 6.8.1 1 catalog⬆️
7 alembic 1.13.1 1.14.0 minor 2 catalog⬆️🧪
8 anyio 4.3.0 4.7.0 minor 2 catalog⬆️🧪
9 astroid 3.3.4 3.3.5 1 catalog🔧
10 async-timeout 4.0.3 🗑️ removed 1 catalog⬆️
11 asyncpg 0.29.0 0.30.0 minor 1 catalog⬆️
12 attrs 23.2.0 24.2.0 MAJOR 2 catalog⬆️🧪
13 black 24.8.0 24.10.0 minor 1 catalog🔧
14 build 1.2.2 1.2.2.post1 1 catalog🔧
15 certifi 2024.2.2 2024.8.30 minor 2 catalog⬆️🧪
16 charset-normalizer 3.3.2 3.4.0 minor 2 catalog⬆️🧪
17 coverage 7.6.1 7.6.8 1 catalog🧪
18 deprecated 1.2.14 1.2.15 1 catalog⬆️
19 dill 0.3.8 0.3.9 1 catalog🔧
20 distlib 0.3.8 0.3.9 1 catalog🔧
21 dnspython 2.6.1 2.7.0 minor 1 catalog⬆️
22 email-validator 2.1.1 2.2.0 minor 1 catalog⬆️
23 faker 29.0.0 33.1.0 MAJOR 1 catalog🧪
24 fastapi 0.115.5 0.115.6 1 catalog⬆️
25 fastapi-cli 0.0.5 0.0.6 1 catalog⬆️
26 faststream 0.5.31 0.5.33 1 catalog⬆️
27 frozenlist 1.4.1 1.5.0 minor 2 catalog⬆️🧪
28 googleapis-common-protos 1.65.0 1.66.0 minor 1 catalog⬆️
29 greenlet 3.0.3 3.1.1 minor 2 catalog⬆️🧪
30 grpcio 1.66.0 1.68.1 minor 1 catalog⬆️
31 httpcore 1.0.4 1.0.7 2 catalog⬆️🧪
32 httptools 0.6.1 0.6.4 1 catalog⬆️
33 httpx 0.27.0 0.28.0 minor 2 catalog⬆️🧪
34 identify 2.6.1 2.6.3 1 catalog🔧
35 idna 3.6 3.10 minor 2 catalog⬆️🧪
36 importlib-metadata 8.0.0 8.5.0 minor 1 catalog⬆️
37 itsdangerous 2.1.2 2.2.0 minor 1 catalog⬆️
38 jinja2 3.1.3 3.1.4 1 catalog⬆️
39 jsonschema 4.21.1 4.23.0 minor 2 catalog⬆️🧪
40 jsonschema-specifications 2023.7.1 2024.10.1 MAJOR 2 catalog⬆️🧪
41 mako 1.3.2 1.3.7 2 catalog⬆️🧪
42 markupsafe 2.1.5 3.0.2 MAJOR 2 catalog⬆️🧪
43 multidict 6.0.5 6.1.0 minor 2 catalog⬆️🧪
44 mypy 1.12.0 1.13.0 minor 2 catalog🧪🔧
45 opentelemetry-api 1.27.0 1.28.2 minor 1 catalog⬆️
46 opentelemetry-exporter-otlp 1.27.0 1.28.2 minor 1 catalog⬆️
47 opentelemetry-exporter-otlp-proto-common 1.27.0 1.28.2 minor 1 catalog⬆️
48 opentelemetry-exporter-otlp-proto-grpc 1.27.0 1.28.2 minor 1 catalog⬆️
49 opentelemetry-exporter-otlp-proto-http 1.27.0 1.28.2 minor 1 catalog⬆️
50 opentelemetry-instrumentation 0.48 0.49 minor 1 catalog⬆️
51 opentelemetry-instrumentation-asgi 0.48 0.49 minor 1 catalog⬆️
52 opentelemetry-instrumentation-asyncpg 0.48 0.49 minor 1 catalog⬆️
53 opentelemetry-instrumentation-fastapi 0.48 0.49 minor 1 catalog⬆️
54 opentelemetry-instrumentation-httpx 0.48 0.49 minor 1 catalog⬆️
55 opentelemetry-instrumentation-logging 0.48 0.49 minor 1 catalog⬆️
56 opentelemetry-instrumentation-redis 0.48 0.49 minor 1 catalog⬆️
57 opentelemetry-instrumentation-requests 0.48 0.49 minor 1 catalog⬆️
58 opentelemetry-proto 1.27.0 1.28.2 minor 1 catalog⬆️
59 opentelemetry-sdk 1.27.0 1.28.2 minor 1 catalog⬆️
60 opentelemetry-semantic-conventions 0.48 0.49 minor 1 catalog⬆️
61 opentelemetry-util-http 0.48 0.49 minor 1 catalog⬆️
62 orjson 3.10.0 3.10.12 1 catalog⬆️
63 packaging 24.0 24.2 minor 3 catalog⬆️🧪🔧
64 pip 24.2 24.3.1 minor 1 catalog🔧
65 pre-commit 3.8.0 4.0.1 MAJOR 1 catalog🔧
66 prometheus-client 0.20.0 0.21.1 minor 1 catalog⬆️
67 prometheus-fastapi-instrumentator 6.1.0 7.0.0 MAJOR 1 catalog⬆️
68 protobuf 4.25.4 5.29.1 MAJOR 1 catalog⬆️
69 psutil 6.0.0 6.1.0 minor 1 catalog⬆️
70 psycopg2-binary 2.9.9 2.9.10 1 catalog⬆️
71 pydantic 2.10.2 2.10.3 1 catalog⬆️
72 pydantic-extra-types 2.9.0 2.10.0 minor 1 catalog⬆️
73 pygments 2.17.2 2.18.0 minor 1 catalog⬆️
74 pyinstrument 4.6.2 5.0.0 MAJOR 1 catalog⬆️
75 pylint 3.3.0 3.3.2 1 catalog🔧
76 pyproject-hooks 1.1.0 1.2.0 minor 1 catalog🔧
77 pytest 8.3.3 8.3.4 1 catalog🧪
78 pytest-benchmark 4.0.0 5.1.0 MAJOR 1 catalog🧪
79 pytest-cov 5.0.0 6.0.0 MAJOR 1 catalog🧪
80 python-multipart 0.0.9 0.0.19 1 catalog⬆️
81 pyyaml 6.0.1 6.0.2 2 catalog⬆️🔧
82 redis 5.0.4 5.2.1 minor 1 catalog⬆️
83 referencing 0.29.3 0.35.1 minor 2 catalog⬆️🧪
84 rich 13.7.1 13.9.4 minor 1 catalog⬆️
85 rpds-py 0.18.0 0.22.3 minor 2 catalog⬆️🧪
86 ruff 0.6.7 0.8.2 minor 1 catalog🔧
87 setuptools 74.0.0 75.6.0 MAJOR 2 catalog⬆️🔧
88 six 1.16.0 1.17.0 minor 2 catalog⬆️🧪
89 sqlalchemy 1.4.52 1.4.54 2 catalog⬆️🧪
90 starlette 0.41.0 0.41.3 1 catalog⬆️
91 tenacity 8.5.0 9.0.0 MAJOR 1 catalog⬆️
92 toolz 0.12.1 1.0.0 MAJOR 1 catalog⬆️
93 tqdm 4.66.2 4.67.1 minor 1 catalog⬆️
94 typer 0.12.3 0.15.1 minor 1 catalog⬆️
95 types-psycopg2 2.9.21.20240819 2.9.21.20241019 1 catalog🧪
96 types-python-dateutil 2.9.0.20240316 2.9.0.20241206 1 catalog⬆️
97 ujson 5.9.0 5.10.0 minor 1 catalog⬆️
98 uvicorn 0.29.0 0.32.1 minor 1 catalog⬆️
99 uvloop 0.19.0 0.21.0 minor 1 catalog⬆️
100 virtualenv 20.26.5 20.28.0 minor 1 catalog🔧
101 watchdog 5.0.2 6.0.0 MAJOR 1 catalog🔧
102 watchfiles 0.21.0 1.0.0 MAJOR 1 catalog⬆️
103 websockets 12.0 14.1 MAJOR 1 catalog⬆️
104 wheel 0.44.0 0.45.1 minor 1 catalog🔧
105 wrapt 1.16.0 1.17.0 minor 1 catalog⬆️
106 yarl 1.9.4 1.18.3 minor 2 catalog⬆️🧪
107 zipp 3.20.1 3.21.0 minor 1 catalog⬆️

Legend:

  • ⬆️ base dependency (only services because packages are floating)
  • 🧪 test dependency
  • 🔧 tool dependency

@pcrespov pcrespov self-assigned this Nov 27, 2024
@pcrespov pcrespov changed the title upgrades reqs catalog ⬆️ upgrades catalog service requirements Nov 27, 2024
@pcrespov pcrespov added t:maintenance Some planned maintenance work a:catalog catalog service labels Nov 27, 2024
@pcrespov pcrespov changed the title ⬆️ upgrades catalog service requirements ⬆️ Maintenance: upgrades catalog service requirements Nov 27, 2024
@pcrespov pcrespov changed the title ⬆️ Maintenance: upgrades catalog service requirements ⬆️ Maintenance: upgrades catalog service requirements Nov 27, 2024
@pcrespov pcrespov added this to the Event Horizon milestone Nov 27, 2024
@pcrespov pcrespov force-pushed the mai/upgrade-catalog branch from 2c032dc to bd3b845 Compare November 27, 2024 09:47
@codecov Codecov
Copy link

codecov bot commented Nov 27, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.31%. Comparing base (0651c06) to head (0ee367d).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (0651c06) and HEAD (0ee367d). Click for more details.

HEAD has 29 uploads less than BASE
Flag BASE (0651c06) HEAD (0ee367d)
unittests 30 1
Additional details and impacted files 
@@             Coverage Diff             @@
##           master    #6846       +/-   ##
===========================================
- Coverage   88.33%   66.31%   -22.03%     
===========================================
  Files        1571      672      -899     
  Lines       61625    31996    -29629     
  Branches     2000      262     -1738     
===========================================
- Hits        54439    21219    -33220     
- Misses       6851    10720     +3869     
+ Partials      335       57      -278
Flag Coverage Δ
integrationtests 65.05% <ø> (-0.01%) ⬇️
unittests 90.57% <ø> (+4.03%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library ∅ <ø> (∅)
pkg_notifications_library ∅ <ø> (∅)
pkg_postgres_database ∅ <ø> (∅)
pkg_service_integration ∅ <ø> (∅)
pkg_service_library ∅ <ø> (∅)
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 77.60% <ø> (-7.78%) ⬇️
agent ∅ <ø> (∅)
api_server ∅ <ø> (∅)
autoscaling ∅ <ø> (∅)
catalog 90.57% <ø> (ø)
clusters_keeper ∅ <ø> (∅)
dask_sidecar ∅ <ø> (∅)
datcore_adapter ∅ <ø> (∅)
director ∅ <ø> (∅)
director_v2 78.82% <ø> (-12.57%) ⬇️
dynamic_scheduler ∅ <ø> (∅)
dynamic_sidecar 59.86% <ø> (-29.89%) ⬇️
efs_guardian ∅ <ø> (∅)
invitations ∅ <ø> (∅)
osparc_gateway_server ∅ <ø> (∅)
payments ∅ <ø> (∅)
resource_usage_tracker ∅ <ø> (∅)
storage ∅ <ø> (∅)
webclient ∅ <ø> (∅)
webserver 59.65% <ø> (-28.66%) ⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0651c06...0ee367d. Read the comment docs.

@pcrespov
Copy link
Member Author

pcrespov commented Nov 27, 2024 •
edited
Loading

@giancarloromeo

|   'exception_details': 'NEWOBJ class argument must be a type, not '
|                        '_AnnotatedAlias',
|   'exception_type': "<class '_pickle.UnpicklingError'>",

@pcrespov pcrespov force-pushed the mai/upgrade-catalog branch from efbba27 to 2dcbf57 Compare November 28, 2024 17:41
@pcrespov pcrespov force-pushed the mai/upgrade-catalog branch from 264f767 to b3bd671 Compare December 2, 2024 11:21
@pcrespov pcrespov marked this pull request as ready for review December 6, 2024 10:54
@pcrespov pcrespov requested a review from GitHK December 6, 2024 10:55
bisgaard-itis
bisgaard-itis reviewed Dec 6, 2024
View reviewed changes
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect your unit tests will start failing due to the compatibility issue mentioned below.

services/catalog/requirements/_test.txt Outdated Show resolved Hide resolved
@pcrespov pcrespov mentioned this pull request Dec 6, 2024
Merged
bisgaard-itis
bisgaard-itis approved these changes Dec 6, 2024
View reviewed changes
Copy link
Contributor

@bisgaard-itis bisgaard-itis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! Thanks. Please consider the below suggestion

requirements/constraints.txt Outdated Show resolved Hide resolved
@pcrespov pcrespov enabled auto-merge (squash) December 6, 2024 14:09
@sonarqubecloud SonarQubeCloud
Copy link

sonarqubecloud bot commented Dec 9, 2024

Quality Gate Passed Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@pcrespov pcrespov merged commit 9a23d6e into ITISFoundation:master Dec 9, 2024
88 of 93 checks passed
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request Dec 10, 2024
Closed
17 tasks
@pcrespov pcrespov deleted the mai/upgrade-catalog branch December 10, 2024 13:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@GitHK GitHK GitHK left review comments

@bisgaard-itis bisgaard-itis bisgaard-itis approved these changes

@matusdrobuliak66 matusdrobuliak66 matusdrobuliak66 approved these changes

@sanderegg sanderegg sanderegg approved these changes

Assignees

@pcrespov pcrespov

Labels
a:catalog catalog service t:maintenance Some planned maintenance work
Projects
None yet
Milestone
Event Horizon
Development

Successfully merging this pull request may close these issues.
5 participants
@pcrespov @GitHK @sanderegg @matusdrobuliak66 @bisgaard-itis