Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(archive-apis)!: use .7z format to archive studies #2013

Merged
merged 28 commits into from
Oct 24, 2024
Merged

feat(archive-apis)!: use .7z format to archive studies #2013

merged 28 commits into from
Oct 24, 2024

Conversation

mabw-rte
Copy link
Contributor

@mabw-rte mabw-rte commented Apr 18, 2024 •
edited
Loading

Context:
After a benchmark comparing .zip vs .7z in order to archive studies, the conclusions drawn from the experiments showed that we may gain better efficiency in both time and memory if we use .7z format.

Goal of PR:
Full migration of studies export into a .7z format.
This PR may solve ANT-1551 and ANT-1549 all at once.

@mabw-rte mabw-rte self-assigned this Apr 18, 2024
laurent-laporte-pro
laurent-laporte-pro reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@laurent-laporte-pro laurent-laporte-pro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lire la doc de 7zip avant de coder 👻

antarest/core/utils/utils.py Outdated Show resolved Hide resolved
antarest/study/storage/abstract_storage_service.py Outdated Show resolved Hide resolved
@laurent-laporte-pro laurent-laporte-pro marked this pull request as draft April 19, 2024 17:35
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch from 6220f9f to 566a8b2 Compare April 23, 2024 14:24
@pull-request-size pull-request-size bot added size/L and removed size/M labels Apr 23, 2024
@mabw-rte mabw-rte marked this pull request as ready for review April 23, 2024 14:44
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch 3 times, most recently from f0e1637 to f44dc68 Compare April 25, 2024 09:26
@makdeuneuv makdeuneuv added this to the v2.17.1 milestone May 6, 2024
@laurent-laporte-pro laurent-laporte-pro force-pushed the feature/1551-7zip-archive branch from f44dc68 to efa15f1 Compare May 21, 2024 14:34
@laurent-laporte-pro laurent-laporte-pro modified the milestones: v2.17.1, v2.17.2 May 24, 2024
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch from bb56aa4 to c406835 Compare May 26, 2024 23:45
@laurent-laporte-pro laurent-laporte-pro mentioned this pull request Jun 6, 2024
Draft
MartinBelthle
MartinBelthle requested changes Jun 11, 2024
View reviewed changes
Copy link
Contributor

@MartinBelthle MartinBelthle left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you specify which features this PR handles because it's not really clear for me.
From my understanding, the only thing it does is handling the archiving/unarchiving process with 7z (and retro compatibility)

antarest/study/storage/rawstudy/raw_study_service.py Outdated Show resolved Hide resolved
tests/study/storage/test_abstract_storage_service.py Outdated Show resolved Hide resolved
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch from 60259df to 7916350 Compare June 12, 2024 17:55
@mabw-rte mabw-rte requested a review from MartinBelthle June 12, 2024 18:51
MartinBelthle
MartinBelthle requested changes Jun 13, 2024
View reviewed changes
Copy link
Contributor

@MartinBelthle MartinBelthle left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not a major bug but your change in the code didn't fix the issue of the . folder inside your .7zarchive.
Also, you didn't answer my previous comment on what does this PR really achieve.

laurent-laporte-pro
laurent-laporte-pro suggested changes Jun 14, 2024
View reviewed changes
Copy link
Contributor

@laurent-laporte-pro laurent-laporte-pro left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Il convient d'améliorer la couverture des tests at de tester avec ZIP et 7z.

Lorsqu'on exporte une étude, on télécharge un fichier au format 7z, mais avec l'extension ".zip". Il faut corriger cela.

antarest/study/service.py Show resolved Hide resolved
antarest/study/storage/rawstudy/model/filesystem/config/model.py Show resolved Hide resolved
antarest/study/storage/rawstudy/model/filesystem/root/input/areas/list.py Show resolved Hide resolved
antarest/study/storage/rawstudy/model/filesystem/root/input/areas/list.py Outdated Show resolved Hide resolved
antarest/study/storage/rawstudy/model/filesystem/lazy_node.py Show resolved Hide resolved
antarest/study/storage/rawstudy/raw_study_service.py Show resolved Hide resolved
antarest/study/storage/rawstudy/raw_study_service.py Outdated Show resolved Hide resolved
tests/storage/business/test_export.py Outdated Show resolved Hide resolved
tests/storage/business/test_export.py Outdated Show resolved Hide resolved
@laurent-laporte-pro laurent-laporte-pro modified the milestones: v2.17.2, v2.18 Jun 19, 2024
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch 2 times, most recently from a0ef379 to 9079c11 Compare October 11, 2024 15:14
mabw-rte added 12 commits October 24, 2024 12:37
@mabw-rte
test(filesystem): retrieve files from inside an archived study in `.7…
8d20874 
…z` format
@mabw-rte
refactor(filesystem): redefine inside_archive_path
dc6d4b6
@mabw-rte
refactor(archiving): move archiving specific utils to archives.py
6c06b5a
@mabw-rte
refactor(archiving): use archiving utils from archives.py
9457834
@mabw-rte
feat(archiving): use archive_dir function for archiving
bd0b9eb
@mabw-rte
feat(archiving): add license headers to archives.py
6d69a25
@mabw-rte
refactor(filesystem): move archiving generic functions/utils to `anta…
de970b2 
…rest.core.utils` modules
@mabw-rte
refactor(study-service): remove unused methods of RawStudyService
cadd289
@mabw-rte
test(archiving): rename TestExtractZip to TestExtractArchive
54bcb2d
@mabw-rte
refactor(archiving): remove hard coded archive extensions .zip and …
0491d75 
…`.7z`
@mabw-rte
style(filesystem): remove Path casting
aef85fe
@mabw-rte
test(import-study): remove unused function build_storage_service
2f2ca59
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch from ee43b1f to 2f2ca59 Compare October 24, 2024 10:38
@mabw-rte mabw-rte force-pushed the feature/1551-7zip-archive branch from c7d973c to 02f3d17 Compare October 24, 2024 11:35
@mabw-rte mabw-rte added enhancement New feature or request feature request api REST API back-end labels Oct 24, 2024
@mabw-rte mabw-rte merged commit 843bd3a into dev Oct 24, 2024
11 checks passed
@mabw-rte mabw-rte deleted the feature/1551-7zip-archive branch October 24, 2024 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MartinBelthle MartinBelthle MartinBelthle left review comments

@flomnes flomnes flomnes left review comments

@laurent-laporte-pro laurent-laporte-pro laurent-laporte-pro requested changes

@sylvlecl sylvlecl sylvlecl approved these changes

Assignees

@mabw-rte mabw-rte

Labels
api REST API back-end enhancement New feature or request feature request size/XXL
Projects
None yet
Milestone
v2.18
Development

Successfully merging this pull request may close these issues.
6 participants
@mabw-rte @sylvlecl @flomnes @laurent-laporte-pro @MartinBelthle @makdeuneuv