-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactoring Preload API #12
Merged
+2,430,437
−59
Merged
Changes from 1 commit
Commits
Show all changes
153 commits
Select commit
Hold shift + click to select a range
4082cab
Implement initial methods in clms.py
b-yogesh a55d133
Refactor code
b-yogesh 52b1e2a
Refactor code again
b-yogesh 31ccd70
Implement get_data_ids generator
b-yogesh b124ef1
Tiny typo fix
b-yogesh 39d8b37
Implement has_data
b-yogesh 3b34c3a
Initial version of describe_data (not working entirely)
b-yogesh 70a0456
Implement describe_data
b-yogesh c075479
Implement get_open_data_params_schema
b-yogesh 49cd69b
Implemented access token class
b-yogesh 4ffa9ab
Implemented access token class
b-yogesh f8ccbfb
[In progress] - open_data implementation
b-yogesh 94ff10d
[In progress] - open_data implementation - implemented _prepare_downl…
b-yogesh 26a6322
[In progress] - open_data implementation - more impl.
b-yogesh f6645f2
[In progress] - open_data implementation - more impl.
b-yogesh 7951b2e
Update make_api_request
b-yogesh e950bb7
temporary bbox and crs handling
b-yogesh 5ccede3
fix condition
b-yogesh 24e93d1
fix get_data_store_params_schema
b-yogesh 2e5f46f
Add constants
b-yogesh fc938bc
Refactoring
b-yogesh 76169d4
Add schema for preload_data
b-yogesh d680129
Remove error message truncation
b-yogesh a91bd5c
Update get_metadata method
b-yogesh 8aa6014
Add initial unsupported datasets check
b-yogesh 13febb6
Raise exceptions for unsupported data and add preload params schema
b-yogesh 63ba00c
Update schema methods
b-yogesh a049218
Add TODOs
b-yogesh a6ccac0
Implement first TODO: change data_id def
b-yogesh 3a1c329
Implement first TODO: add include_attr bool impl
b-yogesh efc0c16
Fix has_data and describe_data
b-yogesh d4810bd
[WIP] queue download refactor
b-yogesh e821dd2
[WIP] refactor existing code into preload and clms classes
b-yogesh 6d640f2
[WIP] implement queue downloads.
b-yogesh 5fbd5aa
[WIP] Further impl. preload
b-yogesh 7926ce4
[WIP] Further impl. preload
b-yogesh d064097
[WIP] Working download impl.
b-yogesh 50b4c52
[WIP] Add initial cancel handler
b-yogesh faaec47
[WIP] Add initial cancel handler
b-yogesh 20e44b4
[WIP] Refactoring
b-yogesh 167baf7
[WIP] Improved download data
b-yogesh 3ea8e7f
[WIP] Improved download data + merging
b-yogesh 0be046b
Finish implementation
b-yogesh 16ee337
Added tests for utils.py
b-yogesh e793038
Added tests for api_token.py
b-yogesh 3abef8e
Refactor clms.py
b-yogesh 425780d
Add docstrings clms.py
b-yogesh db902a3
Add tests for clms.py
b-yogesh 3e3ce73
add preload const
b-yogesh 5d7125d
Refactor
b-yogesh dceb47d
Add Docstrings and Type Annotations
b-yogesh 7dbdb4b
Add example notebook
b-yogesh 0d4018d
Minor fixes
b-yogesh 865ead4
Add missing docs
b-yogesh 3e1b545
Fix tests
b-yogesh 975e498
Update error message
b-yogesh 2176db7
Modify list to iterator in get_data_ids
b-yogesh 715d1cf
Update CLMSDataStoreTutorial.ipynb
b-yogesh c7c79a8
Update README.md
b-yogesh a88eaf9
Create unittest.yml
b-yogesh 70188bd
Add test_cache_manager.py
b-yogesh 56fcf42
Add test_token_handler.py
b-yogesh 65a271f
Rename .github/unittest.yml to .github/workflows/unittest.yml
b-yogesh 7a36783
Add test_processor.py
b-yogesh d98c712
Merge remote-tracking branch 'origin/yogesh_preload-data' into yogesh…
b-yogesh ef40d9a
Update env
b-yogesh 1c7888b
Remove redundant api class
b-yogesh 4546279
Update pyproject.toml
b-yogesh f91ecd1
add ipywidgets
b-yogesh 2e2250a
fix test_clms.py
b-yogesh dd3229a
Add CLMS url as constant
b-yogesh 5b0eda2
Fix tests
b-yogesh a7a612d
Update README.md
b-yogesh b9c9e03
Update README.md
b-yogesh dc622e9
Update CLMSDataStoreTutorial.ipynb
b-yogesh f3efb86
Update .gitignore
b-yogesh 0850fb6
Add numpy for tests
b-yogesh 8ebf8f7
Add missing license text
b-yogesh 7243361
Apply suggestions from code review
b-yogesh 36583c4
Rename classes and fix tests
b-yogesh 78c8943
Convert file_store and cache to properties
b-yogesh da3928e
Remove test_store.py
b-yogesh 54e2f7f
Remove None return doc
b-yogesh 02f0261
Improve docstrings
b-yogesh bdb362d
Update README.md
b-yogesh b637c77
Move functions away from utils to respective files
b-yogesh cc44bdb
Move constants to their respective classes
b-yogesh f178dce
Improve make_api_request
b-yogesh b30329c
Improve make_api_request #2
b-yogesh bb57381
Improve tests
b-yogesh 706c6df
Datastore to MutableDataStore
b-yogesh 0c49099
WIP [New preload API refactoring]
b-yogesh 96be05b
WIP [New preload API refactoring - part 2]
b-yogesh 26c030a
WIP [New preload API refactoring - update tests]
b-yogesh e58be16
update xcube version
b-yogesh 16feebf
Add further tests
b-yogesh 27ab2ef
Remove init comments
b-yogesh eba6439
Ready for release
b-yogesh 8840570
Update download_manager.py and add more tests
b-yogesh def671b
Merge branch 'refs/heads/yogesh_preload-data' into yogesh_preload-data-2
b-yogesh 0f654ab
[WIP] Further Refactoring
b-yogesh 836c34f
[WIP] Update tests
b-yogesh 725ffeb
[WIP] Delete CacheManager
b-yogesh e71f845
[WIP] Update tests
b-yogesh 4c55eb0
[WIP] Update store.py
b-yogesh 5dad841
[WIP] Update tests - cov 92
b-yogesh 2ff4818
[WIP] Add cassettes for test_store.py
b-yogesh f34808e
[WIP] Update and add more tests cov-96
b-yogesh febae11
[WIP] Add more cassettes
b-yogesh df7d512
[WIP] More refactoring
b-yogesh b5ef6a4
[WIP] More refactoring
b-yogesh ed5fb0a
[WIP] Update workflow to include xcube repo install
b-yogesh 6f59ab6
[WIP] Update notebook
b-yogesh 125c390
[WIP] Add --no-deps
b-yogesh 90b0236
[WIP] update unittest.yml
b-yogesh 74bd076
[WIP] update unittest.yml
b-yogesh 7b41350
[WIP] debug unittest.yml
b-yogesh 7ad24c1
[WIP] debug unittest.yml
b-yogesh a27810f
[WIP] debug unittest.yml
b-yogesh 4a59a54
[WIP] debug unittest.yml
b-yogesh 084a3ba
[WIP] debug unittest.yml
b-yogesh ea5299f
[WIP] add requests in deps
b-yogesh 687da65
[WIP] add fsspec, pyproj in deps
b-yogesh b694e48
[WIP] update env.yml
b-yogesh efa0002
[WIP]update unittest.yml to include all xcube deps in the env...yml
b-yogesh 99cb1cd
[WIP] update env.yml and pyproject.toml
b-yogesh c91130e
[WIP] add ipywidgets
b-yogesh 970bca6
[WIP] update CHANGES.md and README.md
b-yogesh ed8a8c3
[WIP] minor update to download_manager.py
b-yogesh 0809ecb
[WIP] remove debug statements from workflow
b-yogesh 958d766
Final refactoring
b-yogesh b894586
Add xcube back in workflow
b-yogesh e8056d9
copyright 2024 -> 2025
b-yogesh c782891
copyright 2024 -> 2025
b-yogesh d5e50b5
Update based on reviewer's comments
b-yogesh 15fe1ea
Update based on reviewer's comments, pt. 2
b-yogesh 0173b25
Add missing cassette
b-yogesh 3aa8cfa
Update notebook + use protocol for data opener ids
b-yogesh c8994d2
Update for better notifications, and removing close() when user cance…
b-yogesh 1a2914d
Multiple imports -> single imports per line
b-yogesh 268f2ff
update based on reviewers comments
b-yogesh 4eeb990
Remove tqdm
b-yogesh 51504a4
add dynamic chunk sizes
b-yogesh 9edb5e4
Update get_data_ids method args type
b-yogesh bc484f9
Fix tests
b-yogesh 702003a
Update notebook
b-yogesh b1a5241
Remove redundant method
b-yogesh 14946d2
Add spaces after the header
b-yogesh 39bd4bf
Update notebook
b-yogesh 5f61c45
Update notebook
b-yogesh d02292f
Remove tqdm dep
b-yogesh 4f8f885
Update CHANGES.md
b-yogesh cfda06c
Update CHANGES.md
b-yogesh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
copyright 2024 -> 2025
commit e8056d9e18dea039e7915db1355941dddaf08a92
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd prefer single imports here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, xcube coding convention says so.