Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactoring Preload API #12

Merged
merged 153 commits into from
Jan 21, 2025
Merged
Changes from 1 commit
Commits
Show all changes
153 commits
Select commit Hold shift + click to select a range
4082cab
Implement initial methods in clms.py
b-yogesh Nov 4, 2024
a55d133
Refactor code
b-yogesh Nov 7, 2024
52b1e2a
Refactor code again
b-yogesh Nov 7, 2024
31ccd70
Implement get_data_ids generator
b-yogesh Nov 7, 2024
b124ef1
Tiny typo fix
b-yogesh Nov 7, 2024
39d8b37
Implement has_data
b-yogesh Nov 7, 2024
3b34c3a
Initial version of describe_data (not working entirely)
b-yogesh Nov 7, 2024
70a0456
Implement describe_data
b-yogesh Nov 8, 2024
c075479
Implement get_open_data_params_schema
b-yogesh Nov 8, 2024
49cd69b
Implemented access token class
b-yogesh Nov 8, 2024
4ffa9ab
Implemented access token class
b-yogesh Nov 8, 2024
f8ccbfb
[In progress] - open_data implementation
b-yogesh Nov 11, 2024
94ff10d
[In progress] - open_data implementation - implemented _prepare_downl…
b-yogesh Nov 12, 2024
26a6322
[In progress] - open_data implementation - more impl.
b-yogesh Nov 12, 2024
f6645f2
[In progress] - open_data implementation - more impl.
b-yogesh Nov 14, 2024
7951b2e
Update make_api_request
b-yogesh Nov 15, 2024
e950bb7
temporary bbox and crs handling
b-yogesh Nov 15, 2024
5ccede3
fix condition
b-yogesh Nov 15, 2024
24e93d1
fix get_data_store_params_schema
b-yogesh Nov 15, 2024
2e5f46f
Add constants
b-yogesh Nov 15, 2024
fc938bc
Refactoring
b-yogesh Nov 15, 2024
76169d4
Add schema for preload_data
b-yogesh Nov 15, 2024
d680129
Remove error message truncation
b-yogesh Nov 15, 2024
a91bd5c
Update get_metadata method
b-yogesh Nov 15, 2024
8aa6014
Add initial unsupported datasets check
b-yogesh Nov 15, 2024
13febb6
Raise exceptions for unsupported data and add preload params schema
b-yogesh Nov 18, 2024
63ba00c
Update schema methods
b-yogesh Nov 18, 2024
a049218
Add TODOs
b-yogesh Nov 19, 2024
a6ccac0
Implement first TODO: change data_id def
b-yogesh Nov 19, 2024
3a1c329
Implement first TODO: add include_attr bool impl
b-yogesh Nov 19, 2024
efc0c16
Fix has_data and describe_data
b-yogesh Nov 19, 2024
d4810bd
[WIP] queue download refactor
b-yogesh Nov 19, 2024
e821dd2
[WIP] refactor existing code into preload and clms classes
b-yogesh Nov 20, 2024
6d640f2
[WIP] implement queue downloads.
b-yogesh Nov 22, 2024
5fbd5aa
[WIP] Further impl. preload
b-yogesh Nov 25, 2024
7926ce4
[WIP] Further impl. preload
b-yogesh Nov 26, 2024
d064097
[WIP] Working download impl.
b-yogesh Nov 27, 2024
50b4c52
[WIP] Add initial cancel handler
b-yogesh Nov 27, 2024
faaec47
[WIP] Add initial cancel handler
b-yogesh Nov 29, 2024
20e44b4
[WIP] Refactoring
b-yogesh Dec 2, 2024
167baf7
[WIP] Improved download data
b-yogesh Dec 2, 2024
3ea8e7f
[WIP] Improved download data + merging
b-yogesh Dec 3, 2024
0be046b
Finish implementation
b-yogesh Dec 5, 2024
16ee337
Added tests for utils.py
b-yogesh Dec 5, 2024
e793038
Added tests for api_token.py
b-yogesh Dec 5, 2024
3abef8e
Refactor clms.py
b-yogesh Dec 6, 2024
425780d
Add docstrings clms.py
b-yogesh Dec 6, 2024
db902a3
Add tests for clms.py
b-yogesh Dec 6, 2024
3e3ce73
add preload const
b-yogesh Dec 6, 2024
5d7125d
Refactor
b-yogesh Dec 6, 2024
dceb47d
Add Docstrings and Type Annotations
b-yogesh Dec 6, 2024
7dbdb4b
Add example notebook
b-yogesh Dec 6, 2024
0d4018d
Minor fixes
b-yogesh Dec 6, 2024
865ead4
Add missing docs
b-yogesh Dec 6, 2024
3e1b545
Fix tests
b-yogesh Dec 10, 2024
975e498
Update error message
b-yogesh Dec 10, 2024
2176db7
Modify list to iterator in get_data_ids
b-yogesh Dec 10, 2024
715d1cf
Update CLMSDataStoreTutorial.ipynb
b-yogesh Dec 10, 2024
c7c79a8
Update README.md
b-yogesh Dec 10, 2024
a88eaf9
Create unittest.yml
b-yogesh Dec 10, 2024
70188bd
Add test_cache_manager.py
b-yogesh Dec 10, 2024
56fcf42
Add test_token_handler.py
b-yogesh Dec 10, 2024
65a271f
Rename .github/unittest.yml to .github/workflows/unittest.yml
b-yogesh Dec 10, 2024
7a36783
Add test_processor.py
b-yogesh Dec 10, 2024
d98c712
Merge remote-tracking branch 'origin/yogesh_preload-data' into yogesh…
b-yogesh Dec 10, 2024
ef40d9a
Update env
b-yogesh Dec 10, 2024
1c7888b
Remove redundant api class
b-yogesh Dec 10, 2024
4546279
Update pyproject.toml
b-yogesh Dec 10, 2024
f91ecd1
add ipywidgets
b-yogesh Dec 10, 2024
2e2250a
fix test_clms.py
b-yogesh Dec 10, 2024
dd3229a
Add CLMS url as constant
b-yogesh Dec 10, 2024
5b0eda2
Fix tests
b-yogesh Dec 10, 2024
a7a612d
Update README.md
b-yogesh Dec 10, 2024
b9c9e03
Update README.md
b-yogesh Dec 11, 2024
dc622e9
Update CLMSDataStoreTutorial.ipynb
b-yogesh Dec 11, 2024
f3efb86
Update .gitignore
b-yogesh Dec 11, 2024
0850fb6
Add numpy for tests
b-yogesh Dec 11, 2024
8ebf8f7
Add missing license text
b-yogesh Dec 11, 2024
7243361
Apply suggestions from code review
b-yogesh Dec 12, 2024
36583c4
Rename classes and fix tests
b-yogesh Dec 12, 2024
78c8943
Convert file_store and cache to properties
b-yogesh Dec 12, 2024
da3928e
Remove test_store.py
b-yogesh Dec 12, 2024
54e2f7f
Remove None return doc
b-yogesh Dec 12, 2024
02f0261
Improve docstrings
b-yogesh Dec 12, 2024
bdb362d
Update README.md
b-yogesh Dec 12, 2024
b637c77
Move functions away from utils to respective files
b-yogesh Dec 12, 2024
cc44bdb
Move constants to their respective classes
b-yogesh Dec 12, 2024
f178dce
Improve make_api_request
b-yogesh Dec 13, 2024
b30329c
Improve make_api_request #2
b-yogesh Dec 13, 2024
bb57381
Improve tests
b-yogesh Dec 13, 2024
706c6df
Datastore to MutableDataStore
b-yogesh Dec 13, 2024
0c49099
WIP [New preload API refactoring]
b-yogesh Dec 30, 2024
96be05b
WIP [New preload API refactoring - part 2]
b-yogesh Jan 2, 2025
26c030a
WIP [New preload API refactoring - update tests]
b-yogesh Jan 2, 2025
e58be16
update xcube version
b-yogesh Jan 2, 2025
16feebf
Add further tests
b-yogesh Jan 3, 2025
27ab2ef
Remove init comments
b-yogesh Jan 6, 2025
eba6439
Ready for release
b-yogesh Jan 6, 2025
8840570
Update download_manager.py and add more tests
b-yogesh Jan 6, 2025
def671b
Merge branch 'refs/heads/yogesh_preload-data' into yogesh_preload-data-2
b-yogesh Jan 6, 2025
0f654ab
[WIP] Further Refactoring
b-yogesh Jan 6, 2025
836c34f
[WIP] Update tests
b-yogesh Jan 6, 2025
725ffeb
[WIP] Delete CacheManager
b-yogesh Jan 6, 2025
e71f845
[WIP] Update tests
b-yogesh Jan 7, 2025
4c55eb0
[WIP] Update store.py
b-yogesh Jan 7, 2025
5dad841
[WIP] Update tests - cov 92
b-yogesh Jan 7, 2025
2ff4818
[WIP] Add cassettes for test_store.py
b-yogesh Jan 8, 2025
f34808e
[WIP] Update and add more tests cov-96
b-yogesh Jan 8, 2025
febae11
[WIP] Add more cassettes
b-yogesh Jan 8, 2025
df7d512
[WIP] More refactoring
b-yogesh Jan 8, 2025
b5ef6a4
[WIP] More refactoring
b-yogesh Jan 9, 2025
ed5fb0a
[WIP] Update workflow to include xcube repo install
b-yogesh Jan 9, 2025
6f59ab6
[WIP] Update notebook
b-yogesh Jan 9, 2025
125c390
[WIP] Add --no-deps
b-yogesh Jan 9, 2025
90b0236
[WIP] update unittest.yml
b-yogesh Jan 9, 2025
74bd076
[WIP] update unittest.yml
b-yogesh Jan 9, 2025
7b41350
[WIP] debug unittest.yml
b-yogesh Jan 9, 2025
7ad24c1
[WIP] debug unittest.yml
b-yogesh Jan 9, 2025
a27810f
[WIP] debug unittest.yml
b-yogesh Jan 9, 2025
4a59a54
[WIP] debug unittest.yml
b-yogesh Jan 9, 2025
084a3ba
[WIP] debug unittest.yml
b-yogesh Jan 9, 2025
ea5299f
[WIP] add requests in deps
b-yogesh Jan 9, 2025
687da65
[WIP] add fsspec, pyproj in deps
b-yogesh Jan 9, 2025
b694e48
[WIP] update env.yml
b-yogesh Jan 9, 2025
efa0002
[WIP]update unittest.yml to include all xcube deps in the env...yml
b-yogesh Jan 10, 2025
99cb1cd
[WIP] update env.yml and pyproject.toml
b-yogesh Jan 10, 2025
c91130e
[WIP] add ipywidgets
b-yogesh Jan 10, 2025
970bca6
[WIP] update CHANGES.md and README.md
b-yogesh Jan 10, 2025
ed8a8c3
[WIP] minor update to download_manager.py
b-yogesh Jan 10, 2025
0809ecb
[WIP] remove debug statements from workflow
b-yogesh Jan 10, 2025
958d766
Final refactoring
b-yogesh Jan 10, 2025
b894586
Add xcube back in workflow
b-yogesh Jan 10, 2025
e8056d9
copyright 2024 -> 2025
b-yogesh Jan 10, 2025
c782891
copyright 2024 -> 2025
b-yogesh Jan 10, 2025
d5e50b5
Update based on reviewer's comments
b-yogesh Jan 15, 2025
15fe1ea
Update based on reviewer's comments, pt. 2
b-yogesh Jan 16, 2025
0173b25
Add missing cassette
b-yogesh Jan 16, 2025
3aa8cfa
Update notebook + use protocol for data opener ids
b-yogesh Jan 16, 2025
c8994d2
Update for better notifications, and removing close() when user cance…
b-yogesh Jan 16, 2025
1a2914d
Multiple imports -> single imports per line
b-yogesh Jan 17, 2025
268f2ff
update based on reviewers comments
b-yogesh Jan 17, 2025
4eeb990
Remove tqdm
b-yogesh Jan 17, 2025
51504a4
add dynamic chunk sizes
b-yogesh Jan 17, 2025
9edb5e4
Update get_data_ids method args type
b-yogesh Jan 20, 2025
bc484f9
Fix tests
b-yogesh Jan 20, 2025
702003a
Update notebook
b-yogesh Jan 20, 2025
b1a5241
Remove redundant method
b-yogesh Jan 20, 2025
14946d2
Add spaces after the header
b-yogesh Jan 20, 2025
39bd4bf
Update notebook
b-yogesh Jan 20, 2025
5f61c45
Update notebook
b-yogesh Jan 20, 2025
d02292f
Remove tqdm dep
b-yogesh Jan 20, 2025
4f8f885
Update CHANGES.md
b-yogesh Jan 20, 2025
cfda06c
Update CHANGES.md
b-yogesh Jan 20, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
copyright 2024 -> 2025
b-yogesh committed Jan 10, 2025
commit e8056d9e18dea039e7915db1355941dddaf08a92
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
MIT License

Copyright (c) 2024 Brockmann Consult GmbH
Copyright (c) 2025 Brockmann Consult GmbH

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
2 changes: 1 addition & 1 deletion test/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion test/test_api_token_handler.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion test/test_clms.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion test/test_download_manager.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion test/test_preload.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion test/test_processor.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
5 changes: 2 additions & 3 deletions test/test_store.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
@@ -27,8 +27,7 @@
from xcube.core.store import new_data_store, DatasetDescriptor
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from xcube.core.store import new_data_store, DatasetDescriptor
from xcube.core.store import new_data_store
from xcube.core.store import DatasetDescriptor

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer single imports here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, xcube coding convention says so.

from xcube.util.jsonschema import JsonObjectSchema

from xcube_clms.constants import DATA_STORE_ID, DATA_ID_SEPARATOR, \
DATA_OPENER_IDS
from xcube_clms.constants import DATA_STORE_ID, DATA_ID_SEPARATOR, DATA_OPENER_IDS
from xcube_clms.preload import ClmsPreloadHandle


2 changes: 1 addition & 1 deletion xcube_clms/__init__.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/api_token_handler.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/clms.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/constants.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/download_manager.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/plugin.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
5 changes: 2 additions & 3 deletions xcube_clms/preload.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
@@ -24,8 +24,7 @@

import fsspec
from xcube.core.store import MutableDataStore
from xcube.core.store.preload import ExecutorPreloadHandle, PreloadState, \
PreloadStatus
from xcube.core.store.preload import ExecutorPreloadHandle, PreloadState, PreloadStatus

from xcube_clms.api_token_handler import ClmsApiTokenHandler
from xcube_clms.constants import (
2 changes: 1 addition & 1 deletion xcube_clms/processor.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/store.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
2 changes: 1 addition & 1 deletion xcube_clms/utils.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
2 changes: 1 addition & 1 deletion xcube_clms/version.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# The MIT License (MIT)
# Copyright (c) 2024 by the xcube development team and contributors
# Copyright (c) 2025 by the xcube development team and contributors
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),