Code

Refactoring Preload API #12

Code

Merged

b-yogesh merged 153 commits intomainxcube-dev/xcube-clms:mainfrom yogesh_preload-data-2xcube-dev/xcube-clms:yogesh_preload-data-2Copy head branch name to clipboardJan 21, 2025

+2,430,437-59

Commits

Commits on Nov 4, 2024

• Implement initial methods in clms.py

  b-yogesh

  committed
  4082cab

  Copy full SHA for 4082cab

Commits on Nov 7, 2024

• Refactor code

  b-yogesh

  committed
  a55d133

  Copy full SHA for a55d133

• Refactor code again

  b-yogesh

  committed
  52b1e2a

  Copy full SHA for 52b1e2a

• Implement get_data_ids generator

  b-yogesh

  committed
  31ccd70

  Copy full SHA for 31ccd70

• Tiny typo fix

  b-yogesh

  committed
  b124ef1

  Copy full SHA for b124ef1

• Implement has_data

  b-yogesh

  committed
  39d8b37

  Copy full SHA for 39d8b37

• Initial version of describe_data (not working entirely)

  b-yogesh

  committed
  3b34c3a

  Copy full SHA for 3b34c3a

Commits on Nov 8, 2024

• Implement describe_data

  b-yogesh

  committed
  70a0456

  Copy full SHA for 70a0456

• Implement get_open_data_params_schema

  b-yogesh

  committed
  c075479

  Copy full SHA for c075479

• Implemented access token class

  b-yogesh

  committed
  49cd69b

  Copy full SHA for 49cd69b

• Implemented access token class

  b-yogesh

  committed
  4ffa9ab

  Copy full SHA for 4ffa9ab

Commits on Nov 11, 2024

• [In progress] - open_data implementation

  b-yogesh

  committed
  f8ccbfb

  Copy full SHA for f8ccbfb

Commits on Nov 12, 2024

• [In progress] - open_data implementation - implemented _prepare_download_request

  b-yogesh

  committed
  94ff10d

  Copy full SHA for 94ff10d

• [In progress] - open_data implementation - more impl.

  b-yogesh

  committed
  26a6322

  Copy full SHA for 26a6322

Commits on Nov 14, 2024

• [In progress] - open_data implementation - more impl.

  b-yogesh

  committed
  f6645f2

  Copy full SHA for f6645f2

Commits on Nov 15, 2024

• Update make_api_request

  b-yogesh

  committed
  7951b2e

  Copy full SHA for 7951b2e

• temporary bbox and crs handling

  b-yogesh

  committed
  e950bb7

  Copy full SHA for e950bb7

• fix condition

  b-yogesh

  committed
  5ccede3

  Copy full SHA for 5ccede3

• fix get_data_store_params_schema

  b-yogesh

  committed
  24e93d1

  Copy full SHA for 24e93d1

• Add constants

  b-yogesh

  committed
  2e5f46f

  Copy full SHA for 2e5f46f

• Refactoring

  b-yogesh

  committed
  fc938bc

  Copy full SHA for fc938bc

• Add schema for preload_data

  b-yogesh

  committed
  76169d4

  Copy full SHA for 76169d4

• Remove error message truncation

  b-yogesh

  committed
  d680129

  Copy full SHA for d680129

• Update get_metadata method

  b-yogesh

  committed
  a91bd5c

  Copy full SHA for a91bd5c

• Add initial unsupported datasets check

  b-yogesh

  committed
  8aa6014

  Copy full SHA for 8aa6014

Commits on Nov 18, 2024

• Raise exceptions for unsupported data and add preload params schema

  b-yogesh

  committed
  13febb6

  Copy full SHA for 13febb6

• Update schema methods

  b-yogesh

  committed
  63ba00c

  Copy full SHA for 63ba00c

Commits on Nov 19, 2024

• Add TODOs

  b-yogesh

  committed
  a049218

  Copy full SHA for a049218

• Implement first TODO: change data_id def

  b-yogesh

  committed
  a6ccac0

  Copy full SHA for a6ccac0

• Implement first TODO: add include_attr bool impl

  b-yogesh

  committed
  3a1c329

  Copy full SHA for 3a1c329

• Fix has_data and describe_data

  b-yogesh

  committed
  efc0c16

  Copy full SHA for efc0c16

• [WIP] queue download refactor

  b-yogesh

  committed
  d4810bd

  Copy full SHA for d4810bd

Commits on Nov 20, 2024

• [WIP] refactor existing code into preload and clms classes

  b-yogesh

  committed
  e821dd2

  Copy full SHA for e821dd2

Commits on Nov 22, 2024

• [WIP] implement queue downloads.

  b-yogesh

  committed
  6d640f2

  Copy full SHA for 6d640f2

Commits on Nov 25, 2024

• [WIP] Further impl. preload

  b-yogesh

  committed
  5fbd5aa

  Copy full SHA for 5fbd5aa

Commits on Nov 26, 2024

• [WIP] Further impl. preload

  b-yogesh

  committed
  7926ce4

  Copy full SHA for 7926ce4

Commits on Nov 27, 2024

• [WIP] Working download impl.

  b-yogesh

  committed
  d064097

  Copy full SHA for d064097

• [WIP] Add initial cancel handler

  b-yogesh

  committed
  50b4c52

  Copy full SHA for 50b4c52

Commits on Nov 29, 2024

• [WIP] Add initial cancel handler

  b-yogesh

  committed
  faaec47

  Copy full SHA for faaec47

Commits on Dec 2, 2024

• [WIP] Refactoring

  b-yogesh

  committed
  20e44b4

  Copy full SHA for 20e44b4

• [WIP] Improved download data

  b-yogesh

  committed
  167baf7

  Copy full SHA for 167baf7

Commits on Dec 3, 2024

• [WIP] Improved download data + merging

  b-yogesh

  committed
  3ea8e7f

  Copy full SHA for 3ea8e7f

Commits on Dec 5, 2024

• Finish implementation

  b-yogesh

  committed
  0be046b

  Copy full SHA for 0be046b

• Added tests for utils.py

  b-yogesh

  committed
  16ee337

  Copy full SHA for 16ee337

• Added tests for api_token.py

  b-yogesh

  committed
  e793038

  Copy full SHA for e793038

Commits on Dec 6, 2024

• Refactor clms.py

  b-yogesh

  committed
  3abef8e

  Copy full SHA for 3abef8e

• Add docstrings clms.py

  b-yogesh

  committed
  425780d

  Copy full SHA for 425780d

• Add tests for clms.py

  b-yogesh

  committed
  db902a3

  Copy full SHA for db902a3

• add preload const

  b-yogesh

  committed
  3e3ce73

  Copy full SHA for 3e3ce73

• Refactor

  b-yogesh

  committed
  5d7125d

  Copy full SHA for 5d7125d

• Add Docstrings and Type Annotations

  b-yogesh

  committed
  dceb47d

  Copy full SHA for dceb47d

• Add example notebook

  b-yogesh

  committed
  7dbdb4b

  Copy full SHA for 7dbdb4b

• Minor fixes

  b-yogesh

  committed
  0d4018d

  Copy full SHA for 0d4018d

• Add missing docs

  b-yogesh

  committed
  865ead4

  Copy full SHA for 865ead4

Commits on Dec 10, 2024

• Fix tests

  b-yogesh

  committed
  3e1b545

  Copy full SHA for 3e1b545

• Update error message

  b-yogesh

  committed
  975e498

  Copy full SHA for 975e498

• Modify list to iterator in get_data_ids

  b-yogesh

  committed
  2176db7

  Copy full SHA for 2176db7

• Update CLMSDataStoreTutorial.ipynb

  b-yogesh

  authored
  715d1cf

  Copy full SHA for 715d1cf

• Update README.md

  b-yogesh

  committed
  c7c79a8

  Copy full SHA for c7c79a8

• Create unittest.yml

  b-yogesh

  authored
  a88eaf9

  Copy full SHA for a88eaf9

• Add test_cache_manager.py

  b-yogesh

  committed
  70188bd

  Copy full SHA for 70188bd

• Add test_token_handler.py

  b-yogesh

  committed
  56fcf42

  Copy full SHA for 56fcf42

• Rename .github/unittest.yml to .github/workflows/unittest.yml

  b-yogesh

  authored
  65a271f

  Copy full SHA for 65a271f

• Add test_processor.py

  b-yogesh

  committed
  7a36783

  Copy full SHA for 7a36783

• Merge remote-tracking branch 'origin/yogesh_preload-data' into yogesh_preload-data

  b-yogesh

  committed
  d98c712

  Copy full SHA for d98c712

• Update env

  b-yogesh

  committed
  ef40d9a

  Copy full SHA for ef40d9a

• Remove redundant api class

  b-yogesh

  committed
  1c7888b

  Copy full SHA for 1c7888b

• Update pyproject.toml

  b-yogesh

  committed
  4546279

  Copy full SHA for 4546279

• add ipywidgets

  b-yogesh

  committed
  f91ecd1

  Copy full SHA for f91ecd1

• fix test_clms.py

  b-yogesh

  committed
  2e2250a

  Copy full SHA for 2e2250a

• Add CLMS url as constant

  b-yogesh

  committed
  dd3229a

  Copy full SHA for dd3229a

• Fix tests

  b-yogesh

  committed
  5b0eda2

  Copy full SHA for 5b0eda2

• Update README.md

  b-yogesh

  committed
  a7a612d

  Copy full SHA for a7a612d

Commits on Dec 11, 2024

• Update README.md

  b-yogesh

  committed
  b9c9e03

  Copy full SHA for b9c9e03

• Update CLMSDataStoreTutorial.ipynb

  b-yogesh

  committed
  dc622e9

  Copy full SHA for dc622e9

• Update .gitignore

  b-yogesh

  committed
  f3efb86

  Copy full SHA for f3efb86

• Add numpy for tests

  b-yogesh

  committed
  0850fb6

  Copy full SHA for 0850fb6

• Add missing license text

  b-yogesh

  committed
  8ebf8f7

  Copy full SHA for 8ebf8f7

Commits on Dec 12, 2024

• Apply suggestions from code review

  Show description for 7243361
  

  b-yogesh

  and

  forman

  authored
  7243361

  Copy full SHA for 7243361

• Rename classes and fix tests

  b-yogesh

  committed
  36583c4

  Copy full SHA for 36583c4

• Convert file_store and cache to properties

  b-yogesh

  committed
  78c8943

  Copy full SHA for 78c8943

• Remove test_store.py

  b-yogesh

  committed
  da3928e

  Copy full SHA for da3928e

• Remove None return doc

  b-yogesh

  committed
  54e2f7f

  Copy full SHA for 54e2f7f

• Improve docstrings

  b-yogesh

  committed
  02f0261

  Copy full SHA for 02f0261

• Update README.md

  b-yogesh

  authored
  bdb362d

  Copy full SHA for bdb362d

• Move functions away from utils to respective files

  b-yogesh

  committed
  b637c77

  Copy full SHA for b637c77

• Move constants to their respective classes

  b-yogesh

  committed
  cc44bdb

  Copy full SHA for cc44bdb

Commits on Dec 13, 2024

• Improve make_api_request

  b-yogesh

  committed
  f178dce

  Copy full SHA for f178dce

• Improve make_api_request #2

  b-yogesh

  committed
  b30329c

  Copy full SHA for b30329c

• Improve tests

  b-yogesh

  committed
  bb57381

  Copy full SHA for bb57381

• Datastore to MutableDataStore

  b-yogesh

  committed
  706c6df

  Copy full SHA for 706c6df

Commits on Dec 30, 2024

• WIP [New preload API refactoring]

  b-yogesh

  committed
  0c49099

  Copy full SHA for 0c49099

Commits on Jan 2, 2025

• WIP [New preload API refactoring - part 2]

  b-yogesh

  committed
  96be05b

  Copy full SHA for 96be05b

• WIP [New preload API refactoring - update tests]

  b-yogesh

  committed
  26c030a

  Copy full SHA for 26c030a

• update xcube version

  b-yogesh

  committed
  e58be16

  Copy full SHA for e58be16

Commits on Jan 3, 2025

• Add further tests

  b-yogesh

  committed
  16feebf

  Copy full SHA for 16feebf

Commits on Jan 6, 2025

• Remove init comments

  b-yogesh

  committed
  27ab2ef

  Copy full SHA for 27ab2ef

• Ready for release

  b-yogesh

  committed
  eba6439

  Copy full SHA for eba6439

• Update download_manager.py and add more tests

  b-yogesh

  committed
  8840570

  Copy full SHA for 8840570

• Merge branch 'refs/heads/yogesh_preload-data' into yogesh_preload-data-2

  Show description for def671b
  b-yogesh

  committed
  def671b

  Copy full SHA for def671b

• [WIP] Further Refactoring

  b-yogesh

  committed
  0f654ab

  Copy full SHA for 0f654ab

• [WIP] Update tests

  b-yogesh

  committed
  836c34f

  Copy full SHA for 836c34f

• [WIP] Delete CacheManager

  b-yogesh

  committed
  725ffeb

  Copy full SHA for 725ffeb

Commits on Jan 7, 2025

• [WIP] Update tests

  b-yogesh

  committed
  e71f845

  Copy full SHA for e71f845

• [WIP] Update store.py

  b-yogesh

  committed
  4c55eb0

  Copy full SHA for 4c55eb0

• [WIP] Update tests - cov 92

  b-yogesh

  committed
  5dad841

  Copy full SHA for 5dad841

Commits on Jan 8, 2025

• [WIP] Add cassettes for test_store.py

  b-yogesh

  committed
  2ff4818

  Copy full SHA for 2ff4818

• [WIP] Update and add more tests cov-96

  b-yogesh

  committed
  f34808e

  Copy full SHA for f34808e

• [WIP] Add more cassettes

  b-yogesh

  committed
  febae11

  Copy full SHA for febae11

• [WIP] More refactoring

  b-yogesh

  committed
  df7d512

  Copy full SHA for df7d512

Commits on Jan 9, 2025

• [WIP] More refactoring

  b-yogesh

  committed
  b5ef6a4

  Copy full SHA for b5ef6a4

• [WIP] Update workflow to include xcube repo install

  b-yogesh

  committed
  ed5fb0a

  Copy full SHA for ed5fb0a

• [WIP] Update notebook

  b-yogesh

  committed
  6f59ab6

  Copy full SHA for 6f59ab6

• [WIP] Add --no-deps

  b-yogesh

  committed
  125c390

  Copy full SHA for 125c390

• [WIP] update unittest.yml

  b-yogesh

  committed
  90b0236

  Copy full SHA for 90b0236

• [WIP] update unittest.yml

  b-yogesh

  committed
  74bd076

  Copy full SHA for 74bd076

• [WIP] debug unittest.yml

  b-yogesh

  committed
  7b41350

  Copy full SHA for 7b41350

• [WIP] debug unittest.yml

  b-yogesh

  committed
  7ad24c1

  Copy full SHA for 7ad24c1

• [WIP] debug unittest.yml

  b-yogesh

  committed
  a27810f

  Copy full SHA for a27810f

• [WIP] debug unittest.yml

  b-yogesh

  committed
  4a59a54

  Copy full SHA for 4a59a54

• [WIP] debug unittest.yml

  b-yogesh

  committed
  084a3ba

  Copy full SHA for 084a3ba

• [WIP] add requests in deps

  b-yogesh

  committed
  ea5299f

  Copy full SHA for ea5299f

• [WIP] add fsspec, pyproj in deps

  b-yogesh

  committed
  687da65

  Copy full SHA for 687da65

• [WIP] update env.yml

  b-yogesh

  committed
  b694e48

  Copy full SHA for b694e48

Commits on Jan 10, 2025

• [WIP]update unittest.yml to include all xcube deps in the env...yml

  b-yogesh

  committed
  efa0002

  Copy full SHA for efa0002

• [WIP] update env.yml and pyproject.toml

  b-yogesh

  committed
  99cb1cd

  Copy full SHA for 99cb1cd

• [WIP] add ipywidgets

  b-yogesh

  committed
  c91130e

  Copy full SHA for c91130e

• [WIP] update CHANGES.md and README.md

  b-yogesh

  committed
  970bca6

  Copy full SHA for 970bca6

• [WIP] minor update to download_manager.py

  b-yogesh

  committed
  ed8a8c3

  Copy full SHA for ed8a8c3

• [WIP] remove debug statements from workflow

  b-yogesh

  committed
  0809ecb

  Copy full SHA for 0809ecb

• Final refactoring

  b-yogesh

  committed
  958d766

  Copy full SHA for 958d766

• Add xcube back in workflow

  b-yogesh

  committed
  b894586

  Copy full SHA for b894586

• copyright 2024 -> 2025

  b-yogesh

  committed
  e8056d9

  Copy full SHA for e8056d9

• copyright 2024 -> 2025

  b-yogesh

  committed
  c782891

  Copy full SHA for c782891

Commits on Jan 15, 2025

• Update based on reviewer's comments

  b-yogesh

  committed
  d5e50b5

  Copy full SHA for d5e50b5

Commits on Jan 16, 2025

• Update based on reviewer's comments, pt. 2

  b-yogesh

  committed
  15fe1ea

  Copy full SHA for 15fe1ea

• Add missing cassette

  b-yogesh

  committed
  0173b25

  Copy full SHA for 0173b25

• Update notebook + use protocol for data opener ids

  b-yogesh

  committed
  3aa8cfa

  Copy full SHA for 3aa8cfa

• Update for better notifications, and removing close() when user cancels the preload

  b-yogesh

  committed
  c8994d2

  Copy full SHA for c8994d2

Commits on Jan 17, 2025

• Multiple imports -> single imports per line

  b-yogesh

  committed
  1a2914d

  Copy full SHA for 1a2914d

• update based on reviewers comments

  b-yogesh

  committed
  268f2ff

  Copy full SHA for 268f2ff

• Remove tqdm

  b-yogesh

  committed
  4eeb990

  Copy full SHA for 4eeb990

• add dynamic chunk sizes

  b-yogesh

  committed
  51504a4

  Copy full SHA for 51504a4

Commits on Jan 20, 2025

• Update get_data_ids method args type

  b-yogesh

  committed
  9edb5e4

  Copy full SHA for 9edb5e4

• Fix tests

  b-yogesh

  committed
  bc484f9

  Copy full SHA for bc484f9

• Update notebook

  b-yogesh

  committed
  702003a

  Copy full SHA for 702003a

• Remove redundant method

  b-yogesh

  committed
  b1a5241

  Copy full SHA for b1a5241

• Add spaces after the header

  b-yogesh

  committed
  14946d2

  Copy full SHA for 14946d2

• Update notebook

  b-yogesh

  committed
  39bd4bf

  Copy full SHA for 39bd4bf

• Update notebook

  b-yogesh

  committed
  5f61c45

  Copy full SHA for 5f61c45

• Remove tqdm dep

  b-yogesh

  committed
  d02292f

  Copy full SHA for d02292f

• Update CHANGES.md

  b-yogesh

  committed
  4f8f885

  Copy full SHA for 4f8f885

• Update CHANGES.md

  b-yogesh

  committed
  cfda06c

  Copy full SHA for cfda06c