fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41
Conversation
|
Thansk @jjjermiah ! You probably want to remove the |
|
thanks for pointing that out @vsoch I also tried formatting and fixing the test suite |
WalkthroughThis update enhances the GitHub Actions workflow for testing, improves handling of Google Cloud Storage in the Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant CI
participant GCS
participant StoragePlugin
User->>CI: Trigger Test Workflow
CI->>StoragePlugin: execute tests
StoragePlugin->>GCS: ensure_bucket_exists()
GCS-->>StoragePlugin: bucket status
StoragePlugin->>StoragePlugin: upload_directory()
StoragePlugin->>GCS: upload files
GCS-->>StoragePlugin: confirm upload
StoragePlugin->>CI: return test results
CI->>User: display results
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
tests/tests.py
Outdated
| def test_storage_nonempty_directory(self, tmp_path): | ||
| # make a directory | ||
| tmpdir = "test_nonemptydir" | ||
|
|
||
| # store the directory | ||
| obj = self._get_obj(tmp_path, f"gcs://snakemake-test-bucket/{tmpdir}") | ||
|
|
||
| stored = False | ||
| try: | ||
| if not self.retrieve_only: | ||
| obj.local_path().mkdir(parents=True, exist_ok=True) | ||
|
|
||
| assert obj.is_directory() | ||
|
|
||
| print(obj.local_path()) | ||
| print("Writing a file in the directory") | ||
| # write a file in the directory | ||
| with open(obj.local_path() / "testfile.txt", "w") as f: | ||
| f.write("test") | ||
| f.flush() | ||
|
|
||
| assert obj.bucket.exists() | ||
| assert obj.local_path().exists() and obj.local_path().is_dir() | ||
| print("Storing the directory") | ||
|
|
||
| obj.store_object() | ||
| stored = True | ||
|
|
||
| assert obj.exists() | ||
| print(obj.mtime()) | ||
| print(obj.size()) | ||
|
|
||
| file = obj.local_path() / "testfile.txt" | ||
| assert file.exists() | ||
| print(file.read_text()) |
There was a problem hiding this comment.
This seems redundant with the test suite, which by now also tests directories. Maybe it can be removed now?
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock
Files selected for processing (5)
- .github/workflows/ci.yml (1 hunks)
- .gitignore (1 hunks)
- snakemake_storage_plugin_gcs/init.py (4 hunks)
- tests/test_fake_gcs.py (2 hunks)
- tests/tests.py (4 hunks)
Additional context used
Path-based instructions (3)
tests/test_fake_gcs.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.tests/tests.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.snakemake_storage_plugin_gcs/__init__.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Ruff
tests/tests.py
77-77: Missing return type annotation for public function
test_storage_nonempty_directoryAdd return type annotation:
None(ANN201)
77-77: Missing type annotation for
selfin method(ANN101)
77-77: Missing type annotation for function argument
tmp_path(ANN001)
99-99: Assertion should be broken down into multiple parts
Break down assertion into multiple parts
(PT018)
snakemake_storage_plugin_gcs/__init__.py
410-410: Missing type annotation for
selfin method(ANN101)
417-417: Missing return type annotation for public function
upload_directoryAdd return type annotation:
None(ANN201)
417-417: Missing type annotation for
selfin method(ANN101)
427-427: Trailing comma missing
Add trailing comma
(COM812)
448-448: Missing return type annotation for public function
removeAdd return type annotation:
None(ANN201)
448-448: Missing type annotation for
selfin method(ANN101)
Additional comments not posted (8)
.gitignore (1)
1-17: The.gitignorefile looks good.The entries are appropriate for a Python project, ensuring that unnecessary files like coverage data, pytest cache, and compiled Python files are not tracked by version control.
tests/test_fake_gcs.py (1)
41-41: Improved readability with added spacing.The addition of a blank line before the
tryblock enhances readability by visually separating the bucket creation logic from the subsequent error handling..github/workflows/ci.yml (1)
97-97: Enhanced test output visibility.Increasing the verbosity of pytest and allowing standard output during tests will aid in debugging by providing more detailed information about the test execution.
tests/tests.py (2)
22-22: Changedfiles_onlytoTrue.This change indicates a shift in focus to testing only file operations, which might impact how the tests are structured and executed.
Ensure that this change aligns with the intended test coverage and does not exclude necessary directory tests.
77-116: New test for non-empty directory storage.The
test_storage_nonempty_directorymethod adds valuable coverage for scenarios involving directory contents, ensuring that both directories and their files are handled correctly.Tools
Ruff
77-77: Missing return type annotation for public function
test_storage_nonempty_directoryAdd return type annotation:
None(ANN201)
77-77: Missing type annotation for
selfin method(ANN101)
77-77: Missing type annotation for function argument
tmp_path(ANN001)
99-99: Assertion should be broken down into multiple parts
Break down assertion into multiple parts
(PT018)
snakemake_storage_plugin_gcs/__init__.py (3)
410-416: New methodensure_bucket_exists.This method encapsulates the logic for checking and creating a bucket, improving modularity and readability of the code.
Tools
Ruff
410-410: Missing type annotation for
selfin method(ANN101)
417-446: New methodupload_directory.This method efficiently handles directory uploads to GCS, centralizing the logic and improving the
store_objectmethod's clarity.Tools
Ruff
417-417: Missing return type annotation for public function
upload_directoryAdd return type annotation:
None(ANN201)
417-417: Missing type annotation for
selfin method(ANN101)
427-427: Trailing comma missing
Add trailing comma
(COM812)
452-460: Refactoredremovemethod for directory handling.The method now effectively deletes all blobs under a directory prefix, enhancing the robustness of directory deletions.
There was a problem hiding this comment.
Actionable comments posted: 2
Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Files ignored due to path filters (1)
poetry.lockis excluded by!**/*.lock
Files selected for processing (1)
- snakemake_storage_plugin_gcs/init.py (5 hunks)
Additional context used
Path-based instructions (1)
snakemake_storage_plugin_gcs/__init__.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Ruff
snakemake_storage_plugin_gcs/__init__.py
410-410: Missing type annotation for
selfin method(ANN101)
417-417: Missing return type annotation for public function
upload_directoryAdd return type annotation:
None(ANN201)
417-417: Missing type annotation for
selfin method(ANN101)
427-427: Trailing comma missing
Add trailing comma
(COM812)
448-448: Missing return type annotation for public function
removeAdd return type annotation:
None(ANN201)
448-448: Missing type annotation for
selfin method(ANN101)
Additional comments not posted (2)
snakemake_storage_plugin_gcs/__init__.py (2)
387-400: Refactoring approved.The refactoring of
store_objectimproves modularity and readability by separating bucket existence checks and directory uploads into distinct methods.
410-416: Addition ofensure_bucket_existsimproves robustness.The method ensures that the bucket exists before performing operations, enhancing the reliability of storage interactions.
Tools
Ruff
410-410: Missing type annotation for
selfin method(ANN101)
There was a problem hiding this comment.
Actionable comments posted: 2
Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Files selected for processing (2)
- snakemake_storage_plugin_gcs/init.py (5 hunks)
- tests/tests.py (3 hunks)
Additional context used
Path-based instructions (2)
tests/tests.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.snakemake_storage_plugin_gcs/__init__.py (1)
Pattern
**/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Ruff
tests/tests.py
44-44: Missing return type annotation for public function
test_storage_nonempty_directoryAdd return type annotation:
None(ANN201)
44-44: Missing type annotation for
selfin method(ANN101)
44-44: Missing type annotation for function argument
tmp_path(ANN001)
66-66: Assertion should be broken down into multiple parts
Break down assertion into multiple parts
(PT018)
snakemake_storage_plugin_gcs/__init__.py
410-410: Missing type annotation for
selfin method(ANN101)
417-417: Missing return type annotation for public function
upload_directoryAdd return type annotation:
None(ANN201)
417-417: Missing type annotation for
selfin method(ANN101)
427-427: Trailing comma missing
Add trailing comma
(COM812)
448-448: Missing type annotation for
selfin method(ANN101)
Additional comments not posted (5)
tests/tests.py (2)
58-78: Print statements for debugging are acceptable.The added print statements help trace the execution flow and are acceptable in test functions according to project standards.
Tools
Ruff
66-66: Assertion should be broken down into multiple parts
Break down assertion into multiple parts
(PT018)
22-22: Verify the impact offiles_only = True.Changing
files_onlytoTruemay affect tests that involve directory operations. Ensure that this change aligns with the intended functionality and does not inadvertently skip necessary tests.Verification successful
No impact found from setting
files_only = True.The change to
files_only = Trueintests/tests.pyis isolated, as there are no other references tofiles_onlyin the codebase. This suggests that the change does not affect other tests or functionalities.Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the impact of changing `files_only` to `True` in the test suite. # Test: Search for usage of `files_only` in the test suite. Expect: No unintended test skips. rg --type python 'files_only'Length of output: 60
Script:
#!/bin/bash # Description: Verify the impact of changing `files_only` to `True` by searching for its usage in the codebase. # Test: Search for usage of `files_only` in the codebase. Expect: No unintended test skips. rg 'files_only'Length of output: 55
snakemake_storage_plugin_gcs/__init__.py (3)
387-400: Refactoring ofstore_objectimproves clarity and efficiency.The delegation of bucket existence checks and separate handling of directory uploads enhance the method's readability and maintainability.
448-460: Improved directory deletion logic approved.The
removemethod's update to delete all blobs matching a directory's prefix enhances the robustness of the deletion process.Tools
Ruff
448-448: Missing type annotation for
selfin method(ANN101)
410-416: Consider adding a type annotation forself.The
ensure_bucket_existsmethod improves modularity by handling bucket existence checks. Consider adding a type annotation forself.def ensure_bucket_exists(self) -> None:Likely invalid or redundant comment.
Tools
Ruff
410-410: Missing type annotation for
selfin method(ANN101)
|
Thanks a lot for the fixes! |
|
and sorry for the slow reply. This somehow slipped my attention. Feel free to contact me via discord or email if something like that happens again in the future. |
🤖 I have created a release *beep* *boop* --- ## [1.1.0](v1.0.0...v1.1.0) (2024-08-19) ### Features * list storage candidates ([#30](#30)) ([c7ba28e](c7ba28e)) ### Bug Fixes * fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. ([#41](#41)) ([27c80dc](27c80dc)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
first contribution here. Think I found the error.
This works for my test Snakefile:
I also tried to add another test case which seems to work.
I see that the test suites in the storage plugins use
testClasses.Would adding actual snakefiles and a script to run them also help or does that not follow the plugin standards?
Summary by CodeRabbit
New Features
ensure_bucket_existsandupload_directory.Bug Fixes
Chores
.gitignorefile to improve repository cleanliness by excluding unnecessary files.Tests