fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41

jjjermiah · 2024-04-26T18:34:55Z

first contribution here. Think I found the error.

This works for my test Snakefile:

rule all:
    input:
        "data/done.txt",
        "data2/done.txt"

rule get_directories:
    input:
        expand(
            "data/{directory}",
            directory = ["a", "b", "c"]
        ),
    output:
        touch("data/done.txt")

    
rule make_a_directory:
    output:
        directory("data/{directory}")
    shell:
        """
        mkdir -p {output}
        touch {output}/test.txt
        """

rule get_files:
    input:
        expand(
            "data2/{directory}.txt",
            directory = ["a", "b", "c"]
        )
    output:
        touch("data2/done.txt")
        
rule make_a_file:
    output:
        "data2/{directory}.txt"
    shell:
        """
        mkdir -p $(dirname {output})
        touch {output}
        """

I also tried to add another test case which seems to work.
I see that the test suites in the storage plugins use testClasses.
Would adding actual snakefiles and a script to run them also help or does that not follow the plugin standards?

Summary by CodeRabbit

New Features
- Introduced new methods for managing Google Cloud Storage: ensure_bucket_exists and upload_directory.
- Added a test case for handling non-empty directories.
Bug Fixes
- Enhanced the deletion process for directories in Google Cloud Storage.
Chores
- Created a .gitignore file to improve repository cleanliness by excluding unnecessary files.
- Updated GitHub Actions CI configuration for improved test output visibility.
Tests
- Enhanced test suite with additional debugging output and a focus on file operations.

vsoch · 2024-04-26T18:48:00Z

Thansk @jjjermiah ! You probably want to remove the __pycache__ stuff.

jjjermiah · 2024-04-26T20:02:55Z

thanks for pointing that out @vsoch I also tried formatting and fixing the test suite

coderabbitai · 2024-08-19T07:18:51Z

Walkthrough

This update enhances the GitHub Actions workflow for testing, improves handling of Google Cloud Storage in the snakemake_storage_plugin_gcs module with new methods, and refines test cases for better clarity and functionality. Key changes include increased verbosity in test outputs, a new .gitignore file to exclude unnecessary files, and adjustments to ensure comprehensive testing of both files and directories.

Changes

Files	Change Summary
`.github/workflows/ci.yml`	Updated `pytest` command to include more verbosity and display print outputs during tests.
`.gitignore`	New file created to prevent tracking of autogenerated files such as `.coverage` and `__pycache__`.
`snakemake_storage_plugin_gcs/__init__.py`	Added `ensure_bucket_exists` and `upload_directory` methods; refactored `store_object` and `remove` methods for improved functionality and clarity.
`tests/tests.py`	Updated `TestStorage` class to prioritize file operations, added tests for non-empty directories, and improved debug output and cleanup.

Sequence Diagram(s)

sequenceDiagram
    participant User
    participant CI
    participant GCS
    participant StoragePlugin

    User->>CI: Trigger Test Workflow
    CI->>StoragePlugin: execute tests
    StoragePlugin->>GCS: ensure_bucket_exists()
    GCS-->>StoragePlugin: bucket status
    StoragePlugin->>StoragePlugin: upload_directory()
    StoragePlugin->>GCS: upload files
    GCS-->>StoragePlugin: confirm upload
    StoragePlugin->>CI: return test results
    CI->>User: display results

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai generate interesting stats about this repository and render them as a table.
- @coderabbitai show all the console.log statements in this repository.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

johanneskoester · 2024-08-19T07:22:00Z

tests/tests.py

+    def test_storage_nonempty_directory(self, tmp_path):
+        # make a directory
+        tmpdir = "test_nonemptydir"
+
+        # store the directory
+        obj = self._get_obj(tmp_path, f"gcs://snakemake-test-bucket/{tmpdir}")
+
+        stored = False
+        try:
+            if not self.retrieve_only:
+                obj.local_path().mkdir(parents=True, exist_ok=True)
+
+                assert obj.is_directory()
+
+                print(obj.local_path())
+                print("Writing a file in the directory")
+                # write a file in the directory
+                with open(obj.local_path() / "testfile.txt", "w") as f:
+                    f.write("test")
+                    f.flush()
+
+                assert obj.bucket.exists()
+                assert obj.local_path().exists() and obj.local_path().is_dir()
+                print("Storing the directory")
+
+                obj.store_object()
+                stored = True
+
+            assert obj.exists()
+            print(obj.mtime())
+            print(obj.size())
+
+            file = obj.local_path() / "testfile.txt"
+            assert file.exists()
+            print(file.read_text())


This seems redundant with the test suite, which by now also tests directories. Maybe it can be removed now?

coderabbitai

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between c7ba28e and d8ee87d.

Files ignored due to path filters (1)

poetry.lock is excluded by !**/*.lock

Files selected for processing (5)

.github/workflows/ci.yml (1 hunks)
.gitignore (1 hunks)
snakemake_storage_plugin_gcs/init.py (4 hunks)
tests/test_fake_gcs.py (2 hunks)
tests/tests.py (4 hunks)

Additional context used

Path-based instructions (3)

tests/test_fake_gcs.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

tests/tests.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

snakemake_storage_plugin_gcs/__init__.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

Ruff

tests/tests.py

77-77: Missing return type annotation for public function test_storage_nonempty_directory

Add return type annotation: None

(ANN201)

77-77: Missing type annotation for self in method

(ANN101)

77-77: Missing type annotation for function argument tmp_path

(ANN001)

99-99: Assertion should be broken down into multiple parts

Break down assertion into multiple parts

(PT018)

snakemake_storage_plugin_gcs/__init__.py

410-410: Missing type annotation for self in method

(ANN101)

417-417: Missing return type annotation for public function upload_directory

Add return type annotation: None

(ANN201)

417-417: Missing type annotation for self in method

(ANN101)

427-427: Trailing comma missing

Add trailing comma

(COM812)

448-448: Missing return type annotation for public function remove

Add return type annotation: None

(ANN201)

448-448: Missing type annotation for self in method

(ANN101)

Additional comments not posted (8)

.gitignore (1)

1-17: The .gitignore file looks good.

The entries are appropriate for a Python project, ensuring that unnecessary files like coverage data, pytest cache, and compiled Python files are not tracked by version control.

tests/test_fake_gcs.py (1)

41-41: Improved readability with added spacing.

The addition of a blank line before the try block enhances readability by visually separating the bucket creation logic from the subsequent error handling.

.github/workflows/ci.yml (1)

97-97: Enhanced test output visibility.

Increasing the verbosity of pytest and allowing standard output during tests will aid in debugging by providing more detailed information about the test execution.

tests/tests.py (2)

22-22: Changed files_only to True.

This change indicates a shift in focus to testing only file operations, which might impact how the tests are structured and executed.

Ensure that this change aligns with the intended test coverage and does not exclude necessary directory tests.

77-116: New test for non-empty directory storage.

The test_storage_nonempty_directory method adds valuable coverage for scenarios involving directory contents, ensuring that both directories and their files are handled correctly.

Tools

Ruff

77-77: Missing return type annotation for public function test_storage_nonempty_directory

Add return type annotation: None

(ANN201)

77-77: Missing type annotation for self in method

(ANN101)

77-77: Missing type annotation for function argument tmp_path

(ANN001)

99-99: Assertion should be broken down into multiple parts

Break down assertion into multiple parts

(PT018)

snakemake_storage_plugin_gcs/__init__.py (3)

410-416: New method ensure_bucket_exists.

This method encapsulates the logic for checking and creating a bucket, improving modularity and readability of the code.

Tools

Ruff

410-410: Missing type annotation for self in method

(ANN101)

417-446: New method upload_directory.

This method efficiently handles directory uploads to GCS, centralizing the logic and improving the store_object method's clarity.

Tools

Ruff

417-417: Missing return type annotation for public function upload_directory

Add return type annotation: None

(ANN201)

417-417: Missing type annotation for self in method

(ANN101)

427-427: Trailing comma missing

Add trailing comma

(COM812)

452-460: Refactored remove method for directory handling.

The method now effectively deletes all blobs under a directory prefix, enhancing the robustness of directory deletions.

tests/tests.py

coderabbitai

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between d8ee87d and 2300bab.

Files ignored due to path filters (1)

poetry.lock is excluded by !**/*.lock

Files selected for processing (1)

snakemake_storage_plugin_gcs/init.py (5 hunks)

Additional context used

Path-based instructions (1)

snakemake_storage_plugin_gcs/__init__.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

Ruff

snakemake_storage_plugin_gcs/__init__.py

410-410: Missing type annotation for self in method

(ANN101)

417-417: Missing return type annotation for public function upload_directory

Add return type annotation: None

(ANN201)

417-417: Missing type annotation for self in method

(ANN101)

427-427: Trailing comma missing

Add trailing comma

(COM812)

448-448: Missing return type annotation for public function remove

Add return type annotation: None

(ANN201)

448-448: Missing type annotation for self in method

(ANN101)

Additional comments not posted (2)

snakemake_storage_plugin_gcs/__init__.py (2)

387-400: Refactoring approved.

The refactoring of store_object improves modularity and readability by separating bucket existence checks and directory uploads into distinct methods.

410-416: Addition of ensure_bucket_exists improves robustness.

The method ensures that the bucket exists before performing operations, enhancing the reliability of storage interactions.

Tools

Ruff

410-410: Missing type annotation for self in method

(ANN101)

snakemake_storage_plugin_gcs/__init__.py

coderabbitai

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between 2300bab and e97d381.

Files selected for processing (2)

snakemake_storage_plugin_gcs/init.py (5 hunks)
tests/tests.py (3 hunks)

Additional context used

Path-based instructions (2)

tests/tests.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

snakemake_storage_plugin_gcs/__init__.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.

Ruff

tests/tests.py

44-44: Missing return type annotation for public function test_storage_nonempty_directory

Add return type annotation: None

(ANN201)

44-44: Missing type annotation for self in method

(ANN101)

44-44: Missing type annotation for function argument tmp_path

(ANN001)

66-66: Assertion should be broken down into multiple parts

Break down assertion into multiple parts

(PT018)

snakemake_storage_plugin_gcs/__init__.py

410-410: Missing type annotation for self in method

(ANN101)

417-417: Missing return type annotation for public function upload_directory

Add return type annotation: None

(ANN201)

417-417: Missing type annotation for self in method

(ANN101)

427-427: Trailing comma missing

Add trailing comma

(COM812)

448-448: Missing type annotation for self in method

(ANN101)

Additional comments not posted (5)

tests/tests.py (2)
58-78: Print statements for debugging are acceptable.

The added print statements help trace the execution flow and are acceptable in test functions according to project standards.

Tools

Ruff

66-66: Assertion should be broken down into multiple parts

Break down assertion into multiple parts

(PT018)

22-22: Verify the impact of files_only = True.

Changing files_only to True may affect tests that involve directory operations. Ensure that this change aligns with the intended functionality and does not inadvertently skip necessary tests.

Verification successful

No impact found from setting files_only = True.

The change to files_only = True in tests/tests.py is isolated, as there are no other references to files_only in the codebase. This suggests that the change does not affect other tests or functionalities.
Scripts executed

The following scripts were executed for the analysis:

Script:
#!/bin/bash
# Description: Verify the impact of changing `files_only` to `True` in the test suite.

# Test: Search for usage of `files_only` in the test suite. Expect: No unintended test skips.
rg --type python 'files_only'
Length of output: 60

Script:
#!/bin/bash
# Description: Verify the impact of changing `files_only` to `True` by searching for its usage in the codebase.

# Test: Search for usage of `files_only` in the codebase. Expect: No unintended test skips.
rg 'files_only'
Length of output: 55
snakemake_storage_plugin_gcs/__init__.py (3)
387-400: Refactoring of store_object improves clarity and efficiency.

The delegation of bucket existence checks and separate handling of directory uploads enhance the method's readability and maintainability.

448-460: Improved directory deletion logic approved.

The remove method's update to delete all blobs matching a directory's prefix enhances the robustness of the deletion process.

Tools

Ruff

448-448: Missing type annotation for self in method

(ANN101)

410-416: Consider adding a type annotation for self.

The ensure_bucket_exists method improves modularity by handling bucket existence checks. Consider adding a type annotation for self.
def ensure_bucket_exists(self) -> None:
Likely invalid or redundant comment.

Tools

Ruff

410-410: Missing type annotation for self in method

(ANN101)

tests/tests.py

snakemake_storage_plugin_gcs/__init__.py

johanneskoester · 2024-08-19T09:44:38Z

Thanks a lot for the fixes!

johanneskoester · 2024-08-19T09:45:16Z

and sorry for the slow reply. This somehow slipped my attention. Feel free to contact me via discord or email if something like that happens again in the future.

🤖 I have created a release *beep* *boop* --- ## [1.1.0](v1.0.0...v1.1.0) (2024-08-19) ### Features * list storage candidates ([#30](#30)) ([c7ba28e](c7ba28e)) ### Bug Fixes * fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. ([#41](#41)) ([27c80dc](27c80dc)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

jjjermiah added 5 commits April 26, 2024 12:06

fix: directory() issue arising from uploading into the wrong gcs path.

a21a4f5

feat: add logging capabilities for future WMS work

f51f586

chore: format with black

f9cf4eb

fix: refactor remove method for directories

3300cf4

tests: add method in tests.py for directory

1b1f738

jjjermiah marked this pull request as draft April 26, 2024 18:35

jjjermiah mentioned this pull request Apr 26, 2024

using directory() issues #31

Closed

chore: clean up __pycache__, and lint. fix tests

244ca41

jjjermiah marked this pull request as ready for review April 26, 2024 20:00

chore: remove pycache

cb14b4c

vsoch requested a review from johanneskoester April 27, 2024 00:39

Merge branch 'main' into main

d8ee87d

johanneskoester reviewed Aug 19, 2024

View reviewed changes

coderabbitai bot reviewed Aug 19, 2024

View reviewed changes

tests/tests.py Outdated Show resolved Hide resolved

fmt

2300bab

coderabbitai bot reviewed Aug 19, 2024

View reviewed changes

snakemake_storage_plugin_gcs/__init__.py Outdated Show resolved Hide resolved

snakemake_storage_plugin_gcs/__init__.py Show resolved Hide resolved

fixes

e97d381

coderabbitai bot reviewed Aug 19, 2024

View reviewed changes

tests/tests.py Show resolved Hide resolved

snakemake_storage_plugin_gcs/__init__.py Show resolved Hide resolved

johanneskoester merged commit 27c80dc into snakemake:main Aug 19, 2024
5 checks passed

github-actions bot mentioned this pull request Aug 19, 2024

chore(main): release 1.1.0 #49

Merged

coderabbitai bot mentioned this pull request Oct 3, 2024

fix: download_directory local path incorrectly stored #54

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41

fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41

jjjermiah commented Apr 26, 2024 •

edited by coderabbitai bot

Loading

vsoch commented Apr 26, 2024

jjjermiah commented Apr 26, 2024

coderabbitai bot commented Aug 19, 2024 •

edited

Loading

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

johanneskoester Aug 19, 2024

coderabbitai bot left a comment

coderabbitai bot left a comment

coderabbitai bot left a comment

johanneskoester commented Aug 19, 2024

johanneskoester commented Aug 19, 2024

fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41

fix: fixing the directory() issue due to uploading the local_prefix instead of GCS prefix. #41

Conversation

jjjermiah commented Apr 26, 2024 • edited by coderabbitai bot Loading

Summary by CodeRabbit

vsoch commented Apr 26, 2024

jjjermiah commented Apr 26, 2024

coderabbitai bot commented Aug 19, 2024 • edited Loading

Walkthrough

Changes

Sequence Diagram(s)

Chat

CodeRabbit Commands (invoked as PR comments)

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

johanneskoester Aug 19, 2024

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

coderabbitai bot left a comment

Choose a reason for hiding this comment

johanneskoester commented Aug 19, 2024

johanneskoester commented Aug 19, 2024

jjjermiah commented Apr 26, 2024 •

edited by coderabbitai bot

Loading

coderabbitai bot commented Aug 19, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)