Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add: get metadata dict from s3 bucket with last modified date and siz… #69

Merged
merged 6 commits into from
Oct 6, 2023
Merged

add: get metadata dict from s3 bucket with last modified date and siz… #69

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
00ceeeb
add: get metadata dict from s3 bucket with last modified date and siz…
Sep 13, 2023
da8fc46
fix: quack ai review
Sep 15, 2023
3c6c03d
fix: quack ai review
Sep 15, 2023
b98b139
fix: quack ai review
Sep 15, 2023
8e3da88
fix: quack ai review
Sep 15, 2023
e343eef
fix: file size should be in byte otherwise file size is always 0
Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion pyro_risks/utils/s3.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ class S3Bucket:
aws_access_key_id='my_access_key_id',
aws_secret_key='my_secret_key'
)

NOTE: credentials should never be in the code.

To upload a file to the bucket, use:
Expand Down Expand Up @@ -133,3 +133,28 @@ def get_file_metadata(self, object_key):
obj = self.bucket.Object(object_key)
metadata = obj.metadata
return metadata

def get_files_metadata(self, pattern=None):
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind adding type annotations? 🙏

Pattern explanation 👈 In Python, type annotations are not enforced. But that can be valuable for debugging using mypy, for your collaborators' understanding, and the autocomplete of your IDE. For more details, feel free to check this resource.

Code example

Here is an example of how this is typically handled:

# def is_upper(variable_name):
def is_upper(variable_name: str) -> bool:
Quack feedback loop 👍👎 This comment is about [missing-type-annotations]. Add the following reactions on this comment to let us know if:
- 👍 that comment was on point
- 👀 that doesn't seem right
- 👎 this isn't important for you right now
- 😕 that explanation wasn't clear

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind adding type annotations? 🙏

Pattern explanation 👈 In Python, type annotations are not enforced. But that can be valuable for debugging using mypy, for your collaborators' understanding, and the autocomplete of your IDE. For more details, feel free to check this resource.

Code example

Here is an example of how this is typically handled:

# def is_upper(variable_name):
def is_upper(variable_name: str) -> bool:
Quack feedback loop 👍👎 This comment is about [missing-type-annotations]. Add the following reactions on this comment to let us know if:
- 👍 that comment was on point
- 👀 that doesn't seem right
- 👎 this isn't important for you right now
- 😕 that explanation wasn't clear

"""
Lists files in the S3 bucket with their size in GB and last modified dates.

Args:
pattern (str): The pattern to filter files by (optional).

Returns:
A dictionnary of file keys (paths), file sizes en GB and last modified dates in the bucket.
"""
files = []
for obj in self.bucket.objects.all():
if not pattern or pattern in obj.key:
file_name = obj.key
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like we need this intermediate variable

Pattern explanation 👈 You shouldn't concede readability, but when it's possible, avoid unnecessary memory allocation. For more details, feel free to check this resource.

Code example

Here is an example of how this is typically handled:

# def make_upper(input_str: str) -> str:
# 	upper_str = input_str.upper()
# 	return upper_str
def make_upper(input_str: str) -> str:
	return input_str.upper()
Quack feedback loop 👍👎 This comment is about [intermediate-variable]. Add the following reactions on this comment to let us know if:
- 👍 that comment was on point
- 👀 that doesn't seem right
- 👎 this isn't important for you right now
- 😕 that explanation wasn't clear

file_size = round(obj.size * 1.0 / (1024*1000000), 2)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like we need this intermediate variable

Pattern explanation 👈 You shouldn't concede readability, but when it's possible, avoid unnecessary memory allocation. For more details, feel free to check this resource.

Code example

Here is an example of how this is typically handled:

# def make_upper(input_str: str) -> str:
# 	upper_str = input_str.upper()
# 	return upper_str
def make_upper(input_str: str) -> str:
	return input_str.upper()
Quack feedback loop 👍👎 This comment is about [intermediate-variable]. Add the following reactions on this comment to let us know if:
- 👍 that comment was on point
- 👀 that doesn't seem right
- 👎 this isn't important for you right now
- 😕 that explanation wasn't clear

file_last_modified = obj.last_modified
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't look like we need this intermediate variable

Pattern explanation 👈 You shouldn't concede readability, but when it's possible, avoid unnecessary memory allocation. For more details, feel free to check this resource.

Code example

Here is an example of how this is typically handled:

# def make_upper(input_str: str) -> str:
# 	upper_str = input_str.upper()
# 	return upper_str
def make_upper(input_str: str) -> str:
	return input_str.upper()
Quack feedback loop 👍👎 This comment is about [intermediate-variable]. Add the following reactions on this comment to let us know if:
- 👍 that comment was on point
- 👀 that doesn't seem right
- 👎 this isn't important for you right now
- 😕 that explanation wasn't clear

files.append(
{
"file_name": file_name,
"file_size": file_size,
"file_last_modified": file_last_modified,
}
)
return files
Loading