-
Notifications
You must be signed in to change notification settings - Fork 921
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Gcs database #12817
base: main
Are you sure you want to change the base?
Gcs database #12817
Changes from 7 commits
5d7ffa4
950bd1a
bcb41e0
61732af
5cfcb44
1b9a20c
ff4e7f1
8065eee
b7ca03a
c3b0cc5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,12 +5,13 @@ | |
# file, You can obtain one at https://mozilla.org/MPL/2.0/. | ||
|
||
import os | ||
from pathlib import Path | ||
import sys | ||
from time import time | ||
|
||
import boto3 | ||
from boto3.exceptions import Boto3Error | ||
from db_s3_utils import ( | ||
from db_utils import ( | ||
DB_FILE, | ||
JSON_DATA_FILE, | ||
JSON_DATA_FILE_NAME, | ||
|
@@ -19,10 +20,22 @@ | |
get_prev_db_data, | ||
set_db_data, | ||
) | ||
from google.cloud import storage | ||
|
||
# ROOT path of the project. A pathlib.Path object. | ||
ROOT_PATH = Path(__file__).resolve().parents[1] | ||
ROOT = str(ROOT_PATH) | ||
|
||
# add bedrock to path | ||
sys.path.append(ROOT) | ||
|
||
# must import after adding bedrock to path | ||
from bedrock.base.config_manager import config # noqa | ||
|
||
CACHE = {} | ||
BUCKET_NAME = os.getenv("AWS_DB_S3_BUCKET", "bedrock-db-dev") | ||
REGION_NAME = os.getenv("AWS_DB_S3_REGION", "us-west-2") | ||
UPLOAD_TO_GCS = config("UPLOAD_TO_GCS", parser=bool, default="false") | ||
|
||
|
||
# Requires setting some environment variables: | ||
|
@@ -41,6 +54,15 @@ def s3_client(): | |
return s3 | ||
|
||
|
||
def gcs_client(): | ||
gcs = CACHE.get("gcs_client") | ||
if not gcs: | ||
gcs = storage.Client() | ||
CACHE["gcs_client"] = gcs | ||
|
||
return gcs | ||
|
||
|
||
def delete_s3_obj(filename): | ||
s3 = s3_client() | ||
s3.delete_object(Bucket=BUCKET_NAME, Key=filename) | ||
|
@@ -63,6 +85,18 @@ def upload_db_data(db_data): | |
except Boto3Error: | ||
return f"ERROR: Failed to upload the new database info file: {db_data}" | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we want anything around the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. My idea was to do kind of a multi step deploy:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yep, sounds good to me. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @bkochendorfer @pmac Can we get this rolled out to prod? Good to do ahead of the postgres/cms tango |
||
if UPLOAD_TO_GCS: | ||
gcs = gcs_client() | ||
bucket = gcs.bucket(BUCKET_NAME) | ||
|
||
# upload the database | ||
db_file = bucket.blob(db_data["file_name"]) | ||
db_file.upload_from_filename(DB_FILE, predefined_acl="public-read") | ||
|
||
# upload the json metadata | ||
db_file_info = bucket.blob(JSON_DATA_FILE_NAME) | ||
db_file_info.upload_from_filename(JSON_DATA_FILE, predefined_acl="public-read") | ||
|
||
return 0 | ||
|
||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we have
config
now, we could optionally convert these (or anywhere else in this file where we useos.getenv
). An example of a string value would just be:Sorry to tack on so much extra.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No problem at all!