Description
Step 2: Describe your environment
- Operating System version: Linux Ubuntu 22.04.1
- Firebase SDK version: 12.5.4
- Firebase Product: Storage
- Python version: 3.10.12
- Pip version: 22.0.2
Step 3: Describe the problem
I am using the Firebase Emulator Suite - Storage and firebase-admin in Python.
I wand to upload files to the Firebase Emulator Suite - Storage and I already set the system's environment variable STORAGE_EMULATOR_HOST to http://localhost:9199.
When I use the function blob.upload_from_filename(filename=file_path) and I upload a file that is smaller than the _DEFAULT_CHUNKSIZE (100 MB), which is defined in blob.py, everything is fine. But if my file exceeds the chunksize (> 100 MB), the file will not be uploaded correctly.
I tried to upload a 200 MB large file via the Firebase Emulator Suite UI in my browser (http://127.0.0.1:4000/storage/<project-id>) and it worked. So the problem is not the emulator itsself.
These are the files I want to upload, which are larger than 700 MB per file.
This is the UI after uploading two of the files. The maximum file size I can reach is 104.86 MB...
The Firebase Emulator Suite does not throw any error, neither my code does.
Relevant Code:
In the following code I am creating the Firebase app object in the __init__ function and use the storage bucket to upload files using the function upload_from_filename() in my custom function upload_zip_from_path():
def __init__(self, app_name: str = "[DEFAULT]", config_yml_path: str = "config.yml"):
configs = FileUtils.load_config_yml(config_yml_path)
self.DEBUG = bool(configs["environment"]["debug"])
self.DATABASE_URL = configs["database"]["url"]
self.BUCKET_URL = configs["storage"]["url"]
token_file_path = configs["database"]["token_file_path"]
# Set host to local Firebase Emulator Suite Storage
if self.DEBUG:
os.environ["STORAGE_EMULATOR_HOST"] = "http://localhost:9199"
try:
self.APP_OBJ = firebase_admin.get_app(app_name)
except ValueError:
# If app does not exist, a value error is thrown
if len(token_file_path) > 0:
# Production: Authentication token needed
token = credentials.Certificate(token_file_path)
self.APP_OBJ = firebase_admin.initialize_app(
credential=token,
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
else:
# Development: No token needed using Firebase Emulator Suite
self.APP_OBJ = firebase_admin.initialize_app(
options={
"databaseURL": self.DATABASE_URL,
"storageBucket": self.BUCKET_URL,
},
name=app_name,
)
# References
self.SID_ROOT = db.reference("sid")
self.USER_ROOT = db.reference("user")
# Firebase app specific
self.APP_NAME = app_name
self.BUCKET = storage.bucket()
def upload_zip_from_path(self, zip_path:str):
# Set file name
if "/" in zip_path:
zip_name = zip_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
zip_name = zip_path
# Get a reference to the Firebase Storage bucket
zip_folder = self.BUCKET.blob(f"zip/{zip_name}")
zip_folder.upload_from_filename(zip_path)
return zip_folder.public_url
On the other hand I tried to chunk the file by myself, but this did not went well.
With the following code the current chunk will overwrite the last chunk - this is not wanted... I wanted to append it to the file.
def resumable_upload(self, file_path, chunk_size=25 * 1024 * 1024):
# Set file name
if "/" in file_path:
destination_blob_name = file_path.split("/")[-1] # "test/abc.zip" -> ["test", "abc.zip"] -> "abc.zip"
else:
destination_blob_name = file_path
# Get a reference to the Firebase Storage bucket
bucket = storage.bucket()
# Create a resumable upload session
blob = bucket.blob(destination_blob_name)
blob.chunk_size = chunk_size
chunk_count = 0
chunk_start = 0
total_size = os.path.getsize(file_path)
with open(file_path, 'rb') as file:
while chunk_start < total_size:
chunk_end = min(chunk_start + chunk_size, total_size)
chunk_data = file.read(chunk_size)
blob.upload_from_string(chunk_data, content_type="application/zip")
chunk_start = chunk_end
chunk_count += 1
print(f'Uploaded chunk {chunk_count} / {total_size // chunk_size} ({chunk_start}/{total_size} bytes)')
print('Resumable upload completed.')
return blob.public_url