- Implementation Owner: @everly-gif
- Start Date: 17/01/2022
- Target Date: 17/04/2022
- Appwrite Issue: NA
- LinkedIn : Everly Precia Suresh
- Resume/CV : View Here
As part of the ongoing process of implementing multiple storage support to make appwrite agnostic, I propose to implement a BackBlaze and Linode Object storage adapter. BackBlaze B2 and Linode object storage has been a go to alternative for AWS S3 among users. By implementing these adapters, the user will now be able to choose to store their files in Backblaze B2 or Linode Object Storage.
What problem are you trying to solve?
By default, Appwrite uses a docker volume to store the user's files which restricts the user to only one choice of storage. It also makes the user's application incompatible to deploy to modern day platforms like Heroku. Therefore, a need for storage adapters to support multiple storage services is required. By implementing storage adapters for Backblaze and Linode , the users will now have a choice to choose from a different storage service with a freedom to deploy to modern day platforms such as Heroku. Hence, making the platform agnostic.
BackBlaze and Linode provide a S3 compatible API that can be utilized in this implementation.
I would divide the implementation of storage adapters into the following steps. By implementing these functions one can effectively implement storage adapters.
- Authentication: Providing account/bucket/file access.
- Bucket Management: Creating and managing the buckets that hold files.
- Upload: Sending files to the cloud.
- Download: Retrieving files from the cloud.
- List: Data checking/selection/comparison.
Two new PHP files to be created :
- src/Storage/Device/LOS.php
- src/Storage/Device/BackBlaze.php
Both of the storage implementation will be extending from the S3 adapter class. Therefore, we can utilize the already implemented AWS V4 signatures for authentication and other functions in the Backblaze and Linode implementation.
For LOS(Linode object storage) ..Device/LOS.php
:
namespace Utopia\Storage\Device;
use Utopia\Storage\Device\S3;
class LOS extends S3
{
/*
Implementation
*/
}
For BackBlaze ..Device/BackBlaze.php
:
namespace Utopia\Storage\Device;
use Utopia\Storage\Device\S3;
class BackBlaze extends S3
{
/*
Implementation
*/
}
The next step would be to implement region constants. Although both of these implementations are s3 compatible the regions these cloud storage provides vary from AWS s3.
Regions for LOS : Atlanta (USA), Frankfurt (Germany), Newark (USA), and Singapore.
Regions for BackBlaze B2 : Sacramento California, Phoenix Arizona, and Amsterdam Netherlands.
Example:
const USWEST0 = 'us-west-000';
The next step would be the implement the constructors. We will pass in the values of root, accessKey, secretKey, bucket, region and acl to complete the implementation.
Example:
public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::USWEST0, string $acl = self::ACL_PRIVATE)
{
parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
$this->headers['host'] = $bucket . '.s3' . '.' . $region . '.backblazeb2.com'; /*the bucket url for BackBlaze stands similar for LOS */
}
The above implements the core functionality of the adapter. The final thing to implement would be to just set the name and description for each of the adapters. After which Updating this information among docs / other files will conclude it.
Below are some of the docs/files I will be editing:
- I will be editing
src/Storage/Storage.php
to add Linode and BackBlaze under the supported devices. - I will be editing the
Readme.md
to hold updated information about the implementation. - I will be writing tests to validate my implementation. Hence, will be adding a new file under
tests/Storage/Device/<storage-adapter-name>.php
.
There won't be any need for new API Endpoints.
There won't be any need for new Data Structures.
There won't be any need for new supporting Libraries.
Since the implementations extends from the S3 adapter class, if there are some changes in the S3 file , especially new API calls that might not be supported by the S3 compatible API provided by BackBlaze and Linode then there is a chance for my implementations to not function. In such a case, the code can be modified with alternatives.
This feature can be benchmarked on the basis if the user is able to choose among multiple storage providers and upload their files to the respective buckets.
I will be using PHPUnit to test this new feature. I will be making uploads and deletion to the buckets and validate its functionality.
The Readme for the utopia/storage repo would support this feature.
For LOS : https://www.linode.com/docs/api/object-storage
For BackBlaze: https://www.backblaze.com/b2/docs/s3_compatible_api.html
There won't be any need to update the contribution guide. But updating the Readme is required.
3. What tutorials (text/video) might help developers understand this feature scope, capabilities, and possible use-cases?
For LOS : https://www.linode.com/docs/api/object-storage
For BackBlaze: https://www.backblaze.com/b2/docs/s3_compatible_api.html
The Merged PR for Digital Ocean Spaces Adapter is a great reference itself : utopia-php/storage#5
- https://blog.cloudflare.com/backblaze-b2-and-the-s3-compatible-api-on-cloudflare/
- https://www.backblaze.com/blog/whats-the-diff-backblaze-s3-compatible-api-vs-b2-native-api/
- https://help.backblaze.com/hc/en-us/articles/360047815993-Does-the-B2-S3-Compatible-API-support-Pre-Signed-URLs
- https://www.linode.com/community/questions/19168/how-do-i-integrate-linode-object-storage-in-php
- utopia-php/storage#5
- https://youtu.be/q88OKsr5l6c
NA
- Timeline of implementation
- Understanding the different codebase among different repo
- Tests
- Since the S3 Adapter has been setup for appwrite, I can continue to implement other S3 compatible APIs among different Cloud Storage Providers during my tenure, this gives the user plenty of options to choose from.