Skip to content

Latest commit

 

History

History
265 lines (191 loc) · 11.4 KB

019-multi-storage-support.md

File metadata and controls

265 lines (191 loc) · 11.4 KB
Raw

Multi Storage Support (BackBlaze and Linode Object Storage)

Summary

As part of the ongoing process of implementing multiple storage support to make appwrite agnostic, I propose to implement a BackBlaze and Linode Object storage adapter. BackBlaze B2 and Linode object storage has been a go to alternative for AWS S3 among users. By implementing these adapters, the user will now be able to choose to store their files in Backblaze B2 or Linode Object Storage.

Problem Statement (Step 1)

What problem are you trying to solve?

By default, Appwrite uses a docker volume to store the user's files which restricts the user to only one choice of storage. It also makes the user's application incompatible to deploy to modern day platforms like Heroku. Therefore, a need for storage adapters to support multiple storage services is required. By implementing storage adapters for Backblaze and Linode , the users will now have a choice to choose from a different storage service with a freedom to deploy to modern day platforms such as Heroku. Hence, making the platform agnostic.

Design proposal (Step 2)

BackBlaze and Linode provide a S3 compatible API that can be utilized in this implementation.

Approach

I would divide the implementation of storage adapters into the following steps. By implementing these functions one can effectively implement storage adapters.

  • Authentication: Providing account/bucket/file access.
  • Bucket Management: Creating and managing the buckets that hold files.
  • Upload: Sending files to the cloud.
  • Download: Retrieving files from the cloud.
  • List: Data checking/selection/comparison.

Implementation and Dependencies

Two new PHP files to be created :

  • src/Storage/Device/LOS.php
  • src/Storage/Device/BackBlaze.php

Both of the storage implementation will be extending from the S3 adapter class. Therefore, we can utilize the already implemented AWS V4 signatures for authentication and other functions in the Backblaze and Linode implementation.

For LOS(Linode object storage) ..Device/LOS.php:

namespace Utopia\Storage\Device;

use Utopia\Storage\Device\S3;

class LOS extends S3
{
/* 
Implementation
*/
}

For BackBlaze ..Device/BackBlaze.php:

namespace Utopia\Storage\Device;

use Utopia\Storage\Device\S3;

class BackBlaze extends S3
{
/* 
Implementation
*/
}

The next step would be to implement region constants. Although both of these implementations are s3 compatible the regions these cloud storage provides vary from AWS s3.

Regions for LOS : Atlanta (USA), Frankfurt (Germany), Newark (USA), and Singapore.
Regions for BackBlaze B2 : Sacramento California, Phoenix Arizona, and Amsterdam Netherlands.

Example:

const USWEST0 = 'us-west-000';

The next step would be the implement the constructors. We will pass in the values of root, accessKey, secretKey, bucket, region and acl to complete the implementation.

Example:

public function __construct(string $root, string $accessKey, string $secretKey, string $bucket, string $region = self::USWEST0, string $acl = self::ACL_PRIVATE)
    {
        parent::__construct($root, $accessKey, $secretKey, $bucket, $region, $acl);
        $this->headers['host'] = $bucket . '.s3' . '.' . $region . '.backblazeb2.com'; /*the bucket url for BackBlaze stands similar for LOS */
    }

Editing Other files

The above implements the core functionality of the adapter. The final thing to implement would be to just set the name and description for each of the adapters. After which Updating this information among docs / other files will conclude it.

Below are some of the docs/files I will be editing:

  • I will be editing src/Storage/Storage.php to add Linode and BackBlaze under the supported devices.
  • I will be editing the Readme.md to hold updated information about the implementation.
  • I will be writing tests to validate my implementation. Hence, will be adding a new file under tests/Storage/Device/<storage-adapter-name>.php.

API Endpoints

There won't be any need for new API Endpoints.

Data Structure

There won't be any need for new Data Structures.

Supporting Libraries

There won't be any need for new supporting Libraries.

Breaking Changes

Since the implementations extends from the S3 adapter class, if there are some changes in the S3 file , especially new API calls that might not be supported by the S3 compatible API provided by BackBlaze and Linode then there is a chance for my implementations to not function. In such a case, the code can be modified with alternatives.

Reliability (Tests & Benchmarks)

Scaling

Benchmarks

This feature can be benchmarked on the basis if the user is able to choose among multiple storage providers and upload their files to the respective buckets.

Tests (UI, Unit, E2E)

I will be using PHPUnit to test this new feature. I will be making uploads and deletion to the buckets and validate its functionality.

Documentation & Content

1. What docs would support this feature?

The Readme for the utopia/storage repo would support this feature.
For LOS : https://www.linode.com/docs/api/object-storage
For BackBlaze: https://www.backblaze.com/b2/docs/s3_compatible_api.html

2. Do we need to update the contribution guide with a new section or a supporting tutorial?

There won't be any need to update the contribution guide. But updating the Readme is required.

3. What tutorials (text/video) might help developers understand this feature scope, capabilities, and possible use-cases?

For LOS : https://www.linode.com/docs/api/object-storage
For BackBlaze: https://www.backblaze.com/b2/docs/s3_compatible_api.html

4. What demo applications can help us demonstrate this feature APIs and capabilities?

The Merged PR for Digital Ocean Spaces Adapter is a great reference itself : utopia-php/storage#5

5. More Resource that helped me understand adapters:

Prior art

NA

Unresolved questions

  • Timeline of implementation
  • Understanding the different codebase among different repo
  • Tests

Future possibilities

  • Since the S3 Adapter has been setup for appwrite, I can continue to implement other S3 compatible APIs among different Cloud Storage Providers during my tenure, this gives the user plenty of options to choose from.