Migrate data from object storage without any service downtime

[yujunz]

What would you like to be added:

Import data from s3 storage without service interruption

Why is this needed:

We have PBs data stored in QingStor and need to move to swift. It would take a long time for such scale and pretty challenging as services are continuously updating the data.

QingStor does provide a tool to migrate data in but not out apparently. I wonder if we can implement such feature in juicefs s3 gateway to seamlessly migrate data from object storage without any service downtime.

[yujunz]

A brief design proposal

1. Create juicefs on target object storage
2. Configure source object storage in juciefs s3 gateway
3. Switch service to access juicefs s3 gateway instead of source object storage.
4. Run juicefs sync to replicate data from source object storage to juicefs in background, skip if exists.
5. For all GET requests, check if it exists in juicefs.
   5.1 if not, fetch data from source and save in juicefs and continue on 5.2
   5.2 if yes, read file from juicefs for response
6. For all PUT requests, check if it exists in juicefs
   6.1 if not, fetch data from source, save in juicefs and continue on 6.2
   6.2 if yes, write data to juicefs
7. For all DELETE requests, check if it exists in juicefs
   7.1 if not, fetch data from source, save in juicefs and continue on 7.2
   7.2 if yes, delete file from juicefs
   7.3 alternatively skip saving in juicefs and simply delete in source object
8. For all LIST requests, check if it is cached in juicefs
   8.1 if not, fetch result from source, cache in juicefs and continue on 8.2
   8.2 if yes, merge result from cache and juicefs, when conflict, juicefs takes precedence
9. Wait for juicefs sync to complete, disable source object storage

[yujunz]

See also https://github.com/caiobrentano/swift-s3-sync

[davies]

As we discussed offline, it's better to create an dedicated tool to do this, which could be built on top of juicefs gateway and library for object storages.

[yujunz]

A summary of discussion. The proposal above based on fallback reading doesn't support rollback during the migration. In case of any API compatibility issues, it won't be easy to switch back to source.

A more practical design might be double write to source and target and sync the existing objects in background. Switch service to target when no more diff is detected during a full sync or several. Continue double write after switch for a period until all services becomes stable. We may rollback to source anytime before the stopping double write.

[yujunz]

Brief execution plan

Proxy: switch writer to proxy for double write, , revert writer back to source on failure.
Sync: Sync data in background, repeat syncing until no more diff detected in a complete run
Reader: switch reader to target, revert reader back to source on failure
Writer: switch writer to target, no more double write, sync back and revert on failure
Decommission: retire source

Could you help review? @davies