add hatrac-async-migrate tool and supporting backend API changes #70
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This tool does not assume that two hatrac servers exist, nor does it call any hatrac REST API to move data. Instead, it exports work from the service database into sqlite3 work file(s) to be processed by workers that have access to both source and destination backing stores. Progress is recorded back to the sqlite work file(s). The existing (source) service can operate unaffected while the workers replicate bulk data to the destination store. A final merge step is performed with the service offline in order to reintegrate destination storage information back into the service database. After merging, the service should be restarted with the destination storage configuration. The backend API enhancements allow the migration tool to supply existing object-version IDs during replication, so that the bulk storage may embed this predefined version ID in the storage name rather than generating a new random ID, where applicable.
warn about problems instantiating service singletons warn about problems importing rest module but continue so code can be used for migration workers etc.
these swapped some validation rules to force failures, but are not appropriate for an actual migration task
show brief usage along with usage error when catching exceptions add explicit help sub-command to get full help
This is an experts-only function that can pattern match which objects in the hatrac service DB we want to migrate.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This tool does not assume that two hatrac servers exist, nor does it call any hatrac REST API to move data. Instead, it exports work from the service database into sqlite3 work file(s) to be processed by workers that have access to both source and destination backing stores. Progress is recorded back to the sqlite work file(s).
The existing (source) service can operate unaffected while the workers replicate bulk data to the destination store. A final merge step is performed with the service offline in order to reintegrate destination storage information back into the service database. After merging, the service should be restarted with the destination storage configuration.
The backend API enhancements allow the migration tool to supply existing object-version IDs during replication, so that the bulk storage may embed this predefined version ID in the storage name rather than generating a new random ID, where applicable.