Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate data from https://talk.zketh.io #1

Open
jacque006 opened this issue Dec 12, 2024 · 2 comments · May be fixed by #10
Open

Migrate data from https://talk.zketh.io #1

jacque006 opened this issue Dec 12, 2024 · 2 comments · May be fixed by #10
Assignees
@jacque006

Comments

@jacque006
Copy link
Collaborator

jacque006 commented Dec 12, 2024 •
edited
Loading

Write a script or define a manual process which can take the SledDB data & static uploaded assets (images, etc.) from an existing freedit instance (specifically https://talk.zketh.io) and merge them into our instance.

Considerations

  • User, Post, and other entry ids may need to be updated to not conflict with existing users/posts/etc.
  • If a user's username from the incoming data matches an existing user, use that user's id instead
  • If done via script this can be a one-shot and does not need to be very robust given the small amount of data.
  • If this script could be useful to other freedit hosters/admins, consider backporting/contributing it to https://github.com/freedit-org/freedit
@jacque006 jacque006 self-assigned this Dec 12, 2024
sripwoud pushed a commit that referenced this issue Dec 13, 2024
@chancehudson @sripwoud
feat: create snapshot every 30 minutes (#1)
98ef178
@sripwoud sripwoud mentioned this issue Dec 15, 2024
Closed
sripwoud pushed a commit that referenced this issue Dec 15, 2024
@chancehudson
feat: create snapshot every 30 minutes (#1)
d8ffa0b
sripwoud pushed a commit that referenced this issue Dec 15, 2024
@chancehudson @sripwoud
feat: create snapshot every 30 minutes (#1)
76bd9ac
sripwoud added a commit that referenced this issue Dec 15, 2024
@sripwoud @chancehudson
chore: merge pseXperiments/main (#4)
34c1835 
* chore: gitignore ds_store

* feat: create snapshot every 30 minutes (#1)

* snapshot pruning (#2)

* feat: snapshot pruning logic, use unix timestamps instead of whatever jiff nonsense

* chore: cleanup

* feat: create snapshots atomically in the fs

* chore: scaffold monorepo (#4)

* chore: move rs code into `apps/server`

* chore: scaffold vite/react app

* feat: render raw html served by rust server in react app

* chore: use `just` as tasks runner

* doc: add main README

* fix `dev-client` just script

* refactor: write default config dirs relatively to exec bin

---------

Co-authored-by: vimwitch <[email protected]>
@jacque006
Copy link
Collaborator Author

Current approach:

  1. Add a path to config for a second (incoming) sled db file
  2. On server startup, spawn a process to load the incoming file and begin processing it
  3. Check for a Migration entry on the incoming sled db. If it is present, skip below and return.
  4. Move only User, Inn, Post, Solo, & Comment entries
    a. If a user with username already exists, keep a mapping of previous to new user ids.
    b. Re-map user ids on other entries to match current db users
  5. Add a Migration entry to the incoming db & complete

@jacque006 jacque006 linked a pull request Dec 17, 2024 that will close this issue
Draft
8 tasks
@jacque006
Copy link
Collaborator Author

jacque006 commented Dec 18, 2024 •
edited
Loading

Playbook for running the migration

  1. Upload migration db & static files to VM. scp or similar.
  2. Stop the current freedit instance/server
  3. Backup current db cp -r ./apps/server/freedit.db/ "./apps/server/backup-$(date +%s).db"
  4. Add an entry to ./apps/server/config.toml with the migration db. Note this must be an absolute path. migration_db = "/home/pse-forums/apps/server/migration.db"
  5. Restart freedit
  6. Monitor migration logs near the end of startup for nominal output.
  7. Go to forum url and spot check & smoke test expected data was migrated.

In case of migration failure

  1. Stop freedit
  2. Remove migration_db from ./apps/server/config.toml`
  3. Restore the previous db cp -r ./apps/server/backup-...db/ ./apps/server/freedit.db
  4. Restart freddit
  5. Go to forum url and spot check & smoke test previous data was restored

Clean up

  1. Remove migration db & static files
  2. Remove migration_db from ./apps/server/config.toml`
  3. Remove ./apps/server/backup-...db/. It would be a good idea to hold onto this for a few days in case something unexpected occurs.

Open questions

  • Can the static/uploaded files be dropped in as is, or do they need to be migrated as well?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jacque006 jacque006

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

WIP Migrate data from existing freedit instance privacy-scaling-explorations/pse-forums
1 participant
@jacque006