Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨(frontend) persistance-documents #62

Merged
merged 6 commits into from
May 24, 2024
Merged

Conversation

AntoLC
Copy link
Collaborator

@AntoLC AntoLC commented May 16, 2024

Purpose

This PR manage the persistance of the documents.
We save the documents in different ways:

  • when the user close the tab or the browser
  • when the user leave the page (go to another pad by example)
  • every 1 minute

  • We save the documents only if the pad has been modified.
  • Documents are collaborative, to not save multiple times the same pad, we save the pad only if the user is the last to have modified the pad.

Because of the collaborative aspect of the pads, the best way to store our pad is to save the Y.Doc, to do so the recommended way is to convert the Y.Doc to a Uint8Array and then to a string (base64).
Our pad are saved as a string in a object in a Minio bucket.

Proposal

  • Save the documents
  • Restore the documents
  • tests frontend
  • Minio set version expiration

@AntoLC AntoLC self-assigned this May 16, 2024
@AntoLC AntoLC force-pushed the feature/persistance-documents branch 4 times, most recently from 8df6c1f to 400ff01 Compare May 21, 2024 10:11
@AntoLC AntoLC added the backend label May 21, 2024
@AntoLC AntoLC requested a review from sampaccoud May 21, 2024 10:16
@AntoLC AntoLC force-pushed the feature/persistance-documents branch 2 times, most recently from b2874ea to 3be4ad5 Compare May 21, 2024 12:41
@AntoLC AntoLC marked this pull request as ready for review May 21, 2024 12:45
@AntoLC AntoLC force-pushed the feature/persistance-documents branch from 3be4ad5 to d53af72 Compare May 21, 2024 12:46
@AntoLC AntoLC changed the title ✨ persistance-documents ✨(frontend) persistance-documents May 21, 2024
- id: Expiration
status: enabled
expiry:
noncurrentDays: 100
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so if you do that all versions will be deleted after 100 days and we keep only the last version?

Copy link
Collaborator Author

@AntoLC AntoLC May 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it is that: https://min.io/docs/minio/linux/administration/object-management/create-lifecycle-management-expiration-rule.html, it should make the deletion automatic.
By default we can have 10000 versions, it is about 5kg per version (even less) gives us max about 50mo per doc.

After, I think we should add a system where the user can save voluntary a version (copy of doc ?), and this "version" will not have an expiration.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, for the moment we are not sure how to setup this config with our host. I think we can remove this commit for the moment (🔧(minio) set version expiration) and add it in another PR.

@AntoLC AntoLC force-pushed the feature/persistance-documents branch from d53af72 to 7aa945b Compare May 22, 2024 08:44
@AntoLC AntoLC requested a review from sampaccoud May 22, 2024 08:47
AntoLC added 6 commits May 24, 2024 10:16
We stopped throwing errors on id pages, often
Next.js just need a rerender to get the id
from the router correctly.
Create a react query hook to update a pad.
PATCH /documents/:id
Adapt some pad types.
This commit manage the persistance of the pad.
We save the pad in different ways:
- when the user close the tab or the browser
- when the user leave the page (go to another pad
  by example)
- every 1 minute
----
- We save the pad only if the pad has been modified.
- Pads are collaborative, to not save multiple
  times the same pad, we save the pad only if
  the user is the last to have modified the pad.
----
Because of the collaborative aspect of the pads,
the best way to store our pad is to save the
Y.Doc, to do so the recommended way is to convert
the Y.Doc to a Uint8Array and then to a string
(base64). Our pad are saved as a string in a
object in a Minio bucket.
It is the webrtc server that will be responsible
for managing the data of the BlockNote Editor.

So to restore the data of a saved pads, we have to
add the initial pad to the room of the webrtc
server, it will then update the data of BlockNote
Editor.
The webrtc server accept Y.Doc, so we
convert our base64 data from Minio to make it
compatible with the server.

By doing so, we avoid the problem of
data lost when multiple users are connecting
and one user already updated the data of the
BlockNote Editor before saving it.
Saved documents has to be a string now.
Before it has to be a json object.
@AntoLC AntoLC force-pushed the feature/persistance-documents branch from 7aa945b to cfa0983 Compare May 24, 2024 08:16
@AntoLC AntoLC enabled auto-merge (rebase) May 24, 2024 08:24
@AntoLC AntoLC merged commit 5c9e4ab into main May 24, 2024
16 of 17 checks passed
@AntoLC AntoLC deleted the feature/persistance-documents branch May 24, 2024 08:27
@AntoLC AntoLC mentioned this pull request Jul 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants