Hot-Reloading Feature Implementation

[mohamedsalem401]

Here is the pull request, which introduces hot-reloading functionality using Chokidar.js. I've invested time and effort to simplify the new feature's architecture and anticipate potential enhancements in the future, such as:

• Collaboration: With our current implementation, individual changes are tracked. which considers the possibility of enabling collaboration in the future.
• Custom Page Types: Our Markdown files are initially converted into JSON objects, allowing testing against custom user schemas or similar features.
• Type Extraction: TypeScript users will appreciate this enhancement, we can generate type definitions (.d.ts files) for individual pages from the Json file.
• Multiple Folders Indexing: A potential challenge is handling links between files located in different folders.

Here's a brief overview of the implementation logic:

1. MD to JSON Conversion: Markdown files are transformed into JSON format, allowing for comparison with their previous versions. Subsequently, we send requests to the database to modify only the necessary parts.
2. Adding a New File: This process includes the following steps:
   • Addition of fileJson
   • Bulk addition of file_tags
   • Bulk addition of tags if they are not already in the database
   • Bulk addition of links
   • Marking a broken link (where the "to" field points to a non-existing file) with the same path as this new file as working and adding it to the database.
3. Changing a File: To update a file, we compare metadata, tags, and links with the old data, followed by sending create or deletion queries to the database.
4. Deleting a File: When a file is deleted, we perform the following actions:
   • Delete fileJson where the ID matches the file ID.
   • Delete file_tags where the file corresponds to the file ID.
   • Remove tags that are no longer used in the file_tags table.
   • Delete links where the from field matches the file ID.
   • Move links where the to field matches the file ID to the list of broken links and remove them from the links table.

Tasks

• Using chokidar.js
• Utilize the --watch flag to initially index the folder and continuously monitor it for file changes.
• Compare changes with the existing data, updating only the modified sections.
• Achieve hot-reloading upon adding, editing, or deleting files, tags, or links.
• Maintain a list of broken links and exclude them from the database.
• If a new file matches the path of a broken link, move this matching link from the broken list to the functioning list and add the new file to the database.
• Update documentation
• Update tests

[changeset-bot]

⚠️ No Changeset found

Latest commit: a7fa720

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[olayway]

@mohamedsalem401 Thank you so much for your contribution! 🎉 I'll review this PR as soon as possible, probably today or tomorrow.

[rufuspollock]

@mohamedsalem401 first this is awesome 🎉 and it works.

In terms of implementation i'm just trying to grok how this works exactly.

For my part, I had been imagining a super-simple implementation like:

• Watch (markdown) files
• When a (markdown) file or files change ... run the markdowndb rebuild script
• Bonus: update markdowndb rebuild script / library so that you can pass it the files to re-process (so that we don't have to rebuild the whole thing)

Here there seems to be a bunch of direct writing to the markdowndb database. Is that right? Plus there seems to be some kind of check to identify if something changed.

And if so, why do we need to do that?

General point: I think we want the do any updating of the database through markdowndb toolchain? (otherwise we risk duplication - and, one day, being out of sync when markdowndb changes)

[mohamedsalem401]

My main concerns are:

1. When a user removes a tag from a file, how can we accurately determine if it should be removed from our records?

2. If a user adds a tag to a file, should we attempt to insert the tag and overlook any errors?

The current proposal suggests updating all related information in three tables (files, file_tags, links) each time a file is modified. This process involves:

• Querying the files table for updates.
• Executing two queries: one to delete tag fields in the file_tags table and one to add new file tags.
• Two queries for updating links (deleting old links and adding new links).

Additionally, implementing a feature to avoid adding links until a file with the corresponding ID appears in a link requires:

• Maintaining a list of file IDs and broken links.
• Queries on all links to identify broken links during file removal.

While this might not be a significant concern for a local database, it could pose challenges if these queries are performed on a remote database, especially considering the frequency of updates during every keystroke in the markdown editing process.

If querying on the database isn't a big deal, we could consider removing and then adding every changed file. This will make the code simpler. What do you think of this idea?

[mohamedsalem401]

Here there seems to be a bunch of direct writing to the markdowndb database. Is that right? Plus there seems to be some kind of check to identify if something changed.

And if so, why do we need to do that?

Well, the idea is to be efficient. Instead of updating the entire file, they want to identify and update only the parts that actually changed. For example, if a user adds a tag, the code compares the new and old versions of the file. Then, it runs just one query on one table (file_tags) to show the new changes. It's a way of making updates by only focusing on what really needs to change.

In simple terms, we compare things to figure out exactly what needs to be updated. This way, we only use the smallest, most necessary update query to make the change happen. It's being super efficient in updating the database

[rufuspollock]

When a user removes a tag from a file, how can we accurately determine if it should be removed from our records?

You run the markdowndb build pipepline for that file.

If a user adds a tag to a file, should we attempt to insert the tag and overlook any errors?

Ditto. And what kind of errors do you imagine?

Well, the idea is to be efficient. Instead of updating the entire file, they want to identify and update only the parts that actually changed. For example, if a user adds a tag, the code compares the new and old versions of the file. Then, it runs just one query on one table (file_tags) to show the new changes. It's a way of making updates by only focusing on what really needs to change.

This is super-cool - but i also wonder if it may be a bit of overkill to start with. I'd build the simplest thing possible and then come back to this if there are problems "YAGNI". In any case it should be

If querying on the database isn't a big deal, we could consider removing and then adding every changed file. This will make the code simpler. What do you think of this idea?

Yes, exactly i would do that. And i would do it in the markdowndb toolchain not in the watcher code.

---

It seems to me here that you may have two (v useful) things combined together:

• "Watch" functionality (that rebuilds the db when file(s) change)
• Improvements to the markdowndb database and build pipeline

[rufuspollock]

OK. I conclusion here is to reintroduce this once we have done #47