Implement an element-wise versioning

[MartinKjunior]

The table contents will be regularly updated in the future as we work on the website. It would be useful to have a versioning system in place such that we (and the users) could track when and what changes have been made to each element (value) in all of the tables. In other words, have a history of all changes for each value available on click, on top of general website versioning which is also necessary.

[Aditish51]

@MartinKjunior Can you give me more idea on that? I guess we need to setup a database and integrate it with the mkdocs documentation.

[MartinKjunior]

@Aditish51 I would agree. I think as we add more sophisticated tools to the website it might be necessary to build a proper database instead of manually writing everything into the markdown files. That should also allow for easier changes to the tables. The only relevant tool, that I know of, in mkdocs for this is the table-reader, which is essentially just a wrapper for panda's data-reading functions which I've mentioned in #55 .

I don't have much experience setting up such a database. When it comes to turning the existing markdown text data into csv files, that's an easy job; I can send you a quick python script I wrote to get the tabular data. However, I assume there are better solutions to storing the data than having a bunch of csv/excel files and reading them in whenever a user requests a webpage with a table on it. Researching the appropriate solution based on the needs of OSIPI could be a nice long-term project if that's something you're interested in. One of the future applications of this lexicon would be having a webapp where a researcher can put their paper and the app will replace all instances of keywords that exist in the lexicon with a hyperlink that connects that keyword to the appropriate table cell. E.g. if my paper talks about using the "Patlak model", then it would turn that keyword into Patlak model that takes you to the definition. I would assume that to achieve that there would be a need for 1. efficient lookup in the database, 2. versioning of the website (mentioned in #17 ). The versioning is quite important so that if a researcher writes a paper in 2025 and uses the definitions used at that time, then the definition is altered in 2026, a reader in 2027 can go to the 2025 version of the website to know how the author of the paper used the terms. Similar idea with versioning of the individual values. The lexicon may also become used by manufacturers in case they start using the OSIPI codes as a standard reference to various quantities and models, which would probably add a need for SQL-like methods of data lookup.

I think this could be the project for you to work on for GSoC if you're interested.

[MRdep]

@Aditish51 I will add a little too: This is really really important step change in the lexicon that we have been been thinking about for a while. The idea is to migrate the lexicon items to a database and read items from there, instead of from hardcoded .mk files. That way we can implement versioning on each specific lexicon item (I'm not actually sure how this could be done!).

This would be a really great project for GSoc, so if you are interested in this please fill in the project form:
Template proposal OSIPI projects GSoC 2024.pdf

[Aditish51]

Thank you @MartinKjunior and @MRdep for sharing details about the project.This could be one of the best opportunity for me to work on this project.I'm excited about the opportunity to contribute to it as a GSoC participant.

The idea of creating a web application to manage lexicon data and facilitate keyword hyperlinking in research papers is both innovative and valuable. After reading your instructions carefully , the possible solution could be:
A) Migrating lexicon items to database:
We will have to design a database schema that can efficiently store our lexicon data, including versioning information about the lexicon. The possible database system can be either small and lightweight databases like- TinyDB that stores data in JSON format but it is suitable for small scale applications or using large scale DBMS like MySQL or MongoDB.Whenever a lexicon entry is updated, instead of overwriting the existing entry,a new entry is created with the updated definition and increment the version number(in date format).This ensures that historical versions of the lexicon entries are preserved in the database.

B) Creating a complete webapp :
Since MkDocs websites are static sites generated from Markdown files and do not include a backend.To add dynamic functionality to our MkDocs website, such as fetching data from a database, we would typically need to integrate it with a backend. For that we will need a backend framework like flask or django and integrate it to mkdocs frontend or to make the webapp more dynamic we will design the frontend from scratch using Reactjs, bootstrap etc. We will then define API endpoints in our backend server to perform operations on the database. API endpoints in our backend server to allow users to retrieve historical versions of lexicon entries. We will have to implement logic to query the database for specific versions of lexicon entries based on version numbers or timestamps. In our frontend application (e.g., MkDocs website), provide a user interface for accessing historical versions of lexicon entries.

I believe my skills in database management, web development, and version control make me well-suited for this project. I'm eager to learn more about the specific requirements and technologies involved and to collaborate with the OSIPI organization to bring this project to fruition.

[MartinKjunior]

@Aditish51 I am glad you're excited to work on this.

Based on your suggestions I do believe you to have the theoretical knowledge necessary to accomplish the project.

It sounds to me like option B will give us much more developmental freedom in the future. I was wondering if it may be possible to do A and then transition into B? Do you think it would be reasonable to first create the database and hook it up to the existing infrastructure (mkdocs) and afterwards work on developing a frontend in Reactjs or would that be too incompatible and a waste of time when having redo the way frontend and backend communicate?

I think it would be useful if you could summarise these two approaches in a presentation (aim for ~15-20 minutes), showing what steps would need to be done in each, their advantages and disadvantages including an expected completion time and whether you think you could achieve it within the time limit of GSoC. Aim this presentation at a non-software engineer audience as I think it would be good if you presented this in front of the rest of the TaskForce 4.2 where there are clinicians (doctors in hospitals) as well as other researchers and their input could be valuable in guiding the specifics of the project. I believe having that information is crucial to creating a product that is directly useful to the experts in the field of perfusion imaging. I also think interactions with experts from other fields and learning to put your ideas across to them would a valuable learning outcome from the GSoC, if you've never done that before. @MRdep would you agree? If so, could you invite Aditish to the next taskforce meeting (probably not the meeting tomorrow, unless @Aditish51 can get the presentation done in a day but I don't want to rush anyone, Aditish just let us know which date suits you best) so they can present their ideas? The meetings are either tomorrow 20/03/2024 or the next is 17/04/2024.

[MRdep]

Hi @MartinKjunior and @Aditish51.

The timings are not great for presenting to the Task Force, since the GSoc projects need to be submitted by April 2nd. I think best to complete the form with your ideas and we can discuss/develop the project on this thread.

Your ideas sound excellent. Perhaps it would be worth making a plan of how you would spend each week, with deliverables, to determine if it can be delivered in the timeframe.

[Aditish51]

Hello @MartinKjunior and @MRdep ,

Thank you for your appreciation. I'm eager to present my idea to TaskForce 4.2 . However, I have a prior commitment and will be traveling on the 20th of March. Unfortunately, this means I won't be available for a meeting on that day.
But I would be honored to present my idea briefly during the meeting scheduled for the 17th of April. During this presentation, I plan to discuss the advantages, disadvantages, and timeline of the project within the GSoC time limit.

@MartinKjunior I believe one of the crucial aspect of the project is the creation of a dedicated website for lexicon data. Integrating the database directly into the MkDocs website isn't feasible due to the static nature of MkDocs. Therefore, we need to complete part A first, which involves setting up the backend and database. Once that's done, we can work on part B, which entails connecting MkDocs with the backend using JavaScript and AJAX requests to fetch data from the backend API for proper communication.

Regarding the frontend setup, we have options. We can use MkDocs as our frontend, or we can opt for a more dynamic and interactive web application by building a frontend from scratch using Reactjs or Bootstrap along with the backend and database. However, if we decide to switch to a new frontend setup, we'll need to make further changes along the APIs to accommodate the requirements and architecture of the new frontend efficiently. This process may be time-consuming, but it's necessary to ensure compatibility and effective communication between the frontend and backend components.

[Aditish51]

Hi, @MartinKjunior and @MRdep Can you please give me a structured description of the project.?It will be a great help for me to describe it in my proposal. I would try to submit my proposal as soon as possible.

[MRdep]

Hi Aditish, I'm just following up with another OSIPI collegue who web dev experience who will hopefully be able to advise on the project. I will be in touch next week, which should give you enough time still to develop the project and submit. Thanks, Ben

…

-------------------------------------------------------------------------- Ben Dickie, PhD Senior Experimental Officer and Senior Research Fellow in Preclinical MRI 7T MRI laboratory, G727 Stopford Building The University of Manchester mob: 07837837807 | e: ***@***.*** ------------------------------------------------------------------

________________________________ From: Aditi Shukla ***@***.***> Sent: 21 March 2024 05:07 To: OSIPI/OSIPI_CAPLEX ***@***.***> Cc: Ben Dickie ***@***.***>; Mention ***@***.***> Subject: Re: [OSIPI/OSIPI_CAPLEX] Implement an element-wise versioning (Issue #71) Hi, @MartinKjunior [github.com]<https://urldefense.com/v3/__https://github.com/MartinKjunior__;!!PDiH4ENfjr2_Jw!F6ILAxtTd2-lIPw4RYuPsSd7I2P2zeRlMknMHy85ZU-YBoxDc9iQSBv1GiKcr-a3_K83RrNymwMaCr5bmBZ5qCwKvW_J5wj2$> and @MRdep [github.com]<https://urldefense.com/v3/__https://github.com/MRdep__;!!PDiH4ENfjr2_Jw!F6ILAxtTd2-lIPw4RYuPsSd7I2P2zeRlMknMHy85ZU-YBoxDc9iQSBv1GiKcr-a3_K83RrNymwMaCr5bmBZ5qCwKvWoWEwIj$> Can you please give me a structured description of the project.?It will be a great help for me to describe it in my proposal. I would try to submit my proposal as soon as possible. — Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/OSIPI/OSIPI_CAPLEX/issues/71*issuecomment-2011228116__;Iw!!PDiH4ENfjr2_Jw!F6ILAxtTd2-lIPw4RYuPsSd7I2P2zeRlMknMHy85ZU-YBoxDc9iQSBv1GiKcr-a3_K83RrNymwMaCr5bmBZ5qCwKvZIhQbqo$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABWSCPYR22A6GC3VIIP46UDYZJTIZAVCNFSM6AAAAABEYHBBBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJRGIZDQMJRGY__;!!PDiH4ENfjr2_Jw!F6ILAxtTd2-lIPw4RYuPsSd7I2P2zeRlMknMHy85ZU-YBoxDc9iQSBv1GiKcr-a3_K83RrNymwMaCr5bmBZ5qCwKvewd9F22$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Aditish51]

Hi @MRdep

Thank you for the update. I'm glad to hear that another OSIPI colleague with web development experience will be assisting me on the project. I appreciate you keeping me informed. I will try to research more on working on my proposal till then.

Looking forward to hearing from you next week.
Thank You

[MRdep]

@Aditish51 I am wondering if you have managed to make a draft plan and complete the application form yet? It would be useful to meet online too in the next few days as OSIPI will need to decide who to put forward for the programme soon.

[Aditish51]

@MRdep I will be submitting the proposal within 2-3 days. I am trying my best to find out a compatible solution on how can we build a strong backend and versioning system for the lexicon generator web app. This project will be the best opportunity to enhance my skills. I will be ready for any of the meet conducted in future on the date assigned.

[Aditish51]

Hello, @MRdep and @MartinKjunior I am done with the proposal. Before submitting it officially Can I submit it for review and feedback. I would be grateful to get a review.

[MRdep]

Hi, Yes of course. Please send to ***@***.*** Thanks

…

-------------------------------------------------------------------------- Ben Dickie, PhD Senior Experimental Officer and Senior Research Fellow in Preclinical MRI 7T MRI laboratory, G727 Stopford Building The University of Manchester mob: 07837837807<tel:07837837807> | e: ***@***.*** ------------------------------------------------------------------

________________________________ From: Aditi Shukla ***@***.***> Sent: 29 March 2024 15:39 To: OSIPI/OSIPI_CAPLEX ***@***.***> Cc: Ben Dickie ***@***.***>; Mention ***@***.***> Subject: Re: [OSIPI/OSIPI_CAPLEX] Implement an element-wise versioning (Issue #71) Hello, @MRdep [github.com]<https://urldefense.com/v3/__https://github.com/MRdep__;!!PDiH4ENfjr2_Jw!Aid75fLDWHGapb2QQrv3LAuMh-TcgibzMpLmgmNx592rgwAmHdlDGUaGIsi2r15jbJXPN1mi5RENu8lxuT0k91Orv4hupql8$> and @MartinKjunior [github.com]<https://urldefense.com/v3/__https://github.com/MartinKjunior__;!!PDiH4ENfjr2_Jw!Aid75fLDWHGapb2QQrv3LAuMh-TcgibzMpLmgmNx592rgwAmHdlDGUaGIsi2r15jbJXPN1mi5RENu8lxuT0k91OrvxNjHlFt$> I am done with the proposal. Before submitting it officially Can I submit it for review and feedback. I would be grateful to get a review. — Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/OSIPI/OSIPI_CAPLEX/issues/71*issuecomment-2027398041__;Iw!!PDiH4ENfjr2_Jw!Aid75fLDWHGapb2QQrv3LAuMh-TcgibzMpLmgmNx592rgwAmHdlDGUaGIsi2r15jbJXPN1mi5RENu8lxuT0k91Orv0BHKWEx$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ABWSCP2567TDN7MZW5BCXOLY2WDKTAVCNFSM6AAAAABEYHBBBGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMRXGM4TQMBUGE__;!!PDiH4ENfjr2_Jw!Aid75fLDWHGapb2QQrv3LAuMh-TcgibzMpLmgmNx592rgwAmHdlDGUaGIsi2r15jbJXPN1mi5RENu8lxuT0k91Orv-CY73Fp$>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Aditish51]

Hello, @MRdep the mail you have mentioned is not visible to me. Can I send it to ismrmosipi@gmail.com or can you please mention it again?