Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement an element-wise versioning #71

Open
MartinKjunior opened this issue Mar 15, 2024 · 21 comments
Open

Implement an element-wise versioning #71

MartinKjunior opened this issue Mar 15, 2024 · 21 comments

Comments

@MartinKjunior
Copy link
Contributor

The table contents will be regularly updated in the future as we work on the website. It would be useful to have a versioning system in place such that we (and the users) could track when and what changes have been made to each element (value) in all of the tables. In other words, have a history of all changes for each value available on click, on top of general website versioning which is also necessary.
@Aditish51
Copy link
Contributor

Aditish51 commented Mar 15, 2024 •
edited
Loading

@MartinKjunior Can you give me more idea on that? I guess we need to setup a database and integrate it with the mkdocs documentation.

@MartinKjunior
Copy link
Contributor Author

@Aditish51 I would agree. I think as we add more sophisticated tools to the website it might be necessary to build a proper database instead of manually writing everything into the markdown files. That should also allow for easier changes to the tables. The only relevant tool, that I know of, in mkdocs for this is the table-reader, which is essentially just a wrapper for panda's data-reading functions which I've mentioned in #55 .

I don't have much experience setting up such a database. When it comes to turning the existing markdown text data into csv files, that's an easy job; I can send you a quick python script I wrote to get the tabular data. However, I assume there are better solutions to storing the data than having a bunch of csv/excel files and reading them in whenever a user requests a webpage with a table on it. Researching the appropriate solution based on the needs of OSIPI could be a nice long-term project if that's something you're interested in. One of the future applications of this lexicon would be having a webapp where a researcher can put their paper and the app will replace all instances of keywords that exist in the lexicon with a hyperlink that connects that keyword to the appropriate table cell. E.g. if my paper talks about using the "Patlak model", then it would turn that keyword into Patlak model that takes you to the definition. I would assume that to achieve that there would be a need for 1. efficient lookup in the database, 2. versioning of the website (mentioned in #17 ). The versioning is quite important so that if a researcher writes a paper in 2025 and uses the definitions used at that time, then the definition is altered in 2026, a reader in 2027 can go to the 2025 version of the website to know how the author of the paper used the terms. Similar idea with versioning of the individual values. The lexicon may also become used by manufacturers in case they start using the OSIPI codes as a standard reference to various quantities and models, which would probably add a need for SQL-like methods of data lookup.

I think this could be the project for you to work on for GSoC if you're interested.

@MRdep
Copy link
Contributor

MRdep commented Mar 19, 2024

@Aditish51 I will add a little too: This is really really important step change in the lexicon that we have been been thinking about for a while. The idea is to migrate the lexicon items to a database and read items from there, instead of from hardcoded .mk files. That way we can implement versioning on each specific lexicon item (I'm not actually sure how this could be done!).

This would be a really great project for GSoc, so if you are interested in this please fill in the project form:
Template proposal OSIPI projects GSoC 2024.pdf

@Aditish51
Copy link
Contributor

Thank you @MartinKjunior and @MRdep for sharing details about the project.This could be one of the best opportunity for me to work on this project.I'm excited about the opportunity to contribute to it as a GSoC participant.

The idea of creating a web application to manage lexicon data and facilitate keyword hyperlinking in research papers is both innovative and valuable. After reading your instructions carefully , the possible solution could be:
A) Migrating lexicon items to database:
We will have to design a database schema that can efficiently store our lexicon data, including versioning information about the lexicon. The possible database system can be either small and lightweight databases like- TinyDB that stores data in JSON format but it is suitable for small scale applications or using large scale DBMS like MySQL or MongoDB.Whenever a lexicon entry is updated, instead of overwriting the existing entry,a new entry is created with the updated definition and increment the version number(in date format).This ensures that historical versions of the lexicon entries are preserved in the database.

B) Creating a complete webapp :
Since MkDocs websites are static sites generated from Markdown files and do not include a backend.To add dynamic functionality to our MkDocs website, such as fetching data from a database, we would typically need to integrate it with a backend. For that we will need a backend framework like flask or django and integrate it to mkdocs frontend or to make the webapp more dynamic we will design the frontend from scratch using Reactjs, bootstrap etc. We will then define API endpoints in our backend server to perform operations on the database. API endpoints in our backend server to allow users to retrieve historical versions of lexicon entries. We will have to implement logic to query the database for specific versions of lexicon entries based on version numbers or timestamps. In our frontend application (e.g., MkDocs website), provide a user interface for accessing historical versions of lexicon entries.

I believe my skills in database management, web development, and version control make me well-suited for this project. I'm eager to learn more about the specific requirements and technologies involved and to collaborate with the OSIPI organization to bring this project to fruition.

@MartinKjunior
Copy link
Contributor Author

@Aditish51 I am glad you're excited to work on this.

Based on your suggestions I do believe you to have the theoretical knowledge necessary to accomplish the project.

It sounds to me like option B will give us much more developmental freedom in the future. I was wondering if it may be possible to do A and then transition into B? Do you think it would be reasonable to first create the database and hook it up to the existing infrastructure (mkdocs) and afterwards work on developing a frontend in Reactjs or would that be too incompatible and a waste of time when having redo the way frontend and backend communicate?

I think it would be useful if you could summarise these two approaches in a presentation (aim for ~15-20 minutes), showing what steps would need to be done in each, their advantages and disadvantages including an expected completion time and whether you think you could achieve it within the time limit of GSoC. Aim this presentation at a non-software engineer audience as I think it would be good if you presented this in front of the rest of the TaskForce 4.2 where there are clinicians (doctors in hospitals) as well as other researchers and their input could be valuable in guiding the specifics of the project. I believe having that information is crucial to creating a product that is directly useful to the experts in the field of perfusion imaging. I also think interactions with experts from other fields and learning to put your ideas across to them would a valuable learning outcome from the GSoC, if you've never done that before. @MRdep would you agree? If so, could you invite Aditish to the next taskforce meeting (probably not the meeting tomorrow, unless @Aditish51 can get the presentation done in a day but I don't want to rush anyone, Aditish just let us know which date suits you best) so they can present their ideas? The meetings are either tomorrow 20/03/2024 or the next is 17/04/2024.

@MRdep
Copy link
Contributor

MRdep commented Mar 19, 2024 •
edited
Loading

Hi @MartinKjunior and @Aditish51.

The timings are not great for presenting to the Task Force, since the GSoc projects need to be submitted by April 2nd. I think best to complete the form with your ideas and we can discuss/develop the project on this thread.

Your ideas sound excellent. Perhaps it would be worth making a plan of how you would spend each week, with deliverables, to determine if it can be delivered in the timeframe.

@Aditish51
Copy link
Contributor

Hello @MartinKjunior and @MRdep ,

Thank you for your appreciation. I'm eager to present my idea to TaskForce 4.2 . However, I have a prior commitment and will be traveling on the 20th of March. Unfortunately, this means I won't be available for a meeting on that day.
But I would be honored to present my idea briefly during the meeting scheduled for the 17th of April. During this presentation, I plan to discuss the advantages, disadvantages, and timeline of the project within the GSoC time limit.

@MartinKjunior I believe one of the crucial aspect of the project is the creation of a dedicated website for lexicon data. Integrating the database directly into the MkDocs website isn't feasible due to the static nature of MkDocs. Therefore, we need to complete part A first, which involves setting up the backend and database. Once that's done, we can work on part B, which entails connecting MkDocs with the backend using JavaScript and AJAX requests to fetch data from the backend API for proper communication.

Regarding the frontend setup, we have options. We can use MkDocs as our frontend, or we can opt for a more dynamic and interactive web application by building a frontend from scratch using Reactjs or Bootstrap along with the backend and database. However, if we decide to switch to a new frontend setup, we'll need to make further changes along the APIs to accommodate the requirements and architecture of the new frontend efficiently. This process may be time-consuming, but it's necessary to ensure compatibility and effective communication between the frontend and backend components.

@Aditish51
Copy link
Contributor

Hi, @MartinKjunior and @MRdep Can you please give me a structured description of the project.?It will be a great help for me to describe it in my proposal. I would try to submit my proposal as soon as possible.

@MRdep
Copy link
Contributor

MRdep commented Mar 23, 2024 via email

@Aditish51
Copy link
Contributor

Hi @MRdep

Thank you for the update. I'm glad to hear that another OSIPI colleague with web development experience will be assisting me on the project. I appreciate you keeping me informed. I will try to research more on working on my proposal till then.

Looking forward to hearing from you next week.
Thank You

@MRdep
Copy link
Contributor

MRdep commented Mar 25, 2024

@Aditish51 I am wondering if you have managed to make a draft plan and complete the application form yet? It would be useful to meet online too in the next few days as OSIPI will need to decide who to put forward for the programme soon.

@Aditish51
Copy link
Contributor

@MRdep I will be submitting the proposal within 2-3 days. I am trying my best to find out a compatible solution on how can we build a strong backend and versioning system for the lexicon generator web app. This project will be the best opportunity to enhance my skills. I will be ready for any of the meet conducted in future on the date assigned.

@Aditish51
Copy link
Contributor

Hello, @MRdep and @MartinKjunior I am done with the proposal. Before submitting it officially Can I submit it for review and feedback. I would be grateful to get a review.

@MRdep
Copy link
Contributor

MRdep commented Mar 29, 2024 via email

@Aditish51
Copy link
Contributor

Aditish51 commented Mar 30, 2024 •
edited
Loading

Hello, @MRdep the mail you have mentioned is not visible to me. Can I send it to [email protected] or can you please mention it again?

@MRdep
Copy link
Contributor

MRdep commented Mar 30, 2024 via email

@Aditish51
Copy link
Contributor

Aditish51 commented Mar 30, 2024 •
edited
Loading

@MRdep I have sent the proposal. Hope so I had matched to the expectations and demands of the project.

Thank You

@MRdep
Copy link
Contributor

MRdep commented Apr 1, 2024 via email

@Aditish51
Copy link
Contributor

Aditish51 commented Apr 1, 2024 •
edited
Loading

Hi @MRdep actually the mail address you mentioned above is not visible to me. I have submitted it officially to the GSoC website as I got the reply from that since it is a holiday I have to directly submit it.
But Can I submit it to you at the slack?

I have submitted the link of doc at the slack. If that does not work I will upload it to this thread.

@Aditish51
Copy link
Contributor

Hello @MRdep ,
Can you please tell me what is the priority of this project ? I would be grateful if my solution fitted all the requirement.

Thank You

@Aditish51
Copy link
Contributor

Aditish51 commented May 2, 2024 •
edited
Loading

Hello, @MRdep ,

My proposal has not been accepted and I think the project itself has not been selected for GSoC. Thank you for giving me the opportunity to propose an idea for this great project I learned a lot during the preparation of my proposal. But if this project is still going to be developed, It would be a great privilege for me to work on this. I request you to give me a chance to work on this. This is one of the best project I have ever came across and could be beneficial to the whole community. I will keep contributing to OSIPI.

Regards
Aditi

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MRdep @MartinKjunior @Aditish51