How do we minimise the existing tech debt?

[ramyaragupathy]

Technical debt quietly accrues in open-source projects, posing a significant threat as updates are deferred due to time and resource constraints. Tasking Manager, having been in maintenance mode since its Version 4 launch, faces this challenge.

Here's our plan to ensure the Tasking Manager's long term viability, scalability and maintainability:

🛠 Redesign Database Schema: Optimize backend queries for speed and efficiency.
🛠 Migrate to Fast API: Enhance performance by transitioning from Flask framework.
🛠 Upgrade Frontend Dependencies on a regular basis.
🛠 Boost overall test coverage.
🛠 Streamline translation management.
🛠 Create a Common Frontend Component Library for a seamless user experience across tools.

Addressing technical debt in the Tasking Manager codebase requires a holistic approach, and we are committed to regular code reviews, refactoring, documentation improvements, and testing practices to remove these impediments.

Please do share your insights and strategies for tackling the problem!

[robsavoye]

For a major refactoring backend project like this, it's usually best to create a new branch, especially when mixed with switching to FastAPI and a better database schema. The redesigned database schema is in the tm-admin project, and has an extensive unit level (still needs more work) test suite. I'll be less painful to switch to using tm-admin instead of modifying the existing code around major schema changes. I recently added an API layer designed to support a backend so other developers don't have to look at low-level SQL code. It's based on extensive reading of the TM backend code. If the backend REST API is the same using FastAPI as the current Flask one, the frontend should be able to be refactored in parallel.

Refactoring both the backend and frontend at the same time is basically a total rewrite. This would be the time to consider any functionality in the UI or backend that should be incorporated while going through this process. To be able to shift resources to refactoring, the current TM code base would have to be in maintenance mode with only critical bugs fixed so effort could be shifted to the refactoring work. This refactoring work would take many months.

As much of the backend functionality as possible should be in modules, which makes writing unit test cases easier. Other existing modules should also be used, like TM-Extractor, osm-rawdata, tm-admin, the UI library, etc... Most of the existing code lacks comments, so all new code should have comment block in mkdocs format for each function and class so we can generate API documentation. Higher level documents that explain data flow, using the API, etc... should also be written.

[spwoodcock]

Completely agree about setting up a new branch and making it work there before deploy.
Tests are also essential going forward - we are getting there slowly from the modules up!

Note that 'mkdocs' isn't a format, but a docs parser.
In each language the docs style would be:

• Python: Type hints + Google style docstrings (or numpy if preferred).
• JS: JSDoc with types set.

[spwoodcock]

I think you covered most of the points @ramyaragupathy !

• Hopefully https://github.com/hotosm/ui goes a good way to solving the 'shared frontend component library' part.
• Translations can also be centralised. We have discussed using a Weblate project as an index of all terms that can be used across all frontends. We can export into whatever format required by the tool (e.g. .po for TM, .xliff for the shared components). To be assessed further.
• As part of the FastAPI migration you are transitioning from a wsgi to an asgi framework - this means full async capability. Async code should process queries more efficiently. It also opens up the possibility of an async database connection, which will have big performance benefits.

One last thing that I am not certain about is the release cycle process for TM currently.
This is by no means the only way, but I would suggest an approach something like (if not already in place), with branches:

• dev: PRs merged here, with CI/CD so that devs can constantly see and test their updates.
• stage: dev merged to stage periodically so that product managers can test combined functionality.
• main: stage merged to main when thorough testing has been complete & approved as the next release.
  • hotfixes are possible on main if required, to fix an urgent bug in production, bypassing the release cycle.

[martien-176]

As a non developer TM user (mapping, validating) i am wondering:

• is "tech debt" also responsible for the ongoing sluggishness that TM is suffering from?
• and if so, is Rob saying that to solve that, the TM should be freezed to maintenance mode, no new features added, and rewritten completely?

Thanks.

[robsavoye]

I'll be honest, but at least for the backend, it's unfixable. This is responsible for the performance issues, as the size of the data in the tables continues to grow. The current implementation just doesn't scale well. Adding async processing and other modern techniques is a bit impossible to graft onto the current code base, so refactoring the backend completely will enable TM to function well for years in the future. Otherwise it's going to just keep getting slower and slower... I've already got a large part of the backend rewritten, but there is still months of work unless we could get more developer resources, which probably isn't going to happen unless we could get some skilled volunteer help. So yes, we need to shift focus to refactoring, or it'll never get done in a reasonable amount of time. Obviously any critical bugs will get fixed as best as possible, but they'll just be band-aids to buy time.

[ramyaragupathy]

Please see the approach outlined for the backend code migration here: https://community.openstreetmap.org/t/migrating-tasking-manager-from-flask-to-fastapi-and-psycopg2-to-asyncpg/116543

cc @prabinoid @kaditya97 @manjitapandey @royallsilwallz @spwoodcock