Fix multiple problems / rewrite Humitifier?

[tymees]

Current state of affairs

So, after some perusing I've found some problems and areas for improvement:

The problems:

• The database schema is not very suited to highs amount of data. This makes the entire DB slow as hell; as a bandaid-solution the DB is now cleaned after a week, but long term the DB schema needs to be fixed
  • And more intelligent cleanup can't hurt
• A scan blocks the webserver from serving things. While asyncio should (tm) alleviate this by giving some CPU time to FastAPI, it doesn't actually do this...
  • Also, am I the only one that thinks having an SSH capable app share memory space/the interpreter with a webserver is an insecure design?
• The scheduler framework (rocketry) used is not maintained anymore
• The SSH framework (parallel-ssh) used is not maintained anymore
  • asyncssh is still maintained, and while the README implies its license is incompatible, it's not... It specifically allows GPL-3 as a secondary license....

The areas for improvement:

• Filtering is implemented in the least performant way possible (hyperbole)
• The way scans are done can be optimized. It's opening an SSH connection N*M times (n = num. of. facts, m = num. of hosts), resulting in large overhead. In addition, it's async flow barely takes advantage of async; the 'io locked' time is only used to create new SSH connections, but with some optimization this time is better spend processing the output.
• The app trusts everything and everyone, and relies on the webserver to protect it from evil
• Scoped acces for developers #72 requires some kind of user-management-like-thing.
• add endpoint for host sync #67 makes more sense than the current 'rollout everything again' procedure manipulate the DB from GITLAB?!?! procedure

Proposed changed

Phase 1: replace the web-app

The first phase would be to split scanning (agent) and data access/presentation (server) into two separate apps. This is to improve security but also performance (no broken webserver during a scan). (I have ulterior motives, namely adding a local-scan feature for ILS PCs in the future)

I'm planning to use Django over FastAPI, as it would speed up development immensely. FastAPI isn't a bad framework, but it does require a lot of manual work to enable basic stuff. Also, it's an API framework, not an web-framework.
Django allows us to 'just import' that stuff, either from Django, libraries, or stuff our developers have made.

• Replicate current functionality in a new Django-app
• Adapt scanner to upload data to new app instead

Phase 2: API & access control for server

The second phase would be adding SAML/OIDC auth to the server app natively, and adding #72 and #73

• Implement authentication
• Implement group-based scoped access to data
• Implement REST API (for FSW)

Phase 3: rewriting the agent

The scanning is handled using two unsupported pieces of software. At the moment, it's not that much of a concern (especially if we detach it from the web-app), but long term we need something that will not attract unpatched zero-days.

Server

• Implement host-sync endpoint in the server
• Implement scan-config endpoint for agent

Agent

• Replace parallel-ssh
• Replace rocketry
• Remove the need for an internal agent-DB
• Implement new scan-config format
  • Fetch scan config from server
  • Fetching target hosts from server
  • Implement local-runner option (optional?)

[tymees]

Oh, somewhere around fase 2-3 I'd like to implement a data-reduction mechanism.

[tymees]

Phase 1 has been completed

Oh, somewhere around fase 2-3 I'd like to implement a data-reduction mechanism.

Partially implemented in phase 1; archived hosts still persist, which will be addressed in phase 3