Remove categories altogether - allow custom ones through the API

[aleksasiriski]

I had one idea, that would also be a good step towards the distributed model:

Remove all categories from the agent, removing the need to have the config for each one and all the other category related code. Instead of that, we introduce "search types" which would currently be general, images and currencies. After that, adjust the API so we have an endpoint for each type:

• /search?q=query&engines=google,bing,duckduckgo&ranking=???
• /images?q=query&engines=google,bing&ranking=???
• /currencies

The idea is: let the user choose which engines to use and how to rank the results, the agent does the scraping and ranking. I would also strongly suggest we do not set any "defaults" for this since it would waste time to code it and then have config as well. Instead, the user in our case would actually be the frontend and the defaults can be set there. The end result would be that categories effectively moved over to the frontend and the perceieved UI stays the same while also allowing us to easily implement custom made categories that users can make using the planned options UI.

Why this is good for the distributed model:

• The proxy aggregator crowdsourced beast of a program can distribute the workload more easily since each engine could be separated using individual agents
• No duplicate code for categories on mutliple layers
• Less "QoL" code updates happening on agents - since this is the component that self-hosters are mainly meant to run themselves

Future:

• Optional ranking - instead of ranking the results we should allow them to be returned in no particular order but with the ranks from the engines the results were returned by, allowing the proxy aggregator to do the ranking (needed if distributing engines across agents)

EDIT: Also important to note: doing the ranking on agent now won't be wasteful because the proxy aggregator will be also written in Go so we can just copy paste that part (reuse the package?), this also include any changes to the algo

[aleksasiriski]

@k4lizen @matijakljajic Would love your opinions

[matijakljajic]

I've already mentioned that I like the idea.

It seems it's a valid solution regarding scalability, especially if we want a true distributed system.

I can't make out the specifics about how the codebase could look like, but it will be more manageable for sure.

[aleksasiriski]

I've started work on this, essentially categories (including ranking) is removed from the config and is instead being read from URL params.

In order to keep the URL short and nice looking, the config (JSON encoded Go object) will be base64 as a single "category" parameter instead of using multiple params for every part of the category (enabled engines, required, preferred, hard timeout, etc.).

This means that after the changes on the backend, we will need to recreate the current categories on the frontend and after that we will be able to let users make their own custom categories that can be persisted using cookies (in order to keep noJS users working).

@k4lizen Hopefully you have the time to add your input on this soon, as it is an interesting change to how we handle requests and a big step towards the creation of the first Hearchco network.

[aleksasiriski]

Continued here: #450

[k4lizen]

I like this change, it makes sense to pull out the configuration to the frontend and let agents be more standalone.

That being said by doing it this way we have a loss of information on the ranking within categories, as for example one search engine could perform better for one category and worse for another, and should be ranked as such.

Though to be honest, the whole category system is kind of in shambles and probably needs to be reworked anyways. An eventual solution for the mentioned loss of information could be passing some ranking modifiers together with the engine information as well, maybe this is what you meant by ranking=??? though as I see it it would need to be multiple paramaters.

We can look into that in the future, the changes as proposed are good.