More efficient query of if server can see event

[erikjohnston]

Currently we implement the question of "can a server see an event" by pulling out the state at the event (and potentially the current state). This is probably needlessly inefficient, especially as its currently in the room server.

[NegativeMjark]

The problem with the "can a server see an event" query is that in order to answer it we need to know if the server is in the room at a given point. A server is in the room if there exists an event in the room state with event["type"] == "m.room.member", event["content"]["membership"] == "join" and SplitN(event["state_key"], ":", 2)[1] == serverName.

Currently we answer this query by iterating over the entire room state, which can be thousands of events for some matrix rooms.

We could improve the query by only selecting those state events with event["type"] == "m.room.member" using existing indexes, but since those events account for the majority of state events in a lot of matrix rooms that doesn't help us much.

We could improve the query by only selecting only those state events where SplitN(event["state_key"], ":", 2)[1] == serverName by adding an extra index somewhere, but for rooms that are bridged from IRC nearly all the state events are for users on a single server, so we still end up loading nearly all the room state for queries for that server.

We could improve the query (StateSnapshotID, ServerName) -> InRoom by caching the result in RAM, but an individual room can have multiple snapshots and there are many rooms and many servers so this approach creates an arms race between remote server request patterns and the size of the in-memory cache. Also it makes starting a server really painful since it runs very slowly until the cache warms.

We could improve the query (StateSnapshotID, ServerName) -> InRoom by precalculating the result when we create a new snapshot and storing the result in the database. This will slow down inserting events in the room server, but might be the best option in terms of overall performance.

[kegsay]

Creating an extra table in roomserver to store this information per:

precalculating the result when we create a new snapshot and storing the result in the database.

seems the most sensible approach here.