Auto-adjust diagnostic ranges on document modification while waiting for update from LSP server

[necto]

Is your feature related or already mentioned on the wishlist? No

Repost from the discussion #4497 connecting this issue with the solution PRs.

The problem

I am using SonarLint CFamily via lsp-sonalint. Unfortunately, the CFamily analyzer is pretty slow at the moment, and it takes several seconds or even minutes to analyze a file after it has been changed.
In the meantime, while I edit the contents, issue locations become stale and point to the wrong parts.
Here is an example:

The SonarLint diagnostic about a TODO comment starts off correctly on the TODO comment (on line 13).
When I delete a line, the diagnostic line stays the same, but the the TODO comment is now actually on line 12,
so I get a confusing issue location.
I then wait for a few seconds for SonarLint to publish the updated set of diagnostics, and the diagnostic location gets the correct line 12.

However, I often edit the files continuously and this problem arises on every line deletion/insertion, so the diagnostics constantly point to wrong locations.

I've noticed similar problem with clangd but to a lesser degree because of a shorter turn-around time.

I could set lsp-diagnostic-clean-after-change to remove the diagnostics altogether, but that means I'd need to restrain myself from changing a file for a few seconds or a minute to see the diagnostics.

Solution

I propose to adjust the diagnostic locations on every change.
Particularly, I would go through the list of diagnostics and check which ones have a location that is after the change, and increment/decrement them to the corresponding amount.

Implementation

My plan is the following:

• Add a few regression tests around the display of diagnostics using a mock LSP server.
  chore: test diagnostic locations with a help of a mock LSP server #4502 + chore: Add more tests for code locations in LSP features #4503
• Add an overlay for each diagnostic, and readjust its range each time lsp-mode passes it to flycheck Fix #4501: Auto adjust stale diagnostic ranges on document changes #4513

[necto]

After some perusing of the lsp-mode codebase I realized, that built-in overlays are perfect for this job. Instead of manually tracking and readjusting all the diagnostic's ranges, I allocate one overlay for each and use their begin and end positions for diagnostic display.
Here is PoC: https://github.com/necto/lsp-mode/tree/az/fixed-transient-ranges

[necto]

Here is how it looks like after #4513: