[Surrey] Exclude Open311-fetched reports from cut_off_date

[davea]

For https://github.com/mysociety/societyworks/issues/4512

[skip changelog]

[codecov]

Codecov Report

Attention: Patch coverage is 61.53846% with 5 lines in your changes missing coverage. Please review.

Project coverage is 82.32%. Comparing base (0a17904) to head (46fe8fc).
Report is 3 commits behind head on master.

Files with missing lines	Patch %	Lines
perllib/FixMyStreet/Cobrand/Surrey.pm	61.53%	3 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5182      +/-   ##
==========================================
- Coverage   82.35%   82.32%   -0.03%     
==========================================
  Files         409      409              
  Lines       31861    31878      +17     
  Branches     5067     5070       +3     
==========================================
+ Hits        26238    26244       +6     
- Misses       4127     4136       +9     
- Partials     1496     1498       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[dracos]

Think query code can be simplified, same outcome, is all :)

[davea]

Based on my understanding of EXPLAIN ANALYZE output I don't think this will undo the improvements that came from adding the index on created, but would appreciate your opinion!

[dracos]

If you look at the analyze when it's only created, you can see a 'nearby' query boils down to:

1. Get the nearby problems using bitmap heap/index scan on problem_radians_latitude_longitude_idx
2. Do a bitmap heap scan using bitmap index scans on both problem_bodies_str_array_idx and problem_created_idx, filtering out state/timestamp based (179 rows)
3. hash together steps 1 and 2, filtering out shown reports
4. sort/limit

With it also checking service, this becomes:

1. Same - nearby problems using bitmap heap/index scan
2. Can only use problem_bodies_str_array_idx now, so bitmap heap scan with that, filter out ones that don't match created/service/state/timestamp (46,317 rows)
3. Same
4. Same

Which is equivalent to what it does when there isn't a cut off date at all; still using a bitmap heap scan, so much better than when it was going wrong. Its runtime is comparable, but it is having to do quite a bit more 'work' (looking at the calculated cost, not the actual time). So yes, think it's fine, we might want to have a think of is there's something we can do to improve the general 'no cut off date' (or here) query, could another index on state, or perhaps a combined (state,created) or (created,state) index, or some other index for this common query help, for example?