Monthly Automatic Solr-ReIndex

[mekarpeles]

Proposal

As a followup to:

• 503s Solr Performance Issues #10287

Create a hands-free, scheduled task to routinely run solr optimize (and possibly automated monthly re-indexing)

Justification

Over time, solr performance can get very bad if we don't regularly run optimize. See #10287

The entire site grinds to a halt

Breakdown

Requirements Checklist

• Document the process for running solr optimize (and link here)
• Automate (hands-free) the process of running solr optimize (may require switching ol-solr[0-1])
• Setup a cron or schedule solr optimize on a cadence (e.g. monthly)
  • We might also consider doing the same for solr full re-indexes (e.g. monthly or quarterly)

Related files

[mekarpeles]

Next step is to run optimize on ol-solr1 (i.e. backup) and see if it is offline during this time.

If optimize makes solr unavailable, we need a way for our services to switch to another ready-to-go version of solr

Can we reverse proxy to the right solr from nginx or haproxy?

We update open library to have a dedicated dns for solr.openlibrary.org and when we run optimize, ol-www0 config would handle requests to solr.openlibrary.org and route according to its config which would get changed and reloaded by the optimize script? T.B.D how we handle logging for this feature if we do this approach