Backing up and cloning the production Solr instance

[rdhyee]

While we are in the process of migrating to new production Solr instances (Build new production Solr instances · Issue #766 · ekansa/open-context-py), it's very helpful to be able to clone the current production Solr instance. In this issue, I will work out a working method for backing up and cloning the production instance.

[rdhyee]

What I thought should work but didn't: either creating a snapshot from which to create an image -- or creating a new image directly. I could make an image but when I spun the image up, I had problems with the ssh keys. I think the problem is that the current production solr instance is old and that any new images made from it is not properly configured as a guest. I managed to ssh in using the Google cloud UI but Solr didn't seem to be working properly. So I've abandoned that approach.

[rdhyee]

I have an outline of a working approach: build a new Solr instance using Vagrant/ansible and then copy over the old Solr indexes from a recent snapshot of the production Solr instance. It seems that I can use Solr 7 (e.g., 7.7.2) and copy the Solr 6 indices over -- but Solr 8 doesn't seem to work [though I can doublecheck.]. Here's the error:

Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource BufferedChecksumIndexInput(MMapIndexInput(path="/var/solr/data/open-context/data/index/segments_1ty3"))): 6 (needs to be between 7 and 9). This version of Lucene only supports indexes created with release 6.0 and later.

It might be better to build a Solr 6.4 version (to match the production version) -- given Major Changes in Solr 8 | Apache Solr Reference Guide 8.1:

It is always strongly recommended that you fully reindex your documents after a major version upgrade.

[rdhyee]

java - Solr indexing issue after upgrading from 4.7 to 7.1 - Stack Overflow:

You can't upgrade through that many versions in a single upgrade. Solr
(Lucene) only supports the index format from the previous version, so
you might have to do each version by itself.

There's a tool bundled with Solr to help you do this - the
IndexUpgrader.

[ekansa]

I think we'll need to fully rebuild the Solr 8 index from scratch. We're making some significant schema updates anyway.

…

On Mon, Dec 9, 2019 at 2:55 PM Raymond Yee ***@***.***> wrote: java - Solr indexing issue after upgrading from 4.7 to 7.1 - Stack Overflow <https://stackoverflow.com/questions/47454434/solr-indexing-issue-after-upgrading-from-4-7-to-7-1> : You can't upgrade through that many versions in a single upgrade. Solr (Lucene) only supports the index format from the previous version, so you might have to do each version by itself. There's a tool bundled with Solr to help you do this - the IndexUpgrader <https://lucene.apache.org/solr/guide/7_1/indexupgrader-tool.html>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#771?email_source=notifications&email_token=AAJ2EVTEC75MK73UZ62EC3LQX3EGLA5CNFSM4JYTRFSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGK73YA#issuecomment-563477984>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2EVVG3WOOY6G7WAOQ65LQX3EGLANCNFSM4JYTRFSA> .

[rdhyee]

Here's some notes on how I built the Solr instance I have for backing the staging server.
Note: [ry-20191213a-solr772(https://console.cloud.google.com/compute/instancesDetail/zones/us-central1-f/instances/ry-20191213a-solr772?project=opencontext-py) is the current production solr instance. Here's how I built a new solr instance that I have attached to staging:

I used open-context-py/build_solr.yml at i766_build_solr · rdhyee/open-context-py to build ry-20191216-solr772. The resulting instance:

ry-20191216-solr772.us-central1-f.opencontext-py -- see Compute Engine - OpenContxt Python Dev - Google Cloud Platform

I then

ssh -N -f -L 8987:localhost:8983 ry-20191216-solr772.us-central1-f.opencontext-py

Mount the oc-solr-20191206-disk drive. I use Compute Engine - OpenContxt Python Dev - Google Cloud Platform and hit the "+Attach existing disk", pick oc-solr-20191206-disk from the drop down (and hit done and save?)

I ssh into the new machine (ssh ry-20191216-solr772.us-central1-f.opencontext-py) and then look for the new drive:

raymondyee@ry-20191216-solr772:~$ sudo lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0  100G  0 disk
└─sda1   8:1    0  100G  0 part /
sdb      8:16   0  100G  0 disk
└─sdb1   8:17   0  100G  0 part

sudo mkdir -p /mnt/disks/solr_backup
sudo mount -o discard,defaults /dev/sdb1 /mnt/disks/solr_backup

set up configuration files:

sudo bash -c "cp -rp /mnt/disks/solr_backup/opt/oc-solr-6/server/solr/open-context/conf/* /var/solr/data/open-context/conf/"

curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=open-context&wt=json"
sudo -u solr bash -c "/opt/solr/bin/solr restart"

Now do a restore of the Solr index (Making and Restoring Backups | Apache Solr Reference Guide 7.7)

using HTTPie – command line HTTP client:

http :8987/solr/open-context/replication command==restore location==/mnt/disks/solr_backup/opt/solr_backups name==20191212203254791 wt==json

or curl

curl "localhost:8987/solr/open-context/replication?command=restore&location=%2Fmnt%2Fdisks%2Fsolr_backup%2Fopt%2Fsolr_backups&name=20191212203254791&wt=json"

You can restorestatus:

http :8987/solr/open-context/replication command==restorestatus wt==json

or

curl "localhost:8987/solr/open-context/replication?command=restorestatus&wt=json"