[Archived] Recovering missing Cassandra data via Orion logs

postmortem - killed cassandra instances and this resulted in it not adding elements because of Local_One setting

early Feb - March 16 2018 - no data (except during OpenData day)

action items:

• MONITOR ORION
• Make an action plan before killing any Cassandra instances
• Why did it die - wasn’t Cassandra supposed to be available for the API even with 1 node running?
• was two nodes not sufficient due to the consensus model? (odd # of nodes needed)
• recover data by streaming kubectl logs OR attempt to get SFMTA/TTC AVL data

Recovering Data Plan:

• stream kubectl logs while getting a timestamp
• use existing Orion modules to format the data and insert it to Cassandra

Requirements:

• make a devbox on google cloud as to be able to run the scripts directly on the prod cassandra