-
Notifications
You must be signed in to change notification settings - Fork 6
[Archived] Recovering missing Cassandra data via Orion logs
Eddy Ionescu edited this page Jul 8, 2018
·
1 revision
postmortem - killed cassandra instances and this resulted in it not adding elements because of Local_One setting
early Feb - March 16 2018 - no data (except during OpenData day)
action items:
- MONITOR ORION
- Make an action plan before killing any Cassandra instances
- Why did it die - wasn’t Cassandra supposed to be available for the API even with 1 node running?
- was two nodes not sufficient due to the consensus model? (odd # of nodes needed)
- recover data by streaming kubectl logs OR attempt to get SFMTA/TTC AVL data
Recovering Data Plan:
- stream kubectl logs while getting a timestamp
- use existing Orion modules to format the data and insert it to Cassandra
Requirements:
- make a devbox on google cloud as to be able to run the scripts directly on the prod cassandra