MongoDB Oplog Backup and Restore System

This system provides automated incremental backup of MongoDB's oplog and includes tools for restoration when needed. The system consists of several scripts that work together to maintain a reliable backup history.

Main Components

mainScript.sh

The orchestrator script that:

• Coordinates the execution of backup and cleanup operations
• Implements lock mechanism to prevent concurrent runs
• Logs all operations to /mongo-oplog-dump/logs/
• Ensures scripts run in the correct sequence
• Handles error checking and logging

doIncrementalMongoOplogDump.sh

Performs the incremental backup by:

• Tracking the last backup point using TIMESTEMP and ITERATION files
• Determining the range of oplog entries to backup
• Creating compressed backups in /mongo-oplog-dump/data/
• Handling first-time backups by starting from the earliest available oplog entry

deleteFoldersCountMoreThen200.sh

Manages backup retention by:

• Monitoring the backup directory for folders matching "oplog.rs_data_till*"
• Maintaining the most recent 200 backup folders
• Automatically removing older backups when count exceeds 200
• Sorting folders by modification time to ensure proper cleanup

Backup Process Flow

1. mainScript.sh initiates the backup process
2. doIncrementalMongoOplogDump.sh performs the incremental backup
3. deleteFoldersCountMoreThen200.sh cleans up old backups
4. All operations are logged for monitoring and troubleshooting

Restore Functionality

restoreOplogFromIncrementalBackup.sh

A restoration tool that:

• Allows point-in-time recovery using oplog entries
• Processes multiple backup folders in chronological order
• Provides interactive confirmation before restoration
• Requires specific timestamp for restoration point

For detailed restoration instructions and steps, please refer to Restoring Guide Lines.md

Requirements

• MongoDB server with authentication enabled
• Sufficient disk space for backups
• Execute permissions for all scripts
• Root access to MongoDB

Directory Structure

/mongo-oplog-dump/
├── data/           # Backup storage location
├── logs/           # Log files directory
└── scripts/        # Location of all operation scripts

Configuration

Each script contains configuration variables at the top:

• MongoDB connection details
• Authentication credentials
• Backup directory paths
• Retention settings

Logging

• All operations are logged to /mongo-oplog-dump/logs/
• Log files are rotated daily
• Each log entry includes timestamp and operation details

Security Notes

• Scripts contain MongoDB authentication credentials
• Ensure proper file permissions
• Keep backup directories secure
• Monitor log files for unauthorized access attempts

Troubleshooting

• Check log files for error messages
• Verify MongoDB connectivity
• Ensure sufficient disk space
• Check file permissions
• Verify lock file is removed after script completion

Support

For restoration procedures and additional details, consult the Restoring Guide Lines.md document.