Batch: ✨ Batch Job For Write Back & Write Allocate Strategy (This is Last Piece 🤗)

[psychology50]

작업 이유

• To implement a batch process that synchronizes last read message status from Redis cache to database

작업 사항

1. Created batch components:

• Reader: Reads last message status data from Redis using cursor
• Processor: Converts Redis key-value data to ChatMessageStatus entity
• Writer: Saves processed data to database using bulk insert

2. Added integration tests to verify:

• Basic batch execution with test data
• Handling of invalid/partial data
• Empty data scenario
• Large dataset processing

3. Configured scheduler to run every 30 minutes using @Scheduled annotation

리뷰어가 중점적으로 확인해야 하는 부분

1. Batch Architecture:

• Is the separation of concerns appropriate? (Reader/Processor/Writer)
• Is the error handling sufficient?

2. Data Integrity:

• Will this implementation prevent data loss?

3. Performance:

• Is the Redis cursor implementation efficient?
• Is the bulk insert strategy optimal?

4. Test Coverage:

• Are the test cases comprehensive enough?
• Are there any missing edge cases?

발견한 이슈

1. Redis Scan Operation Non-Atomicity
   • Since Redis scan operations are non-atomic, there might be edge cases.
   • However, with the batch scheduler running every 30 minutes and Redis cache TTL set to 1 hour, this timing should prevent issues.
2. Bulk Insert Operation Load Concerns
   • Bulk inserts could cause temporary database load spikes. Adjusting the chunk size may help manage this if needed.
3. Post-Sync Redis Data Persistence
   • The current implementation keeps Redis data after synchronization.
   • We might consider adding a cleanup function in the future.
4. MySQL Batch Insert Refactoring
   • The current MySQL batch insert isn’t functioning as a true batch insert. Refactoring is needed to address this.