Custom Migrator Plugin

A WordPress plugin for exporting websites to .hstgr binary format archives with high-performance batch processing.

Batch Processing Configuration

The plugin uses an optimized batch processing system with industry-standard approaches for maximum performance and reliability.

Processing Parameters

Parameter	Value	Description
Batch Size	10,000 files	Maximum files processed per batch cycle
Execution Time	10 seconds	Maximum time per batch before pausing
Memory Threshold	90%	Pause when memory usage exceeds this limit
Chunk Size	512KB	File buffer size for reading/writing
Pause Check Frequency	Every 1,000 files	How often to check pause conditions
Progress Logging	Every 5,000 files	Frequency of progress log entries
I/O Throttling	Every 1,000 files	Throttling applied to prevent system overload
Throttle Delay	0.01 seconds	Brief pause duration when throttling

Processing Approach Comparison

Aspect	Traditional Approach	Custom Migrator	Notes
Timeout Logic	Time-based only (10 seconds)	Time + file count + memory	We use additional safeguards
File Counting	Processes until timeout	Counts files + checks timeout	We limit files per batch
File Discovery	Pre-enumerates all files to CSV	Direct streaming (no CSV)	We eliminated CSV bottleneck
Memory Management	Basic timeout handling	Aggressive 90% monitoring	We proactively manage memory
Resume Tracking	Archive + file byte offsets	Binary offset + file path	We track more resume state

Memory Management Strategy

The plugin uses an aggressive memory management approach:

• Allows memory to grow to 90% of PHP's memory_limit
• Pauses and restarts when threshold is reached
• Preserves resume state with binary offset tracking
• Maximizes performance while maintaining stability

Example Memory Calculation

// If PHP memory_limit = 512M
$memory_limit = 512 * 1024 * 1024;  // 536,870,912 bytes
$threshold = $memory_limit * 0.9;    // 483,183,821 bytes (90%)

// Plugin pauses when memory_get_usage(true) > $threshold

Batch Processing Flow

1. Start Processing: Begin processing files with 10-second timer
2. Check Conditions: Every 1,000 files, verify:
   • Execution time < 10 seconds
   • Memory usage < 90% limit
   • Files processed < 10,000 batch size
3. Pause if Needed: Save state and schedule resume via HTTP request
4. Resume Processing: Continue from exact file position with clean memory
5. Complete: When all files processed successfully

Key Processing Innovations

Traditional time-only approach:

// Process files until 10 seconds elapsed
while ($files_remaining && (microtime(true) - $start) < 10) {
    process_file();
}

Our enhanced approach:

// Process up to 10,000 files OR 10 seconds OR 90% memory
while ($files < 10000 && (microtime(true) - $start) < 10 && memory_usage() < 90%) {
    process_file();
    if ($files % 1000 === 0) check_conditions();
}

Resume Mechanism

The plugin maintains detailed resume state:

{
  "files_processed": 15000,
  "bytes_processed": 2147483648,
  "last_file_path": "/path/to/last/processed/file.jpg",
  "archive_offset": 2151858023,
  "skipped_count": 45,
  "last_update": 1640995200
}

Performance Benefits

• High Throughput: Processes thousands of files per batch
• Memory Efficient: Uses aggressive thresholds with safe restart
• Resume Capability: Never loses progress on interruption
• Session Independent: Works without browser/admin session
• Automation Ready: Supports scripted/curl-based automation

File Structure

custom-migrator/
├── includes/
│   ├── class-exporter.php      # Main export logic with batch processing
│   ├── class-core.php          # Core plugin functionality
│   ├── class-filesystem.php    # File system operations
│   ├── class-database.php      # Database export handling
│   └── class-metadata.php      # Metadata generation
├── admin/
│   ├── class-admin.php         # Admin interface
│   └── js/script.js           # Frontend JavaScript
└── custom-migrator.php        # Main plugin file

Export Output

The plugin generates three files:

1. .hstgr - Binary archive with 4375-byte headers per file
2. .sql.gz - Compressed database dump
3. .metadata - JSON metadata with export information

Technical Notes

• Uses RecursiveDirectoryIterator for efficient file discovery
• Implements binary archive format with structured headers
• Supports large file processing with streaming
• Handles exclusion paths for backup/cache directories
• Maintains compatibility with existing import tools