MDEV-35830 Fix innodb_undo_log_truncate in backup

[dr-m]

• The Jira issue number for this PR is: MDEV-35830

Description

recv_sys_t::parse(): Correctly handle the storing==BACKUP case, and simplify some logic around storing==YES as well.

Release Notes

mariadb-backup --backup could fail when backing up a server where innodb_undo_log_truncate=ON is set.

How can this PR be tested?

The added test mariabackup.undo_truncate is based on an idea of @Thirunarayanan (#3743). It nondeterministically covers this logic, including the function backup_undo_trunc(), for both innodb_encrypt_log=ON and innodb_encrypt_log=OFF.

Basing the PR against the correct MariaDB version

• This is a new feature or a refactoring, and the PR is based against the main branch.
• This is a bug fix, and the PR is based against the earliest maintained branch in which the bug can be reproduced.

PR quality check

• I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
• For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[dr-m]

For the record, this is occasionally covering the following code in 10.6 d5a417b as well as 10.11:

diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc
index c1d7085237f..16c2adb0bb7 100644
--- a/extra/mariabackup/xtrabackup.cc
+++ b/extra/mariabackup/xtrabackup.cc
@@ -1012,6 +1012,7 @@ static void backup_file_op_fail(uint32_t space_id, int type,
 
 static void backup_undo_trunc(uint32_t space_id)
 {
+  abort();
   undo_trunc_ids.insert(space_id);
 }
 

./mtr --parallel=4 mariabackup.undo_truncate{,} --repeat=20

With the patch, it would typically crash on the 2nd or 3rd run of 10.6. On 10.11, it would require more attempts for me.

[dr-m]

This requires more space than just 1 byte. I think that the maximum is around 20 bytes, for decrypting 1 byte of payload. I will revise this accordingly. I’d rather not allocate the entire decrypt_buf for the storing==BACKUP case.

Error:Run-Time Check Failure #2 - Stack around the variable 'buf' was corrupted. At C:\Buildbot\amd64-windows\build\storage\innobase\log\log0recv.cc:3137

and

==150369==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fff4c0e4ca1 at pc 0x7f643eb7a061 bp 0x7fff4c0e4ba0 sp 0x7fff4c0e4350
WRITE of size 3 at 0x7fff4c0e4ca1 thread T0
    #0 0x7f643eb7a060 in __interceptor_memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:827
    #1 0x55f09f4fcd49 in recv_buf::memcpy(void*, unsigned long) const /home/buildbot/amd64-debian-12-asan-ubsan/build/storage/innobase/log/log0recv.cc:2239
    #2 0x55f09f4fcd49 in recv_buf::copy_if_needed(unsigned char const*, unsigned char*, recv_buf, unsigned long) /home/buildbot/amd64-debian-12-asan-ubsan/build/storage/innobase/log/log0recv.cc:2283
    #3 0x55f09f4fcd49 in recv_sys_t::parse_mtr_result recv_sys_t::parse<recv_buf, (recv_sys_t::store)1>(recv_buf&, bool) /home/buildbot/amd64-debian-12-asan-ubsan/build/storage/innobase/log/log0recv.cc:2716

[dr-m]

The worst-case buffer size will be 1+5+5+1 bytes here. Thanks to rlen=1we know that even with the maximal-length encoding of space_id and page_no, the first byte will be between EXTENDED+2 and EXTENDED+10, that is, we will not need additional byte(s) for encoding the length of the record.