MDEV-35046 SIGSEGV in list_delete in optimized builds when using pseu…

[DaveGosselin-MariaDB]

…do_slave_mode

This patch disallows OPTIMIZE TABLE when (1) in pseudo slave mode and (2) after the session has run XA PREPARE, matching the behavior of the system when pseudo slave mode is disabled.

This patch fixes a memory corruption issue where we would tell innodb to delete a table, which it correctly dropped, but then the SQL layer would not remove it from the correct trx_t structure. Later, when the session closed and we iterated tables on trx_t (specifically mod_tables), we would attempt to access memory associated with the dropped table and crash.

When in pseudo slave mode, we "forget" that we're within an XA transaction during XA PREPARE, and we can't reject statements like OPTIMIZE TABLE. XA PREPARE with pseudo-slave mode enabled resets the XA transaction sim state to one-phase commit by clearing the cached XA transaction state (that is, by setting to NULL the xid_state.xid_cache_element in slave_applier_reset_xa_trans. So we need a way to remember that we're in pseudo slave mode for a particular transaction and also remember whatever transaction ID is in flight.

With that information, we can lookup the XA transaction ID when dropping a table and, if it exists, use that instead of the XA state information associated between the handler and THD. Of course, as a fallback, we will still look at the XA state (trx_t) associated between the handler and THD when the XA transaction doesn't exist (because we're not in XA, or it rolled-back, etc). This fixes the memory corruption by removing the table from the correct trx_t structure when we tell innodb to delete it, keeping consistency.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}