MDEV-35151 DDL recovery for high-level indexes #3520

svoj · 2024-09-13T16:03:34Z

[] The Jira issue number for this PR is: MDEV-35151

Description

Various fixes to make DDL recovery work with high-level indexes.

Release Notes

None.

How can this PR be tested?

mtr

Basing the PR against the correct MariaDB version

This is a new feature or a refactoring, and the PR is based against the main branch.
This is a bug fix, and the PR is based against the earliest maintained branch in which the bug can be reproduced.
This is a bug fix to a new feature and the PR is based against the feature tree.

PR quality check

I checked the CODING_STANDARDS.md file and my PR conforms to this where appropriate.
For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

CLAassistant · 2024-09-13T16:03:40Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

vuvova · 2024-09-14T09:16:53Z

sql/ddl_log.cc

+      file->ha_rename_table(idx_from, idx_to);
+    }
+  }
+


I'm not sure this is a good approach. It feels like you put too much inside one DDL_RENAME_PHASE_TABLE entry. There can be crashes between renames of individual vector indexes (presuming there will be many some day).

Perhaps, DDL_RENAME_PHASE_TABLE should be one ha_rename_table only? That is, a table with hlindexes would generate one DDL_RENAME_PHASE_TABLE entry per index (and one for the main table, of course)?

Same for DROP, etc.

There can be crashes between renames of individual vector indexes (presuming there will be many some day).

Yes, sure. And my code handles it inline with what original code have been doing, that is ignores non_existing_table_errors(), e.g. if handler files were moved and .frm not. Or as you suggested above crash between different high-level indexes rename.

... a table with hlindexes would generate one DDL_RENAME_PHASE_TABLE entry per index (and one for the main table, of course)?

PHASE is just an enum member of an ACTION, it can't have stuff like table name or whatever attached.

ACTION is more logical (or high-level) entity, which may do many things like binlogging, handling stat tables, triggers, handler files and of course .frm.

Here're options:

We could add special phases to handle high-level indexes, but there's going to be no much difference between my solution. And it will introduce compatibility issues. Though it is very true that older versions won't be able to handle high-level indexes properly even with this patch.

We can't re-use existing actions like DDL_LOG_RENAME_TABLE_ACTION, as I mentioned before they're way to sophisticated for high-level indexes.

In theory we could try re-using special actions like DDL_LOG_RENAME_ACTION for high level indexes. That is issue one DDL_LOG_RENAME_TABLE_ACTION and then DDL_LOG_RENAME_ACTION for each high-level index. It may work, it has no compatibility concerns, but it is a question if we're going to be able to integrate it with e.g. ALTER.

Your thoughts?

I don't think I'm much of an expert in ddl logging. Let's use your approach if it works.

vuvova · 2024-09-14T09:17:32Z

sql/ddl_log.cc

+      file->ha_rename_table(idx_from, idx_to);
+    }
+  }
+


Same for DROP, etc.

mysql-test/main/AAA.test

storage/maria/ha_maria.cc

vuvova · 2024-09-28T20:33:46Z

could you please rebase it on top of the latest bb-11.6-MDEV-32887-vector ? It's impossible to review now, it looks like it includes basically everything

…e upon using non-default table options and system variables extend the option_list expicitly on CREATE/ALTER, not implicitly on parsing.

…th vector key and unique key InnoDB rename needs the same workaround for hlindexes as it has for partitions

… DISTINCT don't apply distinct optimization to order by a vector index

When using the default innodb_log_buffer_size=2m, mariadb-backup --backup would spend a lot of time re-reading and re-parsing the log. For reads, it would be beneficial to memory-map the entire ib_logfile0 to the address space (typically 48 bits or 256 TiB) and read it from there, both during --backup and --prepare. We will introduce the Boolean read-only parameter innodb_log_file_mmap that will be OFF by default on most platforms, to avoid aggressive read-ahead of the entire ib_logfile0 in when only a tiny portion would be accessed. On Linux and FreeBSD the default is innodb_log_file_mmap=ON, because those platforms define a specific mmap(2) option for enabling such read-ahead and therefore it can be assumed that the default would be on-demand paging. This parameter will only have impact on the initial InnoDB startup and recovery. Any writes to the log will use regular I/O, except when the ib_logfile0 is stored in a specially configured file system that is backed by persistent memory (Linux "mount -o dax"). We also experimented with allowing writes of the ib_logfile0 via a memory mapping and decided against it. A fundamental problem would be unnecessary read-before-write in case of a major page fault, that is, when a new, not yet cached, virtual memory page in the circular ib_logfile0 is being written to. There appears to be no way to tell the operating system that we do not care about the previous contents of the page, or that the page fault handler should just zero it out. Many references to HAVE_PMEM have been replaced with references to HAVE_INNODB_MMAP. The predicate log_sys.is_pmem() has been replaced with log_sys.is_mmap() && !log_sys.is_opened(). Memory-mapped regular files differ from MAP_SYNC (PMEM) mappings in the way that an open file handle to ib_logfile0 will be retained. In both code paths, log_sys.is_mmap() will hold. Holding a file handle open will allow log_t::clear_mmap() to disable the interface with fewer operations. It should be noted that ever since commit 685d958 (MDEV-14425) most 64-bit Linux platforms on our CI platforms (s390x a.k.a. IBM System Z being a notable exception) read and write /dev/shm/*/ib_logfile0 via a memory mapping, pretending that it is persistent memory (mount -o dax). So, the memory mapping based log parsing that this change is enabling by default on Linux and FreeBSD has already been extensively tested on Linux. ::log_mmap(): If a log cannot be opened as PMEM and the desired access is read-only, try to open a read-only memory mapping. xtrabackup_copy_mmap_snippet(), xtrabackup_copy_mmap_logfile(): Copy the InnoDB log in mariadb-backup --backup from a memory mapped file.

…E with a wrong data

…abled

… area assertion failures upon EITS collection with vector type don't collect histograms for vector columns

ONLINE ALTER didn't expect XA PREPARE to fail. Mark rollback on failed prepare with the XA_ROLLBACK_ONLY state, detect that in ONLINE ALTER

…NCT with vector type add MYSQL_TYPE_VECTOR to an if() over field->type() values

This reverts commit 79b7f89.

This reverts commit db658f9.

…ILED

… area assertion failures upon EITS collection with vector type

test case

… DISTINCT MariaDB#2 mhnsw index can be used only for ORDER BY. not for GROUP BY

…_key_to_innobase upon query from table with usual key on vector field add test

with streaming implemened mhnsw no longer needs to know the LIMIT in advance. let's just cap it to avoid allocating too much memory for the one step result set

…ry from empty table

…s and VEC_ToText accepts let VEC_FromText validate that the vector l2squared isn't NaN. VEC_ToText still prints everything.

… on table with vector field test case

…ue of mhnsw_min_limit mhnsw_min_limit must not be larger than candidates queue size

…OMTEXT with an invalid argument

Since high-level index tables do not participate in thr_multi_lock(), added explicit call to THR_LOCK::start_trans(). This is needed mostly for Aria to handle transaction logging.

vuvova · 2024-10-25T14:58:27Z

cherry-picked into the vector branch, thanks! closing.

svoj requested a review from vuvova September 13, 2024 16:03

vuvova reviewed Sep 14, 2024

View reviewed changes

vuvova requested changes Sep 14, 2024

View reviewed changes

vuvova reviewed Sep 14, 2024

View reviewed changes

storage/maria/ha_maria.cc Outdated Show resolved Hide resolved

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch from 658f587 to 70ff3b6 Compare September 14, 2024 17:51

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch 11 times, most recently from e231991 to 4557941 Compare September 25, 2024 19:07

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch 2 times, most recently from cadf2be to fe9092a Compare September 29, 2024 19:54

svoj force-pushed the bb-11.6-MDEV-32887-vector-ddl branch 2 times, most recently from 3d2f51e to 5bea61a Compare October 2, 2024 15:22

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch 2 times, most recently from 1d2a5fc to b29dca9 Compare October 6, 2024 19:59

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch 2 times, most recently from a8a5206 to 8bb7868 Compare October 16, 2024 14:34

vuvova and others added 4 commits October 17, 2024 09:42

MDEV-35092 Server crash, hang or ASAN errors in mysql_create_frm_imag…

1054d9f

…e upon using non-default table options and system variables extend the option_list expicitly on CREATE/ALTER, not implicitly on parsing.

MDEV-35104 Invalid (old?) table or database name upon DDL on table wi…

4851c9c

…th vector key and unique key InnoDB rename needs the same workaround for hlindexes as it has for partitions

MDEV-35105 Assertion `tab->join->order' fails upon vector search with…

4a7d5ad

… DISTINCT don't apply distinct optimization to order by a vector index

vuvova added 11 commits October 19, 2024 14:22

add MYSQL_TYPE_VECTOR for MySQL compatibility. protocol

b438ae0

MDEV-35178 Assertion failure in Field_vector::store upon INSERT IGNOR…

57a66e1

…E with a wrong data

MDEV-35176 ASAN errors in Field_vector::store with optimizer_trace en…

83503da

…abled

MDEV-35177 Unexpected ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, diagnostics…

1a59071

… area assertion failures upon EITS collection with vector type don't collect histograms for vector columns

MDEV-35182 crash in online_alter_end_trans with XA over vector indexes

b24e34e

ONLINE ALTER didn't expect XA PREPARE to fail. Mark rollback on failed prepare with the XA_ROLLBACK_ONLY state, detect that in ONLINE ALTER

MDEV-35191 Assertion failure in Create_tmp_table::finalize upon DISTI…

1959d43

…NCT with vector type add MYSQL_TYPE_VECTOR to an if() over field->type() values

Revert "add MYSQL_TYPE_VECTOR for MySQL compatibility. protocol"

763ec2a

This reverts commit 79b7f89.

Revert "add MYSQL_TYPE_VECTOR for MySQL compatibility"

540ad30

This reverts commit db658f9.

MDEV-35160 RBR does not work with vector type, ER_SLAVE_CONVERSION_FA…

d8447d9

…ILED

MDEV-35177 Unexpected ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, diagnostics…

8f2be85

… area assertion failures upon EITS collection with vector type

MDEV-35204 mysqlbinlog --verbose fails on row events with vector type

4dfa605

test case

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch from 4b57276 to 4dfa605 Compare October 19, 2024 12:47

vuvova and others added 10 commits October 19, 2024 20:05

MDEV-35195 Assertion `tab->join->order' fails upon vector search with…

9452b2e

… DISTINCT MariaDB#2 mhnsw index can be used only for ORDER BY. not for GROUP BY

MDEV-35203 ASAN errors or assertion failures in row_sel_convert_mysql…

f9f651f

…_key_to_innobase upon query from table with usual key on vector field add test

MDEV-35194 non-BNL join fails on assertion

3a3e8fa

with streaming implemened mhnsw no longer needs to know the LIMIT in advance. let's just cap it to avoid allocating too much memory for the one step result set

MDEV-35212 Server crashes in Item_func_vec_fromtext::val_str upon que…

20673d6

…ry from empty table

MDEV-35210 Vector type cannot store values which VEC_FromText produce…

643f036

…s and VEC_ToText accepts let VEC_FromText validate that the vector l2squared isn't NaN. VEC_ToText still prints everything.

MDEV-35205 Server crash in online alter upon concurrent ALTER and DML…

99dd28d

… on table with vector field test case

MDEV-35213 Server crash or assertion failure upon query with high val…

93b15e9

…ue of mhnsw_min_limit mhnsw_min_limit must not be larger than candidates queue size

MDEV-35215 ASAN errors in Item_func_vec_fromtext::val_str upon VEC_FR…

c290163

…OMTEXT with an invalid argument

MDEV-34919 - Aria crashes with high-level (vector) indexes

670f3a6

Since high-level index tables do not participate in thr_multi_lock(), added explicit call to THR_LOCK::start_trans(). This is needed mostly for Aria to handle transaction logging.

DDL recovery for high-level indexes

3869f7a

svoj force-pushed the bb-11.6-MDEV-32887-vector-ddl branch from 5bea61a to 3869f7a Compare October 21, 2024 14:32

svoj changed the title ~~DDL recovery for high-level indexes (CREATE, DROP and RENAME)~~ DDL recovery for high-level indexes Oct 21, 2024

vuvova changed the title ~~DDL recovery for high-level indexes~~ MDEV-35222 DDL recovery for high-level indexes Oct 21, 2024

vuvova changed the title ~~MDEV-35222 DDL recovery for high-level indexes~~ MDEV-35151 DDL recovery for high-level indexes Oct 21, 2024

vuvova force-pushed the bb-11.6-MDEV-32887-vector branch 2 times, most recently from 1b6db26 to 62afa2f Compare October 23, 2024 18:27

vuvova closed this Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MDEV-35151 DDL recovery for high-level indexes #3520

MDEV-35151 DDL recovery for high-level indexes #3520

svoj commented Sep 13, 2024 •

edited

Loading

CLAassistant commented Sep 13, 2024 •

edited

Loading

vuvova Sep 14, 2024

vuvova Sep 14, 2024

svoj Sep 14, 2024 •

edited

Loading

vuvova Sep 28, 2024

vuvova Sep 14, 2024

vuvova commented Sep 28, 2024

vuvova commented Oct 25, 2024

MDEV-35151 DDL recovery for high-level indexes #3520

MDEV-35151 DDL recovery for high-level indexes #3520

Conversation

svoj commented Sep 13, 2024 • edited Loading

Description

Release Notes

How can this PR be tested?

Basing the PR against the correct MariaDB version

PR quality check

CLAassistant commented Sep 13, 2024 • edited Loading

vuvova Sep 14, 2024

Choose a reason for hiding this comment

vuvova Sep 14, 2024

Choose a reason for hiding this comment

svoj Sep 14, 2024 • edited Loading

Choose a reason for hiding this comment

vuvova Sep 28, 2024

Choose a reason for hiding this comment

vuvova Sep 14, 2024

Choose a reason for hiding this comment

vuvova commented Sep 28, 2024

vuvova commented Oct 25, 2024

svoj commented Sep 13, 2024 •

edited

Loading

CLAassistant commented Sep 13, 2024 •

edited

Loading

svoj Sep 14, 2024 •

edited

Loading