Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Two-node vfs2 unit test #674

Closed
wants to merge 73 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
c0e8569
Format sources and delete SM diagram
cole-miller Jul 16, 2024
f2f3f83
Rename vfs2_apply_uncommitted and vfs2_unapply
cole-miller Jul 16, 2024
f418816
Remove FLUSH state and invariant
cole-miller Jul 17, 2024
b137aaf
Remove ad-hoc tracing
cole-miller Jul 17, 2024
64831fe
Implement hash table updates for region0
cole-miller Jul 17, 2024
10ba5f0
Rename vfs2_commit to vfs2_apply
cole-miller Jul 17, 2024
be847c7
Allow moving from WTX_ACTIVE to WTX_ACTIVE
cole-miller Jul 17, 2024
19f66d2
Fix rollback test
cole-miller Jul 17, 2024
4ded6bb
Suppose multiple nodes in vfs2 test
cole-miller Jul 17, 2024
0e3054d
Add comments to vfs2 tests
cole-miller Jul 17, 2024
993a673
Use sqlite3_randomness to fill salts
cole-miller Jul 17, 2024
bfb0815
Calculate checksums for the WAL header
cole-miller Jul 17, 2024
2d80f98
Don't move to POLLED state unless a transaction was found
cole-miller Jul 17, 2024
2f42cfe
Replace vfs2_wal_frame with dqlite_vfs_frame
cole-miller Jul 17, 2024
0a7cc12
Initialize page size in vfs2_add_uncommitted if not earlier
cole-miller Jul 17, 2024
b34e6f3
Remove obsolete comments
cole-miller Jul 17, 2024
0820ad9
Remove redundant out argument from vfs2_poll
cole-miller Jul 17, 2024
f4ff5a2
Make vfs2 tests more concise
cole-miller Jul 19, 2024
cc0ff2e
Add a new vfs2 test for transaction replication
cole-miller Jul 19, 2024
bc18c51
Fix WAL header initialization
cole-miller Jul 17, 2024
afa657e
Move shm finalization to maybe_close_entry
cole-miller Jul 19, 2024
8e1f5c9
Use parentheses in OK test macro
cole-miller Jul 19, 2024
67a49a5
Add more extensive comments to vfs2.h
cole-miller Jul 19, 2024
6d21769
Rename open_test_db, accept name as argument
cole-miller Jul 19, 2024
cf2690f
Rename check_wals
cole-miller Jul 19, 2024
88a3131
Rename constants describing main file header
cole-miller Jul 19, 2024
7987e0d
Clean up vfs2 rollback test
cole-miller Jul 19, 2024
6febc7b
Add TODOs about long WALs
cole-miller Jul 19, 2024
7834e33
Add comments about handling of the hash table
cole-miller Jul 19, 2024
13fe315
Clarify comment about releasing the write lock
cole-miller Jul 19, 2024
ba61610
Initialize W-i header with all unused read marks
cole-miller Jul 22, 2024
73b279d
Fix comparison to SQLITE_OK
cole-miller Jul 22, 2024
4b3c238
Simplify and correct W-i hdr cksum calculations
cole-miller Jul 22, 2024
6c2fdc9
Declare sm relationship in two-node vfs2 test
cole-miller Jul 22, 2024
dc3be92
Check validity of WAL headers at startup
cole-miller Jul 22, 2024
05d9e91
Share WAL header calculation between vfs2.c and UT
cole-miller Jul 23, 2024
8912b8e
Require the page size to be known before add_uncommitted
cole-miller Jul 24, 2024
97c9cb3
Split the WAL swap and the WAL header write
cole-miller Jul 24, 2024
ffe06d7
Fix handling of page size
cole-miller Jul 24, 2024
7f20503
Factor out shared memory into a struct
cole-miller Jul 23, 2024
549bff9
Walk WAL-cur at startup to set wal_cursor
cole-miller Jul 24, 2024
43ec47c
Rewrite the logic for WAL swap in add_uncommitted
cole-miller Jul 24, 2024
70ef37e
Add comments to vfs2 tests explaining what's going on
cole-miller Jul 24, 2024
ad7d864
Clean up startup_both_nonempty test
cole-miller Jul 24, 2024
f1adcb5
Add some ancillary state machines for observability
cole-miller Jul 25, 2024
59c9036
Use a NR constant for the wtx sm configuration
cole-miller Jul 25, 2024
6594fb8
Add a new vfs2 test involving a restart
cole-miller Jul 25, 2024
d7490da
Work around integer promotion headache
cole-miller Jul 25, 2024
8b31e83
Draw an updated state machine diagram
cole-miller Jul 25, 2024
eff99ca
Refactor vfs2_poll
cole-miller Jul 25, 2024
ad203cb
Tiny formatting fix
cole-miller Jul 25, 2024
2d9fc07
Make assert_wal_sizes and prepare_wals more flexible
cole-miller Jul 26, 2024
e60109b
Disable checkpoint-on-close in vfs2 unit tests
cole-miller Jul 28, 2024
d924a06
Fix commit marker calculation
cole-miller Jul 29, 2024
f902cba
Add additional checks to leader_and_follower test
cole-miller Jul 29, 2024
bfce72d
Use a prepared statement in rollback test
cole-miller Jul 29, 2024
fde6dca
Return to separate allocations for shm regions
cole-miller Jul 29, 2024
89f695b
Add a TODO about WALs with non-native byte order
cole-miller Aug 1, 2024
aff5f3a
Introduce WAL_MAX_VERSION constant
cole-miller Aug 1, 2024
6c979b3
Remove WAL-prev invalidation
cole-miller Aug 1, 2024
1381861
Fix off-by-one errors in shm region selection
cole-miller Aug 1, 2024
1583dd1
Tweak shm_grow
cole-miller Aug 1, 2024
01f2f92
Assert that SQLite doesn't try to delete the WAL-index
cole-miller Aug 1, 2024
c296805
Add missing error handling for shm_add_pgno
cole-miller Aug 1, 2024
b3ca21c
Add a TODO about the proper handling of stopping points in walk_wal
cole-miller Aug 1, 2024
2b2877b
Correct description of the WTX_BASE state
cole-miller Aug 1, 2024
64c3600
Add vfs2_unhide to wtx sm diagram
cole-miller Aug 1, 2024
527881b
Correct flags for shm and ckpt state machines
cole-miller Aug 1, 2024
9ac2372
More explicit
cole-miller Aug 1, 2024
0bfc0d9
Remark about old frames in walk_wal
cole-miller Aug 1, 2024
f515bef
Remove redundancy between shm_init and shm_restart
cole-miller Aug 1, 2024
fe47c99
Fold the first frame write into the loop in vfs2_add_uncommitted
cole-miller Aug 1, 2024
3951d1e
Add a TODO about the broken read mark implementation
cole-miller Aug 1, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 35 additions & 54 deletions test/unit/test_vfs2.c
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
#define PAGE_SIZE 512
#define PAGE_SIZE_STR "512"

#define OK(rv) munit_assert_int(rv, ==, 0)
cole-miller marked this conversation as resolved.
Show resolved Hide resolved

SUITE(vfs2);

struct node {
Expand Down Expand Up @@ -94,14 +96,12 @@ static void prepare_wals(const char *dbname,
{
char buf[PATH_MAX];
ssize_t n;
int rv;
if (wal1 != NULL) {
snprintf(buf, sizeof(buf), "%s-xwal1", dbname);
int fd1 = open(buf, O_RDWR | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH);
munit_assert_int(fd1, !=, -1);
rv = ftruncate(fd1, 0);
munit_assert_int(rv, ==, 0);
OK(ftruncate(fd1, 0));
n = write(fd1, wal1, wal1_len);
munit_assert_llong(n, ==, (ssize_t)wal1_len);
close(fd1);
Expand All @@ -111,8 +111,7 @@ static void prepare_wals(const char *dbname,
int fd2 = open(buf, O_RDWR | O_CREAT,
S_IRUSR | S_IWUSR | S_IRGRP | S_IWGRP | S_IROTH);
munit_assert_int(fd2, !=, -1);
rv = ftruncate(fd2, 0);
munit_assert_int(rv, ==, 0);
OK(ftruncate(fd2, 0));
n = write(fd2, wal2, wal2_len);
munit_assert_llong(n, ==, (ssize_t)wal2_len);
close(fd2);
Expand All @@ -139,6 +138,13 @@ static void check_wals(const char *dbname, off_t wal1_len, off_t wal2_len)
(rv < 0 && errno == ENOENT && wal2_len == 0));
}

static sqlite3_file *main_file(sqlite3 *db)
{
sqlite3_file *fp;
sqlite3_file_control(db, "main", SQLITE_FCNTL_FILE_POINTER, &fp);
return fp;
}

/**
* Single-node test with several transactions and a checkpoint.
*/
Expand All @@ -149,40 +155,29 @@ TEST(vfs2, basic, set_up, tear_down, 0, NULL)
int rv;

sqlite3 *db = open_test_db(node);
sqlite3_file *fp;
sqlite3_file_control(db, "main", SQLITE_FCNTL_FILE_POINTER, &fp);
rv = sqlite3_exec(db, "CREATE TABLE foo (bar INTEGER)", NULL, NULL,
NULL);
munit_assert_int(rv, ==, SQLITE_OK);
sqlite3_file *fp = main_file(db);
OK(sqlite3_exec(db, "CREATE TABLE foo (bar INTEGER)", NULL, NULL,
NULL));

struct vfs2_wal_slice sl;
rv = vfs2_poll(fp, NULL, &sl);
munit_assert_int(rv, ==, 0);
rv = vfs2_unhide(fp);
munit_assert_int(rv, ==, 0);
OK(vfs2_poll(fp, NULL, &sl));
OK(vfs2_unhide(fp));
munit_assert_uint32(sl.start, ==, 0);
munit_assert_uint32(sl.len, ==, 2);

rv = sqlite3_exec(db, "INSERT INTO foo (bar) VALUES (17)", NULL, NULL,
NULL);
munit_assert_int(rv, ==, SQLITE_OK);
tracef("aborting...");
rv = vfs2_abort(fp);
munit_assert_int(rv, ==, 0);
OK(sqlite3_exec(db, "INSERT INTO foo (bar) VALUES (17)", NULL, NULL,
NULL));
OK(vfs2_abort(fp));

rv = sqlite3_exec(db, "INSERT INTO foo (bar) values (22)", NULL, NULL,
NULL);
munit_assert_int(rv, ==, 0);
rv = vfs2_poll(fp, NULL, &sl);
munit_assert_int(rv, ==, 0);
OK(sqlite3_exec(db, "INSERT INTO foo (bar) values (22)", NULL, NULL,
NULL));
OK(vfs2_poll(fp, NULL, &sl));
munit_assert_uint32(sl.start, ==, 2);
munit_assert_uint32(sl.len, ==, 1);
rv = vfs2_unhide(fp);
munit_assert_int(rv, ==, 0);
OK(vfs2_unhide(fp));

sqlite3_stmt *stmt;
rv = sqlite3_prepare_v2(db, "SELECT * FROM foo", -1, &stmt, NULL);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_prepare_v2(db, "SELECT * FROM foo", -1, &stmt, NULL));
rv = sqlite3_step(stmt);
munit_assert_int(rv, ==, SQLITE_ROW);
munit_assert_int(sqlite3_column_count(stmt), ==, 1);
Expand All @@ -192,18 +187,15 @@ TEST(vfs2, basic, set_up, tear_down, 0, NULL)

int nlog;
int nckpt;
rv = sqlite3_wal_checkpoint_v2(db, "main", SQLITE_CHECKPOINT_PASSIVE,
&nlog, &nckpt);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_wal_checkpoint_v2(db, "main", SQLITE_CHECKPOINT_PASSIVE,
&nlog, &nckpt));
munit_assert_int(nlog, ==, 3);
munit_assert_int(nckpt, ==, 3);

rv = sqlite3_exec(db, "INSERT INTO foo (bar) VALUES (101)", NULL, NULL,
NULL);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_exec(db, "INSERT INTO foo (bar) VALUES (101)", NULL, NULL,
NULL));

rv = sqlite3_reset(stmt);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_reset(stmt));
rv = sqlite3_step(stmt);
munit_assert_int(rv, ==, SQLITE_ROW);
munit_assert_int(sqlite3_column_count(stmt), ==, 1);
Expand All @@ -213,19 +205,16 @@ TEST(vfs2, basic, set_up, tear_down, 0, NULL)
munit_assert_int(rv, ==, SQLITE_DONE);

dqlite_vfs_frame *frames;
rv = vfs2_poll(fp, &frames, &sl);
munit_assert_int(rv, ==, 0);
OK(vfs2_poll(fp, &frames, &sl));
munit_assert_uint(sl.len, ==, 1);
munit_assert_not_null(frames);
munit_assert_not_null(frames[0].data);
sqlite3_free(frames[0].data);
sqlite3_free(frames);

rv = vfs2_unhide(fp);
munit_assert_int(rv, ==, 0);
OK(vfs2_unhide(fp));

rv = sqlite3_reset(stmt);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_reset(stmt));
rv = sqlite3_step(stmt);
munit_assert_int(rv, ==, SQLITE_ROW);
munit_assert_int(sqlite3_column_count(stmt), ==, 1);
Expand All @@ -236,8 +225,7 @@ TEST(vfs2, basic, set_up, tear_down, 0, NULL)
rv = sqlite3_step(stmt);
munit_assert_int(rv, ==, SQLITE_DONE);

rv = sqlite3_finalize(stmt);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_finalize(stmt));
sqlite3_close(db);
return MUNIT_OK;
}
Expand Down Expand Up @@ -299,7 +287,6 @@ TEST(vfs2, startup_one_nonempty, set_up, tear_down, 0, NULL)
struct fixture *f = data;
struct node *node = &f->nodes[0];
char buf[PATH_MAX];
int rv;

snprintf(buf, PATH_MAX, "%s/%s", node->dir, "test.db");

Expand All @@ -309,14 +296,8 @@ TEST(vfs2, startup_one_nonempty, set_up, tear_down, 0, NULL)
make_wal_hdr(wal2_hdronly, 0, 17, 103);
prepare_wals(buf, NULL, 0, wal2_hdronly, sizeof(wal2_hdronly));
sqlite3 *db = open_test_db(node);
cole-miller marked this conversation as resolved.
Show resolved Hide resolved
sqlite3_file *fp;
sqlite3_file_control(db, "main", SQLITE_FCNTL_FILE_POINTER, &fp);
tracef("create table...");
rv = sqlite3_exec(db, "CREATE TABLE foo (n INTEGER)", NULL, NULL, NULL);
munit_assert_int(rv, ==, SQLITE_OK);
tracef("closing...");
rv = sqlite3_close(db);
munit_assert_int(rv, ==, SQLITE_OK);
OK(sqlite3_exec(db, "CREATE TABLE foo (n INTEGER)", NULL, NULL, NULL));
OK(sqlite3_close(db));

check_wals(buf, WAL_SIZE_FROM_FRAMES(2), WAL_SIZE_FROM_FRAMES(0));

Expand Down