invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

bhandras · 2024-06-12T10:35:27Z

Change Description

This pull request adds the migration of old key-value (KV) invoices to the new native SQL schema when the --db.use-native-sql flag is set, unless the --db.skip-sql-invoice-migration flag is also specified.

Please note that since we currently do not support running on mixed database backends for users of bbolt or etcd, an additional step is required to migrate their KV database to SQL first. For more context, please see lightninglabs/lndinit#21.

coderabbitai · 2024-06-12T10:35:32Z

Important

Review skipped

Auto reviews are limited to specific labels.

🏷️ Labels to auto review (1)

llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai generate unit testing code for this file.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and generate unit testing code.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

aakselrod · 2024-09-05T17:38:12Z

invoices/sql_migration.go

+		var paymentHash lntypes.Hash
+		if invoice.Terms.PaymentPreimage != nil {
+			paymentHash = invoice.Terms.PaymentPreimage.Hash()
+		} else {


This doesn't correctly handle the case of an invoice generated by receiving a spontaneous AMP payment, which should have an empty payment hash IIUC. To test this, run the migration on Bob's DB resulting from the itest sendpayment_amp_invoice/external_payaddr.

Nice catch, thank you for pointing this out! Fix is coming soon as well as the rest of the missing parts. Migrating spontaneous AMP invoices required building an index at migration time so we're able to map add indexes to payment hashes.

looks like spontaneous amp payments still have a valid payment hash tho or ?

Previously we intentially did not set settled_at and settle_index when inserting a new invoice as those fields are set when we settle an invoice through the usual invoice update. As migration requires that we set these nullable fields, we can safely add them.

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

Certain invoices may not have a deterministic payment hash. For such invoices we still store the payment hashes in our KV database, but we do not have a sufficient index to retrieve them. This PR adds such index to the SQL database that will be used during migration to retrieve payment hashes.

This commit runs the invoice migration if the user has a KV SQL backend configured.

aakselrod · 2024-10-02T20:26:55Z

Sorry for the lack of review from me, I'll re-review late this week/early next week! I'm just now getting back to working on this project after some other higher-priority stuff landed on my plate.

ProofOfKeags · 2024-10-01T22:37:49Z

go.mod

+// Temporary replace until https://github.com/lightningnetwork/lnd/pull/8831 is
+// merged.
+replace github.com/lightningnetwork/lnd/sqldb => ./sqldb


Seems like this commit should be dropped.

ProofOfKeags · 2024-10-03T21:12:45Z

sqldb/sqlc/amp_invoices.sql.go

@@ -235,6 +235,34 @@ func (q *Queries) GetAMPInvoiceID(ctx context.Context, setID []byte) (int64, err
 	return invoice_id, err
 }

+const insertAMPSubInvoice = `-- name: InsertAMPSubInvoice :execresult


I'd love if you could explain how this magic string is consumed by things. I figured out that it has to do with code generation but I'd like to know what tool consumes this and creates the actual go code InsertAMPSubInvoice

ProofOfKeags · 2024-10-03T21:29:08Z

sqldb/sqlc/migrations/000005_invoice_migration.up.sql

+-- migration_tracker is a table that keeps track of custom migrations that have
+-- been run on the database. This table is used to ensure that migrations are
+-- idempotent and are only run once.
+CREATE TABLE IF NOT EXISTS migration_tracker (
+        -- migration_id is the id of the migration.
+        migration_id TEXT NOT NULL,
+
+        -- migration_ts is the timestamp at which the migration was run.
+        migration_ts TIMESTAMP,
+
+        PRIMARY KEY (migration_id)
+);


How did we handle this before?

ProofOfKeags · 2024-10-03T21:30:00Z

sqldb/sqlc/migrations/000005_invoice_migration.up.sql

+        migration_id TEXT NOT NULL,
+
+        -- migration_ts is the timestamp at which the migration was run.
+        migration_ts TIMESTAMP,


If there isn't a really solid reason to abbreviate this as ts I'd rather spell it out as timestamp or just time

ProofOfKeags · 2024-10-07T09:16:51Z

sqldb/sqlc/migrations/000005_invoice_migration.down.sql

@@ -1 +1,2 @@
+DROP TABLE IF EXISTS migration_tracker;


I wonder if this makes more sense to go at the beginning of the PR series. Not a big deal but it seems like this is unrelated to the core of the PR and is more around to give us atomic/idempotence guarantees on migrations.

ProofOfKeags · 2024-10-07T09:40:01Z

invoices/sql_migration.go

+					len(k))
+			}
+
+			copy(paymentHash[:], k)


Maybe a dumb question, but why not just put k directly into InsertInvoicePaymentHashAndKeyParams{...}

Maybe it was a defense mechanism before loopvar was introduced ?

ProofOfKeags · 2024-10-07T09:58:25Z

invoices/sql_migration.go

@@ -135,3 +139,222 @@ func createInvoiceHashIndex(ctx context.Context, db kvdb.Backend,
 		})
 	}, func() {})
 }
+
+// MigrateSingleInvoice migrates a single invoice to the new SQL schema. Note
+// that prefect equality between the old and new schema is not possible as the


s/prefect/perfect/g

ProofOfKeags · 2024-10-07T10:07:58Z

invoices/sql_migration.go

+			InvoiceID:    invoiceID,
+		}
+
+		if htlc.MppTotalAmt != 0 {


Why are we excluding the 0 case?

because non-mpp payments don't have this value so we want to keep it null in that case.

ProofOfKeags · 2024-10-07T10:08:12Z

invoices/sql_migration.go

+			)
+		}
+
+		if !htlc.ResolveTime.IsZero() {


Why are we excluding the 0 case here too?

we want to keep the resolve time null for still unresolved htlcs

ProofOfKeags · 2024-10-07T10:23:22Z

invoices/sql_migration_test.go

+// have the AMP feature bit set and contain AMP HTLCs. Note that this function
+// is not supposed to generate correct invoices, but rather random invoices for
+// testing the SQL migration.
+func generateTestInvoice(t *testing.T, mpp bool, amp bool) *Invoice {


Since this is just a migration code test, and therefore its expected lifetime is small I don't think you should change this, however, it is worth noting that much of this code could have been avoided by using rapid for all of the generation. I mention this just so that next time you encounter the need for this you need not roll your own random gen stuff. We introduced it in #9032

ziggie1984

Really great work 👌, had a first pass.

Now trying it out on some dbs and different systems to see if we run into memory issues when migrating dbs > 1 mio invoices.

ziggie1984 · 2024-10-17T12:55:20Z

itest/lnd_invoice_migration_test.go

+	}
+
+	// Skip the test if the node is already running with native SQL.
+	if bob.Cfg.NativeSQL {


hmm but does this mean bob already migrated ?

ziggie1984 · 2024-10-18T15:00:23Z

config_builder.go

+		if !d.cfg.DB.SkipSQLInvoiceMigration {
+			err = invoices.MigrateInvoicesToSQL(
+				ctx, dbs.GraphDB.Backend, dbs.GraphDB,
+				sqlInvoiceDB, 10,


Nit: define defaults for migration batching

ziggie1984 · 2024-10-19T10:16:55Z

invoices/sql_migration.go

+
+	// If the invoice is settled, we'll also set the timestamp and the index
+	// at which it was settled.
+	if invoice.State == ContractSettled {


Should we log an error with we encounter a settled state which has no index and no settledate ?

ziggie1984 · 2024-10-19T12:24:55Z

invoices/sql_store.go

+	}
+
+	if invoice.Memo != nil {
+		params.Memo = sql.NullString{


use sqldb.SQLStr

ziggie1984 · 2024-10-19T12:31:48Z

invoices/sql_migration.go

+	"github.com/lightningnetwork/lnd/lntypes"
+	"github.com/lightningnetwork/lnd/sqldb/sqlc"
+)
+


could you please describe which invoices have no payment hash in itself but still have the payment_hash index filled ?

ziggie1984 · 2024-10-20T22:22:03Z

invoices/kv_sql_migration_test.go

+
+		const batchSize = 11
+		err := invpkg.MigrateInvoicesToSQL(
+			ctxb, store1.Backend, store1, store2, batchSize,


call store1 boltdb maybe ?

ziggie1984 · 2024-10-20T22:23:59Z

invoices/kv_sql_migration_test.go

+			IndexOffset: 0,
+			// As a sanity check, fetch more invoices than we have
+			// to ensure that we did not add any extra invoices.
+			NumMaxInvoices: 9999,


do we know how much the db had initially would be an important information here.

ziggie1984 · 2024-10-20T22:29:54Z

itest/lnd_invoice_migration_test.go

+	result1, err := db.QueryInvoices(ht.Context(), query)
+	require.NoError(ht, err)
+
+	numInvoices := len(result1.Invoices)


why don't we do a length check here for 35 invoices ?

ziggie1984 · 2024-10-20T22:32:23Z

itest/lnd_invoice_migration_test.go

+	bob.SetExtraArgs([]string{"--db.use-native-sql"})
+
+	// Now run the migration flow three times to ensure that each run is
+	// idempotent.


But I do understand it correctly that even though we run it 3 times, the migration only happens once when we start bob via the native sql flag at the beginning ?

ziggie1984 · 2024-10-20T22:37:42Z

itest/lnd_misc_test.go

-// testNativeSQLNoMigration tests that nodes that have invoices would not start
-// up with native SQL enabled, as we don't currently support migration of KV
-// invoices to the new SQL schema.
-func testNativeSQLNoMigration(ht *lntest.HarnessTest) {


but we still have a check in place that as soon as I start a boltdb version if the nativesql flag set, that it will fail with a detailed error that these two settings are not supported ?

lightninglabs-deploy · 2024-11-04T00:34:54Z

@bhandras, remember to re-request review from reviewers when ready

bhandras self-assigned this Jun 12, 2024

bhandras added database Related to the database/storage of LND migration labels Jun 12, 2024

bhandras added this to the 0.19.0 milestone Jun 12, 2024

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from 6682b50 to 338e1f0 Compare June 12, 2024 15:30

bhandras force-pushed the sql-invoice-migration branch 3 times, most recently from d2a329f to 6379a8b Compare June 21, 2024 17:37

bhandras mentioned this pull request Jul 31, 2024

mod: bump kvdb to v1.4.10 #8959

Merged

bhandras force-pushed the sql-invoice-migration branch 2 times, most recently from 5fe92e2 to a7bf598 Compare August 14, 2024 09:38

aakselrod reviewed Sep 5, 2024

View reviewed changes

bhandras force-pushed the sql-invoice-migration branch 5 times, most recently from b6f0ac8 to b983851 Compare September 17, 2024 14:42

bhandras changed the title ~~[wip] invoices: migrate KV invoices to native SQL~~ invoices: migrate KV invoices to native SQL for users of KV SQL backends Sep 17, 2024

bhandras force-pushed the sql-invoice-migration branch from b983851 to 706b444 Compare September 17, 2024 14:51

bhandras marked this pull request as ready for review September 17, 2024 14:51

bhandras force-pushed the sql-invoice-migration branch 7 times, most recently from 96f0cbe to bfe4ad5 Compare September 19, 2024 15:08

bhandras requested a review from aakselrod September 19, 2024 15:11

saubyk requested review from ziggie1984 and removed request for aakselrod September 19, 2024 15:11

bhandras force-pushed the sql-invoice-migration branch 4 times, most recently from d30d10c to 6a6074c Compare September 20, 2024 14:08

bhandras requested a review from aakselrod September 20, 2024 17:30

saubyk requested review from ProofOfKeags and removed request for aakselrod October 1, 2024 16:46

bhandras added 13 commits October 1, 2024 22:16

mod: temporarily replace sqldb with local version

ba1c6fb

multi: add call to directly insert an AMP sub-invoice

cd2b552

sqldb: add a temporary index to store KV invoice hash to ID mapping

74bd006

sqldb: add table to track custom SQL migrations

5987960

This commit adds the migration_tracker table which we'll use to track if a custom migration has already been done.

invoices: extract method to create invoice insertion params

d42abea

invoices: add migration code for a single invoice

0078293

invoices: add migration code that runs a full invoice DB SQL migration

7a4dd7f

lnd: run invoice migration on startup

29a2ce7

This commit runs the invoice migration if the user has a KV SQL backend configured.

itest: add integration test for invoice migration

27293cf

itest: remove obsolete itest

812a052

docs: update release notes for 0.19.0

e5b6d81

bhandras force-pushed the sql-invoice-migration branch from 6a6074c to e5b6d81 Compare October 1, 2024 20:16

ProofOfKeags reviewed Oct 7, 2024

View reviewed changes

ziggie1984 reviewed Oct 20, 2024

View reviewed changes

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

Are you sure you want to change the base?

invoices: migrate KV invoices to native SQL for users of KV SQL backends #8831

Conversation

bhandras commented Jun 12, 2024 • edited Loading

Change Description

coderabbitai bot commented Jun 12, 2024 • edited Loading

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aakselrod commented Oct 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ziggie1984 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightninglabs-deploy commented Nov 4, 2024

bhandras commented Jun 12, 2024 •

edited

Loading

coderabbitai bot commented Jun 12, 2024 •

edited

Loading

CodeRabbit Configuration File (`.coderabbit.yaml`)