Create a standardized (probably-inefficient) frozen database.

[manreo]

This proposal is for the standardization of a (probably-inefficient) frozen database. The emphasis in this database is readability and clarity and not so much about efficiency. The db contain enough information so that one can resume a node using this file alone.

The proposal is that all implementation will support the export to this standardized-database (the ability to import from the standardized-database to each implementation database format is not related).

I can imagine something like this:

MrMan@Computer1:~$ c-lightning export stand-db
Warning!
You are about to export your database to a standardized version, be careful!
For security reasons, this will result in a shutdown and lock of your node. Loading this standardized-db in a different machine and resuming your node can result in funds lost!

Why?

1. First, for the obvious reason, it is clear that lnd has >90% of the nodes, this is nice for lnd, but I assume that even they do not want to see the other implementations give-up and stop their development. Obviously, if there will be fewer implementations, the network and protocol will be weaker.
   Now that we got this out of the way
2. I think that having a standardized-db will help to verify that the specification is clear. I have a feeling that  Clarify relative order of some messages after reestablish #810 and New channel with ACQIN is not active lightningnetwork/lnd#4575 would have been avoided if there was a standardized-db.
3. Easier way to move between machines - if, for example, one sees that his machine is becoming unstable, and he wants to move to a different one (hopefully from arm to amd64), it will be much easier to do lnd load than just copy all files and hope that it works.
4. Improve testing, I have a feeling that an automated test that exports the regular-db to the standardized-db, and then import it to all different implementations will be a good "?fuzzing?" test.
5. Don't be evil. Locking someone to a specific implementation is never nice. Currently, it is easier to move out of Google-photos than it is to move between lightning implementations.

Others

A readable standardized-db will probably take more storage, but I don't think it is a problem as this step will not be regularly executed. And only by people who understand what they are doing. It is important to estimate the amount of storage required before starting the export.

csv_data_from_bigsun

df = pd.read_csv("from_bigsun.txt")
df.implementation.value_counts()
lnd 6459
c-lightning 577
eclair 153

@cdecker @rustyrussell @ZmnSCPxj @Roasbeef 
Please feel free to edit my proposal to add more reasons. I am aware of my lack of knowledge in the field.

[fluxdeveloper]

Reading CSV files is error prone, and they're not very fit for expressing relations. There's little point in each implementation duplicating this functionality when they're probably just going to load the data into a relational DB anyway.

Consider using Sqlite for any interop format. It is simple, available anywhere and probably what any mobile wallet is going to use anyway. If you need a readable CSV file, it can be extracted from the db with a simple query.

[manreo]

Hi @fluxdeveloper thank for the replay. I never said that is should be a CSV.
I'm not very proficient in databases, Sqlite might be good, I assume most of the difficulty will be in saving channels in closing/opening/limbo states.
Thanks

[t-bast]

I agree that this would be desirable for end-users, and implementing export/import to/from a single format is a much better idea than implementing one for each implementation's format. Curious to hear other implementers' feedback, but I'm in favor of this.

[t-bast]

Note that one difficulty may be that seeds are also different depending on the implementations...
And the way keys are derived from the seeds differ as well, it may not be an issue but it's worth looking into, it could be problematic.

[ZmnSCPxj]

A big issue that does not seem to be considered yet is: should the standardized database be complete enough to also include private keys?

Each implementation has its own way of generating private keys for channel, I believe most or all of them use some deterministic derivation method, but because this is not standardized, each implementation uses a different deterministic derivation method (that may or may not be BIP32-compatible). If so, it seems to me that a standardized database is fairly useless, as each implementation cannot do anything with the channels exported by another implementation, even if they were given the root private key in a separate mechanism. We would have to first standardize the derivation from the root private key to the per-channel key, which will be painful for at least one implementation (basically if it is not the derivation method used by your implementation now you are going to cry rivers of blood).

If the standardized database contains private keys, then that it sufficient to steal funds and any mechanism to create a standardized database must be put in a higher-security domain (in e.g. the same way that C-Lightning cordons off the private keys in a separate process lightning_hsmd that is the only process that opens the file containing the root private keys). We would have to have strict warnings to users that the database contains enough data to steal funds if somebody else gets to it. Note in particular that C-Lightning does not keep private keys in the database, so C-Lightning devs regularly ask users to give copies of the db to us for debugging etc; if they send the standardized database containing private keys rather than the C-Lightning-specific db, that would be extremely troublesome, as devs would now be responsible for partially-custodial control of user funds.

Probably the wisest thing to do would be to have both:

• A standardized format for channel data.
  • Ideally, by itself, it should be enough to close channels in an emergency, with caveats and possible loss of funds if some features like opiton_dataloss_protect or option_static_pubkey were not negotiated at channel opening.
• A *separate standardized format for channel private keys, with lots of warnings regarding how the channel private keys are enough to have the funds stolen.
  • In combination with the above standard channel database, it should at least be enough to close channels mutually even without option_dataloss_protect or option_static_pubkey as long as the database is the most recent snapshot and no channel updates occurred after the database snapshot (e.g. if an implementation supports an offline snapshotting mode, if you shut down the implementation, then take a standardized database snapshot, then erase the original node, in combination with this channel private keys snapshot, you can recreate the state enough on a different implementation to at least do a mutual close, or a safe unilateral close).

[manreo]

Thanks @ZmnSCPxj and @t-bast

I understand that

(basically if it is not the derivation method used by your implementation now you are going to cry rivers of blood).

But can I assume that implementing it now will cost much less blood than implementing it a year from now?

About the channel private keys, I don't know how large this file is, but my assumption is that this file is not very complicated and not very large. If this is true, than maybe for this db readability is not very important, and it can be secure/compressed/encrypted.

Thanks

[t-bast]

We discussed this during the spec meeting (#821) and the overall consensus from implementers was that this would be undoable in practice.

Maybe in a future world when the lightning stack is more stable, but not now.

[mateuszmp]

I wish this issue is revised to allow migration between lnd and CLN without channel closures