Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade bech32 dependency (iterative) #3270

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

optout21
Copy link
Contributor

@optout21 optout21 commented Aug 23, 2024

Continuation of #3244 . Fixes #3176 .

Changes summary:

  • Upgraded bech32 dependency to 0.11.0 (from 0.9)
  • The previously used type bech32::u5 can be replaced by bech32::Fe32, mostly as a drop-in replacement.
  • u5::try_from_u8 is now Fe32::try_from
  • bech32::Error is mostly replaced by bech32::primitives::decode::CheckedHrpstringError
  • ToBase32 (and Base32Len) traits are discontinued in bech32, they have been reimplemented in iterative style as Base32Iterable. The implementations (Bolt11InvoiceFeatures/Payment*/...) have been moved to lightning-invoice crate (as the trait is defined there)
  • FromBase32 trait is discontinued in bech32, it has been taken over to lighning-invoice crate. The implementations have been moved to lightning-invoice crate (as the trait is defined there)

Here are some minor issues identified

  • Hrp/checksum parsing has changed (uses new iterative API)
  • Fe32 does not have Ord, so it had to be explicitly implemented for RawTaggedField
  • Fe32 does not have Default, for which some workaround was needed. This will be added in next bech32 0.12 release (see Add Ord trait to Fe32 (derived) rust-bitcoin/rust-bech32#186)
  • Conversions of bech32 error types are not always optimal
  • Parsing of tagged values in an invoice stays own implementation (not supported by bech32 crate)

TODO:

  • fix fuzz target
  • fix fmt
  • squash
  • Incremental-mutants

@optout21 optout21 force-pushed the bech32-iterser branch 2 times, most recently from 9372141 to b677ddb Compare August 23, 2024 22:00
Copy link

codecov bot commented Aug 23, 2024

Codecov Report

Attention: Patch coverage is 93.87755% with 27 lines in your changes missing coverage. Please review.

Project coverage is 89.64%. Comparing base (66fb520) to head (7da1b51).

Files with missing lines Patch % Lines
lightning-invoice/src/de.rs 86.99% 10 Missing and 6 partials ⚠️
lightning/src/offers/parse.rs 63.63% 2 Missing and 2 partials ⚠️
lightning-invoice/src/test_ser_de.rs 97.00% 0 Missing and 3 partials ⚠️
lightning-invoice/src/lib.rs 96.55% 2 Missing ⚠️
lightning-invoice/src/ser.rs 98.65% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3270      +/-   ##
==========================================
- Coverage   89.64%   89.64%   -0.01%     
==========================================
  Files         126      126              
  Lines      102676   102753      +77     
  Branches   102676   102753      +77     
==========================================
+ Hits        92045    92109      +64     
- Misses       7912     7932      +20     
+ Partials     2719     2712       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@optout21 optout21 force-pushed the bech32-iterser branch 5 times, most recently from 8eb0c4b to 1fa1e99 Compare August 26, 2024 14:59
@optout21 optout21 changed the title [Draft] Upgrade bech32 dependency (iterative) Upgrade bech32 dependency (iterative) Aug 27, 2024
@optout21 optout21 marked this pull request as ready for review August 27, 2024 19:17
@optout21 optout21 mentioned this pull request Aug 29, 2024
4 tasks
@optout21
Copy link
Contributor Author

CI: A few misses in iterative-mutants on the Bolt11InvoiceFeatures serialization method (unchanged). To see if checks can be added, or the method can be rewritten in a simpler form.

@optout21 optout21 force-pushed the bech32-iterser branch 7 times, most recently from 55e4259 to 5fa337a Compare August 31, 2024 04:50
@optout21
Copy link
Contributor Author

CI: All builds OK. Iterative mutants is satisfied now, but the build was interrupted (?) after the tests (verified the logs + locally).

lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/lib.rs Outdated Show resolved Hide resolved
lightning-invoice/src/ser.rs Outdated Show resolved Hide resolved
lightning-invoice/src/ser.rs Show resolved Hide resolved
lightning-invoice/src/de.rs Show resolved Hide resolved
@optout21 optout21 force-pushed the bech32-iterser branch 2 times, most recently from 9abd92e to ca0e089 Compare September 8, 2024 15:01
@optout21
Copy link
Contributor Author

optout21 commented Sep 8, 2024

Added a minor cleanup: hash_from_parts is called from a single place (from signable_hash(); in the other place in the SignedRawBolt11Invoice parser, it's called through signable_hash()), this way one version is sufficient.

@optout21
Copy link
Contributor Author

optout21 commented Sep 8, 2024

A note regarding bech32->bytes conversion with padding:
This is needed currently when computing hash of an invoice (hash_from_parts), and, if we want to switch for Features using bech32 library (as mentioned in a review comments above), also in the Features decoding.

I've raised an issue to rust-bech32, to possible offer an option for bech32->bytes conversion where last bits are not dropped but padded, that would be the nice solution.

I also realized padding can be done in an iterator adapter (counts the elements and and the end optionally outputs padding), so this padding can be applied nicely in the chained iterators approach (as discusses above).

@optout21
Copy link
Contributor Author

optout21 commented Sep 9, 2024

Awaiting (re)review, @TheBlueMatt . Thx.

Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is shaping up nicely. We took this opportunity to avoid allocations in the serialization pipeline (sadly with allocations to avoid GATs...) but there's also a ton of really useless allocations in the deserialization pipeline, some of which we can trivially remove. We certainly don't have to fix it all in one go, but it'd be nice to reduce them where its easy here.

In any case, please feel free to go ahead and rewrite the commit history to have a logical progression so its easy to review :)

lightning-invoice/src/test_ser_de.rs Show resolved Hide resolved
lightning-invoice/src/de.rs Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Show resolved Hide resolved
lightning-invoice/src/lib.rs Outdated Show resolved Hide resolved
lightning-invoice/src/lib.rs Outdated Show resolved Hide resolved
lightning-invoice/src/lib.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
@optout21
Copy link
Contributor Author

Commits squashed

lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Outdated Show resolved Hide resolved
lightning-invoice/src/de.rs Show resolved Hide resolved
lightning-types/Cargo.toml Outdated Show resolved Hide resolved
@TheBlueMatt
Copy link
Collaborator

We should get a second reviewer on this.

fuzz/Cargo.toml Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade bech32 dependency
3 participants