Upgrade to tendermint v0.35.x #737

evan-forbes · 2022-04-27T19:38:58Z

Description

This PR inserts all of the celestia specific code into the tendermint v0.35.x branch. Fair warning, afaict the tests for v0.35.x are currently flakier than they were in v0.34.x. Sometimes certain statesync or p2p unit tests fail and need to be re-ran, but they seem to pass on the second attempt.

the brief companion ADR can be found in #739

Closes #567

Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]>

nmt_wrapper: add extend nmt wrapper with tree method patch filter test Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: Viacheslav Gonkivskyi <[email protected]>

Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: John Adler <[email protected]>

Co-authored-by: John Adler <[email protected]> Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]>

patch remainting tests update to go 1.17 switch back to using rsmt2d v0.5 patch protobuf encoding tests linter

* update with social links and careers section * make careers section not a subsection of resources Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Ismail Khoffi <[email protected]>

* Modify README.md - summarize LL and why it's useful - add banner - add further resources * Delet tokei.rs link seems down * Update README.md * Update community link * Update readme Go version badge. * Update community link * Update readme Go version badge. * Rebranding (#415) * new logo * Update README.md * review feedback * remove install + quick start sections as links are broken * finish rebrand and update golang Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Mustafa Al-Bassam <[email protected]> Co-authored-by: John Adler <[email protected]> Co-authored-by: rene <[email protected]>

Add messages when converting to protobuf version of block data (#580) * export parsing functions * add messages to proto version of data * remove redundant test * remove redundant test

* index malleated transactions properly * linter * fix malleated transaction indexing and event tracking * fix malleated event test * fix test * fix other test * update proto building and formatting to use the latest tendermint specs style generation and formatting * fix app tests linter fix max size tests

* Fix super linter CI * Update linter.yml * Port linter fix from #608 * Add missed ADRs * fix typo

* Use codeowners from master. * Remove reviewers for dependabot prs.

* add ability to create nmt proofs by progressively generating the square * rename var to be more specific * fix file name change * core logic and testing * export needed function * export needed function * generate nmtProof proto files * update remaining tests * update proto files * update test to use new format for verifying proofs * make tx proofs more ergonomic and handle multiple proofs * integrate into rpc core * cache original square size * move proving logic to pkg * add original square size * change name of function to TxInclusion * typo * linter * better docs Co-authored-by: John Adler <[email protected]> * use passed codec arg instead of using the default Co-authored-by: John Adler <[email protected]> * use already existing constant * return error for txSharePosition * bubble error instead of panicking * use uint cause it's the right thing to do even tho go fights you * include the lengths of bytes in the nmt proofs * use passed codec Co-authored-by: John Adler <[email protected]> * add comment to better explain division * linter Co-authored-by: John Adler <[email protected]>

* init adding DataCommitment rpc * updates data commitment json response name * fix import * Update rpc/client/interface.go Co-authored-by: John Adler <[email protected]> * Update rpc/core/blocks.go Co-authored-by: John Adler <[email protected]> * Update rpc/core/blocks.go Co-authored-by: John Adler <[email protected]> * lint * adds test for data commitment (still not working) * fixes data commitment creation * fixes data commitment test * adds data commitment RPC test * update comment * updates openapi specs * add blocks sanity check for data commitment * rename blockByQuery to heightsByQuery to reflect more what the function is doing * moves DataCommitmentBlocksLimit to consts * adds tests for data commitment limits Co-authored-by: John Adler <[email protected]> run make mockery patch compute shares for pkg files finish remaining rpc refactor

modify to use entire block data instead of only txs fix compile issue in kvstore linter use correct square size param in test remove persistent keystore modifying the second transaction of everyblock

mockery fix test gofmt fix test gofmt move UNKNOWN response behaviour to reject update mocks fix naming of some fields in RoundState fix base abci app to pass data and accept blocks

* add missing ProcessProposal handlers for the socket client * disable test_apps

try extending the deadline attempt to tune timer increase timeouts increase github CI timeout wait 5 seconds before checking on nodes patch test left over from rebase

chore: update proposal chore: add write adr template chore: add implement feature template chore: add refs to spec chore: minor change to proposal Update .github/ISSUE_TEMPLATE/write-adr.md Co-authored-by: John Adler <[email protected]> chore: remove a few lines from bug report

evan-forbes · 2022-05-03T21:12:53Z

@sweexordious please review 134eeef as I had to change a few things to the DataCommitment endpoint to adapt to a slight refactor of tendermint's rpc

liamsi · 2022-05-03T21:39:26Z

abci/example/kvstore/persistent_kvstore.go

+	// if len(req.BlockData.Txs) >= 1 {
+	// 	req.BlockData.Txs[0] = []byte("modified tx")
+	// }


I removed this comment, it was added in the original PR for a test I think, but then removed in later refactors. I originally commented it out, as it was causing tests to fail.

liamsi · 2022-05-03T21:40:44Z

abci/types/application.go

 	BeginBlock(RequestBeginBlock) ResponseBeginBlock // Signals the beginning of a block
 	DeliverTx(RequestDeliverTx) ResponseDeliverTx    // Deliver a tx for full processing
 	EndBlock(RequestEndBlock) ResponseEndBlock       // Signals the end of a block, returns changes to the validator set
 	Commit() ResponseCommit                          // Commit the state and return the application Merkle root hash
+	ProcessProposal(RequestProcessProposal) ResponseProcessProposal


Can we move this up to PrepareProposal and also a doc comment like everywhere else?

nice find, yeah totally 79b9561

I made them slightly celestia specific by mentioning block data, because there are no comments yet on upstream master, but we can switch to whatever those end up being in a future update

liamsi · 2022-05-03T21:42:06Z

docs/celestia-architecture/README copy.md

@@ -0,0 +1,37 @@
+---


is the filename intentionally README copy.md?

this was a remnant of the old master branch, I'm assuming no, and consolidated the readmes here a544403

liamsi · 2022-05-03T21:42:31Z

docs/celestia-architecture/README copy.md

+- [ADR 001: Erasure Coding Block Propagation](./adr-001-block-propagation.md)
+- [ADR 002: Sampling erasure coded Block chunks](./adr-002-ipld-da-sampling.md)
+- [ADR 003: Retrieving Application messages](./adr-003-application-data-retrieval.md)
+- [ADR 004: Data Availability Sampling Light Client](./adr-004-mvp-light-client.md)


This seems incomplete?

good catch, fixed here a544403

liamsi · 2022-05-04T07:13:34Z

pkg/prove/proof.go

+// It is possible that a transaction spans more than one row. In that case, we
+// have to return two proofs.


Orthogonal to this PR but doesn't this also support cases with more than 2 proofs?

yeah it should. We don't have an explicit test for this, though. I ran the existing test which uses random data a few thousand times, and a multiple proofs never showed up, so I'm not entirely convinced it is actually working. Although that could partially be the tests fault. We still don't support returning proofs for txs that span more than two shares #621 so that makes it more difficult to test this.

liamsi · 2022-05-04T07:19:26Z

types/block.go

-	// Data.Txs field:                      1 byte
-	MaxOverheadForBlock int64 = 11
+	// Data fields:                         6 bytes
+	// Hash in Data                        32 bytes


Isn't the overhead about protobuf overhead only? If so, how come we count the hash in data?

good point, added another var to hopefully make the distinction clearer

d3aadef

liamsi · 2022-05-04T07:20:39Z

types/block.go

+	// The messages included in this block.
+	// TODO: how do messages end up here? (abci) app <-> ll-core?
+	// A simple approach could be: include them in the Tx above and
+	// have a mechanism to split them out somehow? Probably better to include
+	// them only when necessary (before proposing the block) as messages do not
+	// really need to be processed by tendermint
+	Messages Messages `json:"msgs"`


Orthogonal to this PR but worth noting: This comment is outdated.

good point, removed here so we don't have to make an issue just for this 67ed4ac

liamsi · 2022-05-04T07:24:03Z

types/block.go

+	}
+	return w.Export()
+}
+


Orthogonal to this PR: we should consider moving the added methods and types (e.g. SplitIntoShares, Message etc into their own files instead. Such that it becomes more obvious which functionality we've added.

I completely agree! This was something that I want to do for v0.36, where we should have almost every change be either a deletion from an existing file, or an addition in a new file. Being relatively strict with that rule will be much more doable when we don't have to change large portions of the boiler plate code for ABCI

.github/workflows/linter.yml

Makefile

README.md

Co-authored-by: John Adler <[email protected]>

.github/workflows/e2e-nightly-34x.yml

… block

amodulardev

lgtm

evan-forbes self-assigned this Apr 27, 2022

evan-forbes changed the base branch from v0.34.x-celestia to v0.35.x-celestia April 27, 2022 19:39

evan-forbes added the T:tendermint Type: upstream tendermint changes we want label Apr 27, 2022

evan-forbes changed the title ~~anually upgrade v0.35~~ manually upgrade v0.35 Apr 27, 2022

evan-forbes force-pushed the evan/manually-upgrade-v0.35 branch from 4d5c293 to 780bae1 Compare May 3, 2022 14:11

add basic consts

12e1fa1

evan-forbes force-pushed the evan/manually-upgrade-v0.35 branch from 780bae1 to 0ac8bdf Compare May 3, 2022 18:56

evan-forbes and others added 19 commits May 3, 2022 14:12

add the data availability header and corresponding logic

116b7af

Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]>

add nmt wrapper

eee8f35

nmt_wrapper: add extend nmt wrapper with tree method patch filter test Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: Viacheslav Gonkivskyi <[email protected]>

Modify the block data to hold evidence and messages

86df652

Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]> Co-authored-by: John Adler <[email protected]>

add share merging and splitting

bf2d8b4

Co-authored-by: John Adler <[email protected]> Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Hlib Kanunnikov <[email protected]>

update to go 1.17

c4b4ed6

patch remainting tests update to go 1.17 switch back to using rsmt2d v0.5 patch protobuf encoding tests linter

update with social links and careers section (#542)

d59c60e

* update with social links and careers section * make careers section not a subsection of resources Co-authored-by: Ismail Khoffi <[email protected]> Co-authored-by: Ismail Khoffi <[email protected]>

change makeblock to accept messages

7685bac

Add messages when converting to protobuf version of block data (#580) * export parsing functions * add messages to proto version of data * remove redundant test * remove redundant test

add celestia ADRs

54f125d

Fix super linter CI (#713)

03e9219

* Fix super linter CI * Update linter.yml * Port linter fix from #608 * Add missed ADRs * fix typo

Fix codeowners (#699)

b8a7920

* Use codeowners from master. * Remove reviewers for dependabot prs.

abci: PrepareProposal (#6544)

07f9a05

modify to use entire block data instead of only txs fix compile issue in kvstore linter use correct square size param in test remove persistent keystore modifying the second transaction of everyblock

add processproposal proto/boilerplate/logic

2c9552d

mockery fix test gofmt fix test gofmt move UNKNOWN response behaviour to reject update mocks fix naming of some fields in RoundState fix base abci app to pass data and accept blocks

Temporarily disable socket client app test (#714)

14ad024

* add missing ProcessProposal handlers for the socket client * disable test_apps

increase timeouts for CI e2e test

e094b58

try extending the deadline attempt to tune timer increase timeouts increase github CI timeout wait 5 seconds before checking on nodes patch test left over from rebase

evan-forbes force-pushed the evan/manually-upgrade-v0.35 branch from 0ac8bdf to 1587994 Compare May 3, 2022 19:16

This was referenced May 3, 2022

ADR 008 Updating to tendermint v0.35.4 #739

Merged

Add celestia specific changes to tendermint v0.35.x #735

Closed

evan-forbes changed the title ~~manually upgrade v0.35~~ Upgrade to tendermint v0.35.x May 3, 2022

liamsi reviewed May 3, 2022

View reviewed changes

liamsi reviewed May 4, 2022

View reviewed changes

liamsi approved these changes May 4, 2022

View reviewed changes

adlerjohn requested changes May 4, 2022

View reviewed changes

.github/workflows/linter.yml Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

Makefile Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

evan-forbes and others added 9 commits May 4, 2022 10:19

use correct default branch

02a9dae

Co-authored-by: John Adler <[email protected]>

use the correct default branch in the readme

a5a3df7

Co-authored-by: John Adler <[email protected]>

replace the references of v0.34 with v0.35 in the e2e CI

4996539

delete comment

46b13bc

add comment to ABCI interface for new methods

79b9561

add missing ADR documentation and consolidate readmes

a544403

remove outdated comment

67ed4ac

remove unused var from the makefile

15686cc

add back proto-gen to proto-all in makefile

91ad715

adlerjohn requested changes May 4, 2022

View reviewed changes

.github/workflows/e2e-nightly-34x.yml Show resolved Hide resolved

evan-forbes added 2 commits May 4, 2022 13:56

rename file to reflect versioning change

e6cee7b

use a second variable to clarify the number of bytes used by an empty…

d3aadef

… block

evan-forbes requested a review from adlerjohn May 5, 2022 18:36

adlerjohn approved these changes May 5, 2022

View reviewed changes

evan-forbes merged commit 7497e2c into v0.35.x-celestia May 5, 2022

evan-forbes deleted the evan/manually-upgrade-v0.35 branch May 5, 2022 19:42

Magkoooh approved these changes Nov 28, 2024

View reviewed changes

amodulardev reviewed Nov 28, 2024

View reviewed changes

amodulardev approved these changes Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to tendermint v0.35.x #737

Upgrade to tendermint v0.35.x #737

evan-forbes commented Apr 27, 2022 •

edited

Loading

evan-forbes commented May 3, 2022 •

edited

Loading

liamsi May 3, 2022

evan-forbes May 4, 2022

liamsi May 3, 2022

evan-forbes May 4, 2022 •

edited

Loading

liamsi May 3, 2022

evan-forbes May 4, 2022 •

edited

Loading

liamsi May 3, 2022

evan-forbes May 4, 2022

liamsi May 4, 2022

evan-forbes May 4, 2022

liamsi May 4, 2022

evan-forbes May 4, 2022

liamsi May 4, 2022

evan-forbes May 4, 2022

liamsi May 4, 2022

evan-forbes May 4, 2022

amodulardev left a comment

		// It is possible that a transaction spans more than one row. In that case, we
		// have to return two proofs.

Upgrade to tendermint v0.35.x #737

Upgrade to tendermint v0.35.x #737

Conversation

evan-forbes commented Apr 27, 2022 • edited Loading

Description

evan-forbes commented May 3, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes May 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-forbes May 4, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amodulardev left a comment

Choose a reason for hiding this comment

evan-forbes commented Apr 27, 2022 •

edited

Loading

evan-forbes commented May 3, 2022 •

edited

Loading

evan-forbes May 4, 2022 •

edited

Loading

evan-forbes May 4, 2022 •

edited

Loading