ADR 008 Updating to tendermint v0.35.4

docs/celestia-architecture/adr-007-minimal-changes-to-tendermint.md

@@ -0,0 +1,237 @@
+# ADR 007: From Ukraine, with Love
+
+## Changelog
+
+- 2021-08-20: Initial Description
+- 2022-05-03: Update pointing to ADR 008
+
+## Context
+
+Currently, our fork of tendermint includes changes to how to erasure block data, minor changes to the header to commit
+to that data, additions to serve data availability sampling, along with some miscellaneous modification to adhere to the
+spec. Instead of incorporating all of these changes into our fork of tendermint, we will only make the strictly
+necessary changes and the other services and their code to the new celestia-node repo. Notably, we will also refactor
+some of the remaining necessary changes to be more isolated from the rest of the tendermint codebase. Both of these
+strategies should significantly streamline pulling updates from upstream, and allow us to iterate faster since most
+changes will be isolated to celestia-node.
+
+Update: many of the changes described below have since been minimized or removed. Please see ADR 008 for a summarized list of changes. Notably, we removed intermediate state roots, adopted two new methods from ABCI++ instead of PreprocessTxs, and are still signing over the PartSetHeader. 
+
+## Decision
+
+Treat tendermint more as a "black box".
+
+## Detailed Design
+
+### Overview
+
+We keep the bare-minimum changes to tendermint in our fork, celestia-core. Where necessary and possible we augment the
+tendermint node in a separate process, via celestia-node, which communicates with the tendermint node via RPC. All data
+availability sampling logic, including all Celestia-specific networking logic not already provided by tendermint, is
+moved into celestia node:
+
+![core node relation](./img/core-node-relation.png)
+
+The detailed design of celestia-node will be defined in the repository itself.
+
+### Necessary changes to tendermint
+
+#### Changing the repo import names to celestiaorg
+
+- Rebrand (https://github.com/celestiaorg/celestia-core/pull/476)
+
+#### Changes to the README.md other basic things
+
+- update github templates (https://github.com/celestiaorg/celestia-core/pull/405)
+- update README.md (https://github.com/celestiaorg/celestia-core/pull/10)
+
+#### Adding the extra types of block data
+
+- Update core data types (https://github.com/celestiaorg/celestia-core/pull/17)
+    - Create the Message/Messages types
+    - Proto and the tendermint version
+    - Create the IntermediateStateRoots type
+    - Proto and the tendermint version
+- Data availability for evidence (https://github.com/celestiaorg/celestia-core/pull/19)
+    - Add both types to `types.Data`
+    - Modify proto
+    - Add `EvidenceData` to `types.Data`
+
+#### Add the HeaderHash to the Commit
+
+- Add header hash to commit(https://github.com/celestiaorg/celestia-core/pull/198)
+
+#### Adding the consts package in types
+
+#### Remove iavl as a dependency
+
+- remove iavl as a dependency (https://github.com/celestiaorg/celestia-core/pull/129)
+
+#### Using the `DataAvailabilityHeader` to calculate the DataHash
+
+The `DataAvailabilityHeader` struct will be used by celestia-core as well as by the celestia-node. It might make sense
+to (eventually) move the struct together with all the DA-related code into a separate repository and go-module.
+@Wondertan explored this as part of [#427](https://github.com/celestiaorg/celestia-core/pull/427#issue-674512464). This
+way all client implementations can depend on that module without running into circular dependencies. Hence, we only
+describe how to hash the block data here:
+
+- Update core types (https://github.com/celestiaorg/celestia-core/pull/17)
+    - Replace the `Data.Hash()` with `DAH.Hash()`
+    - Use DAH to fill DataHash when filling the header
+    - Fill the DAH when making a block to generate the data hash
+
+#### Add availableDataOriginalSharesUsed to the header
+
+- Add availableDataOriginalSharesUsed to the header (https://github.com/celestiaorg/celestia-core/pull/262)
+
+#### Reap some number of transactions probably using the app or some other mech
+
+- Enforce a minimum square size (https://github.com/celestiaorg/celestia-core/pull/282)
+- Use squares with a width that is a power of two(https://github.com/celestiaorg/celestia-core/pull/331)
+- Adopt reamping from the mempool to max square size (https://github.com/celestiaorg/celestia-core/issues/77)
+- Proposal: Decide on a mech to pick square size and communicate that to the
+  app (https://github.com/celestiaorg/celestia-core/issues/454)
+- Also see ABCI++ for a less hacky solution
+
+#### Filling the DAH using share merging and splitting
+
+- Compute Shares (not merged) (https://github.com/celestiaorg/celestia-core/pull/60)
+    - part II (not merged) (https://github.com/celestiaorg/celestia-core/pull/63)
+    - while this was not merged, we will need some function to compute the shares that make up the block data
+- Share Splitting (https://github.com/celestiaorg/celestia-core/pull/246)
+    - Serialize each constituent of block data
+    - Split into shares
+        - Txs (contiguous)
+        - Messages (not contiguous)
+        - Evidence (contiguous)
+        - IntermediateStateRoots (contiguous)
+- Combine shares into original square
+- ExtendBlockData
+- Generate nmt root of each row and col
+- Use those roots to generate the DataHash
+- Share Merging (https://github.com/celestiaorg/celestia-core/pull/261)
+    - Sort by namespace
+    - Parse each reserved type
+    - Parse remaining messages
+
+#### Add the wrapper around nmt to erasure namespaces
+
+- Implement rsmt tree wrapper for nmt (https://github.com/celestiaorg/celestia-core/pull/238)
+
+#### Add PreprocessTxs to ABCI
+
+- Add PreprocessTxs method to ABCI (https://github.com/celestiaorg/celestia-core/pull/110)
+- Add method to ABCI interface
+- Create sync and async versions
+- Add sync version the the CreateProposalBlock method of BlockExecutor
+
+#### Fill the DAH while making the block
+
+- Basic DA functionality (https://github.com/celestiaorg/celestia-core/pull/83)
+
+#### Only produce blocks on some interval
+
+- Control block times (https://github.com/tendermint/tendermint/issues/5911)
+
+#### Stop signing over the PartSetHeader
+
+- Replace canonical blockID with just a hash in the CononicalVote
+- Replace the LastBlockID in the header with just a hash
+
+#### Optionally remove some unused code
+
+- Removing misc unsued code (https://github.com/celestiaorg/celestia-core/pull/208)
+- Remove docs deployment (https://github.com/celestiaorg/celestia-core/pull/134)
+- Start deleting docs (https://github.com/celestiaorg/celestia-core/pull/209)
+- Remove tendermint-db in favor of badgerdb (https://github.com/celestiaorg/celestia-core/pull/241)
+- Delete blockchain 2 until further notice (https://github.com/celestiaorg/celestia-core/pull/309)
+- We don’t need to support using out of process apps
+
+#### Nice to Haves
+
+- More efficient hashing (https://github.com/celestiaorg/celestia-core/pull/351)
+
+We should also take this opportunity to refactor as many additions to tendermint into their own package as possible.
+This will hopefully make updating to future versions of tendermint easier. For example, when we fill the data
+availability header, instead of using a method on `Block`, it could be handled by a function that takes `types.Data` as
+input and returns the DAH, the number of shares used in the square, along with the obligatory error.
+
+```go
+func FillDataAvailabilityHeader(data types.Data) (types.DataAvailabilityHeader, numOrigDataShares, error)
+```
+
+We could perform a similar treatment to the `splitIntoShares` methods and their helper method `ComputeShares`. Instead
+of performing the share splitting logic in those methods, we could keep it in a different package and instead call the
+equivalent function to compute the shares.
+
+Beyond refactoring and some minor additions, we will also have to remove and revert quite a few changes to get to the
+minimum desired changes specified above.
+
+### Changes that will need to be reverted
+
+#### IPLD Plugin
+
+- Introduction (https://github.com/celestiaorg/celestia-core/pull/144)
+- Initial integration (https://github.com/celestiaorg/celestia-core/pull/152)
+- Custom Multihash (https://github.com/celestiaorg/celestia-core/pull/155)
+- Puting data during proposal (https://github.com/celestiaorg/celestia-core/pull/178)
+- Module name (https://github.com/celestiaorg/celestia-core/pull/151)
+- Update rsmt2d (https://github.com/celestiaorg/celestia-core/pull/290)
+- Make plugin a package (https://github.com/celestiaorg/celestia-core/pull/294)
+
+#### Adding DAH to Stuff
+
+- Adding DAH to Proposal (https://github.com/celestiaorg/celestia-core/pull/248/files)
+- Blockmeta (https://github.com/celestiaorg/celestia-core/pull/372)
+
+#### Embedding DAS
+
+- GetLeafData (https://github.com/celestiaorg/celestia-core/pull/212)
+- RetrieveBlockData (https://github.com/celestiaorg/celestia-core/pull/232)
+- ValidateAvailability (https://github.com/celestiaorg/celestia-core/pull/270)
+- Prevent double writes to IPFS (https://github.com/celestiaorg/celestia-core/pull/271)
+- Stop Pinning (https://github.com/celestiaorg/celestia-core/pull/276)
+- Rework IPFS Node (https://github.com/celestiaorg/celestia-core/pull/334)
+- Refactor for putting the block (https://github.com/celestiaorg/celestia-core/pull/338)
+- Config for IPFS node (https://github.com/celestiaorg/celestia-core/pull/340)
+- IPLD Dag instead of CoreAPI (https://github.com/celestiaorg/celestia-core/pull/352)
+- Adding the DAG to the blockstore (https://github.com/celestiaorg/celestia-core/pull/356)
+- Saving and Loading using IPFS (https://github.com/celestiaorg/celestia-core/pull/374)
+- Manual Providing (https://github.com/celestiaorg/celestia-core/pull/375)
+- Refactor node provider (https://github.com/celestiaorg/celestia-core/pull/400)
+- DAS in light client workaround (https://github.com/celestiaorg/celestia-core/pull/413)
+
+#### BlockID and PartSetHeader
+
+- Decouple ParSetHeader from BlockID (https://github.com/celestiaorg/celestia-core/pull/441)
+- Stop Signing over the PartSetHeader (https://github.com/celestiaorg/celestia-core/pull/457)
+- We still don’t want to sign over the PartSetHeader, but we will not be able to use the same mechanism used in the
+  linked PR, as that way requires decoupling of the PSH from the BlockID
+- Remove PSH from some consensus messages (https://github.com/celestiaorg/celestia-core/pull/479)
+
+Note: This ADR overrides ADR 005 Decouple BlockID and the PartSetHeader. The PartSetHeader and the BlockID will mostly
+remain the same. This will make pulling changes from upstream much easier
+
+## Status
+
+Accepted
+
+## Consequences
+
+### Positive
+
+- Pulling changes from upstream is streamlined
+- Separation of functionality will help us iterate faster
+- Creates a great opportunity for reconsidering past design choices without fully starting from scratch
+- Prepare for future designs
+- Don’t have to have two p2p stacks in a single repo
+
+### Negative
+
+- Perform some computation multiple times
+- Running multiple nodes instead of a single node is less convenient for node operators (but only in the case the full
+  celestia-node wants to participate in the consensus protocol)
+
+## References
+
+Tracking Issue #491

docs/celestia-architecture/adr-008-updating-to-tendermint-v0.35.x.md

@@ -0,0 +1,53 @@
+# ADR 008: Updating to tendermint v0.35.x 
+
+## Changelog
+
+- 2022-05-03: Initial document describing changes to tendermint v0.35.x
+
+## Context
+
+Building off of ADR 007, we have further distilled the necessary changes to tendermint and continued to move added logic to other repos. Specifically, we have moved generation of the data hash, efficient construction of the data square, and a message inclusion check to celestia-app via adopting two new methods from ABCI++. This document is to serve as a guide for the remaining changes made on top of tendermint v0.35.4.
+
+### Changes to tendermint
+
+#### Misc
+
+- [update github templates](https://github.com/celestiaorg/celestia-core/pull/405)
+- [update README.md](https://github.com/celestiaorg/celestia-core/pull/737/commits/be9039d4e0f5d876ec3d8d4521be3374172d7992)
+- [updating to go 1.17](https://github.com/celestiaorg/celestia-core/pull/737/commits/6094b7338082d106f81da987dffa56eb540a675e)
+- [adding the consts package](https://github.com/celestiaorg/celestia-core/pull/737/commits/fea8528b0177230b7e75396ae05f7a9b5da23741)
+
+#### Changing the way the data hash is calculated
+
+To enable data availability sampling, Celestia uses a proprietary data square format to encode its block data. The data hash is generated from this data square by calculating namespace merkle tree root over each row and column. In the following changes, we implement encoding and decoding of block data to the data square format and tooling to generate the data hash. More details over this design can be found in our (archived but still very useful) [specs repo](https://github.com/celestiaorg/celestia-specs)
+
+- [Adding the Data Availability Header](https://github.com/celestiaorg/celestia-core/pull/737/commits/116b7af4000920030a373363487ef9a9f084e066)
+- [Adding a wrapper for namespaced merkle trees](https://github.com/celestiaorg/celestia-core/pull/737/commits/eee8f352cb6a1687a9f6b470abe28bbd4eb66413)
+- [Adding Messages and Evidence to the block data](https://github.com/celestiaorg/celestia-core/pull/737/commits/86df6529a7c0cc1112c34b6bf1b5364aa0518dec)
+- [Adding share splitting and merging for block encoding](https://github.com/celestiaorg/celestia-core/pull/737/commits/bf2d8b46c1caf1fed52e7db9bf8aa6a9847d84ab)
+- [Modifying MakeBlock to also accept Messages](https://github.com/celestiaorg/celestia-core/pull/737/commits/bb970a417356ab030c934ccd2bd39c9641af45f8)
+
+#### Adding PrepareProposal and ProcessProposal ABCI methods from ABCI++
+
+- [PrepareProposal](https://github.com/celestiaorg/celestia-core/pull/737/commits/07f9a05444db763c44ff81f564e7350ddf57e5a4)
+- [ProcessProposal](https://github.com/celestiaorg/celestia-core/pull/737/commits/2c9552db09841f2bbebc1ec34653b2441def9f13)
+
+more details on how we use these new methods in the app can be found in the [ABCI++ Adoption ADR](https://github.com/celestiaorg/celestia-app/blob/master/docs/architecture/ADR-001-ABCI%2B%2B.md).
+
+#### Wrapping Malleated Transactions
+
+Tendermint and the cosmos-sdk were not built to handle malleated transactions (txs that are submitted by the user, but modified by the block producer before being included in a block). While not a final solution, we have resorted to adding the hash of the original transaction (the one that is not modified by the block producer) to the modified one. This allows us to track the transaction in the event system and mempool.
+
+- [Index malleated Txs](https://github.com/celestiaorg/celestia-core/pull/737/commits/a54e3599a5ef6b2ba8b63f586aed8185a3f59e4d)
+
+#### Create NMT Inclusion Proofs for Transactions
+
+Since the block data that is committed over is encoded as a data square and we use namespaced merkle trees to generate the row and column roots of that square, we have to create transaction inclusion proofs also using nmts and a data square. The problem is that the block data isn't stored as a square, so in order to generate the inclusion proofs, we have to regenerate a portion of the square. We do that here.
+
+- [Create namespace merkle tree inclusion proofs for transactions included in the block](https://github.com/celestiaorg/celestia-core/pull/737/commits/01051aa5fef0693bf3bda801e39c80e5746b9c33)
+
+#### Adding the DataCommitment RPC endpoint
+
+This RPC endpoint is used by quantum gravity bridge orchestrators to create a commitment over the block data of a range of blocks.
+
+- [Adding the DataCommitment RPC endpoint](https://github.com/celestiaorg/celestia-core/pull/737/commits/134eeefb7af41afe760d4adc5b22a9d55e36bc3e)