Why not store the extension in the header log digest?

[boundless-forest]

After looking through the implementation of the avail node, I noticed that you guys forked the substrate system pallet to add an extension that contains the critical field to achieve data verification, it's a critical design of the avail. I have a puzzle that why not store this extension in the header log digest which support putting custom data inside the header, seems like a better solution rather than changing the original substrate header, which will bring increased maintainness burden in the long future. Maybe I am missing something else here.

[Leouarz]

Hey @boundless-forest , thank you for this insightful question regarding our implementation. Your observations about why the extension wasn't stored in the header log digest, despite its ability to hold custom data, is quite interesting. Here's a detailed explanation of our design choices:

1. Accessibility of the Extension: The extension type is designed to enable easy retrieval of some or all parts of the data without incurring overhead. This feature is a critical factor in our design considerations, ensuring efficient accessibility across different components of our infrastructure, such as light clients.

2. Complexity in Digest Item Retrieval: The Digest is an array of logs. If we were to store the extension there, it would introduce an overhead in identifying and reconstructing the correct log, even if they are ordered (which is not the case systematically). This added complexity is something we aimed to avoid.

3. Intended Use of the Digest: The digest is primarily utilized for consensus-related purposes. We are cautious about mixing different functionalities, especially in the event of changes in the upstream. Our goal is to keep our modifications compartmentalized, even in the context of forking the system pallet, to avoid potential conflicts and complexities.

4. Reliability Concerns with DigestItem: The current code base for the DigestItem shows that the item we could use is somewhat unsupported and experimental:

   /// Some other thing. Unsupported and experimental.
   Other(Vec<u8>),

   This raises concerns about reliability, prompting us to seek a more stable solution.

5. Versioning of the Header: Incorporating the extension in the way we have also facilitates easier versioning of the header. This approach is part of our forward planning to ensure future compatibility and ease of updates.

6. Necessity of Forking System Pallet for Block Construction: Lastly, building this extension at the very end of the block construction process would necessitate forking the system pallet regardless. Therefore, we opted to rely on a more robust extension that we could have greater control over.

It's important to note that both approaches, storing the extension in the header log digest or in its current form, could technically work. The choice we made reflects a series of strategic decisions, prioritizing certain aspects over others based on our specific requirements and long-term vision for the system.

I hope this response clarifies the design choices we made in implementing the avail node. We appreciate your engagement and the depth of your inquiry, as it reflects a keen understanding of the complexities involved in blockchain technology development.

[boundless-forest]

Thank you for your comprehensive answer to my question. I have learned something from it.