Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Initial consensus node to block node communication #17769

Merged
merged 47 commits into from
Feb 21, 2025
Merged

Conversation

petreze
Copy link
Contributor

@petreze petreze commented Feb 7, 2025

Description:
This PR improves and provides an implementation of gRPC streaming to block nodes. This implementation is subject to change and only provides a happy path for streaming.

Addition of Block Node Connection Management

  • New Protocol Buffers Definition: Added a new internal protocol buffers file network/block_node_connections.proto to define the configuration for connecting to block nodes. This includes messages for individual block node configurations and overall connection configurations.

  • Configuration Files: Added a new JSON configuration file hedera-node/configuration/dev/block-nodes.json to specify block node connection details, as an example.

Java Class Modifications

  • Hedera.java: Updated the Hedera class to initialize block node connections, handle their shutdown during different platform statuses, and ensure connections are established before starting the platform.

  • HederaInjectionComponent.java: Added a new method to provide the BlockNodeConnectionManager.

  • ServicesMain.java: Modified the main method to initialize block node connections before starting the platform.

Dependency Injection and Module Updates

  • BlockStreamModule.java: Updated the BlockStreamModule to provide the BlockNodeConnectionManager and handle different block stream writer modes, including a new mode for both file and gRPC streaming.

Related issue(s):

Fixes #17730

Notes for reviewer:

Checklist

  • Documented (Code comments, README, etc.)
  • Tested (unit, integration, etc.)

derektriley and others added 14 commits January 16, 2025 19:27
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
Signed-off-by: Derek Riley <[email protected]>
# Conflicts:
#	hedera-node/configuration/dev/application.properties
#	hedera-node/hedera-app/src/jmh/java/com/hedera/node/app/blocks/BlockRetentionManagerBenchmark.java
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/Hedera.java
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/BlockRetentionManager.java
#	hedera-node/hedera-app/src/test/java/com/hedera/node/app/blocks/BlockRetentionManagerTest.java
Signed-off-by: Petar Tonev <[email protected]>
@petreze petreze added this to the v0.60 milestone Feb 7, 2025
@petreze petreze self-assigned this Feb 7, 2025
@petreze petreze requested review from a team as code owners February 7, 2025 12:06
Signed-off-by: Petar Tonev <[email protected]>
Copy link

codecov bot commented Feb 7, 2025

Codecov Report

Attention: Patch coverage is 18.67704% with 209 lines in your changes missing coverage. Please review.

Project coverage is 68.92%. Comparing base (0515075) to head (3ac9098).

Files with missing lines Patch % Lines
...cks/impl/streaming/BlockNodeConnectionManager.java 9.01% 110 Missing and 1 partial ⚠️
...app/blocks/impl/streaming/BlockNodeConnection.java 0.00% 49 Missing ⚠️
...app/blocks/impl/streaming/GrpcBlockItemWriter.java 25.00% 18 Missing and 3 partials ⚠️
...-app/src/main/java/com/hedera/node/app/Hedera.java 28.57% 19 Missing and 1 partial ⚠️
...rc/main/java/com/hedera/node/app/ServicesMain.java 0.00% 3 Missing ⚠️
...era/node/app/blocks/impl/streaming/BlockState.java 0.00% 2 Missing ⚠️
.../com/hedera/node/app/blocks/BlockStreamModule.java 75.00% 1 Missing ⚠️
...a/node/app/blocks/impl/BlockStreamManagerImpl.java 0.00% 0 Missing and 1 partial ⚠️
...com/hedera/node/config/data/BlockStreamConfig.java 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##               main   #17769      +/-   ##
============================================
- Coverage     69.01%   68.92%   -0.10%     
- Complexity    23050    23060      +10     
============================================
  Files          2655     2659       +4     
  Lines         99675    99829     +154     
  Branches      10289    10297       +8     
============================================
+ Hits          68795    68809      +14     
- Misses        26981    27121     +140     
  Partials       3899     3899              
Files with missing lines Coverage Δ
...locks/impl/streaming/BlockNodeConfigExtractor.java 100.00% <100.00%> (ø)
...app/blocks/impl/streaming/FileBlockItemWriter.java 72.22% <ø> (ø)
...edera/node/config/types/BlockStreamWriterMode.java 100.00% <100.00%> (ø)
.../com/hedera/node/app/blocks/BlockStreamModule.java 75.00% <75.00%> (+15.00%) ⬆️
...a/node/app/blocks/impl/BlockStreamManagerImpl.java 95.37% <0.00%> (ø)
...com/hedera/node/config/data/BlockStreamConfig.java 50.00% <0.00%> (-50.00%) ⬇️
...era/node/app/blocks/impl/streaming/BlockState.java 0.00% <0.00%> (ø)
...rc/main/java/com/hedera/node/app/ServicesMain.java 27.21% <0.00%> (-0.57%) ⬇️
...-app/src/main/java/com/hedera/node/app/Hedera.java 48.95% <28.57%> (-1.79%) ⬇️
...app/blocks/impl/streaming/GrpcBlockItemWriter.java 25.00% <25.00%> (ø)
... and 2 more

... and 8 files with indirect coverage changes

Impacted file tree graph

Copy link

codacy-production bot commented Feb 7, 2025

Coverage summary from Codacy

See diff coverage on Codacy

Coverage variation Diff coverage
-0.10% (target: -1.00%) 21.40%
Coverage variation details
Coverable lines Covered lines Coverage
Common ancestor commit (0515075) 99458 72533 72.93%
Head commit (3ac9098) 99612 (+154) 72547 (+14) 72.83% (-0.10%)

Coverage variation is the difference between the coverage for the head and common ancestor commits of the pull request branch: <coverage of head commit> - <coverage of common ancestor commit>

Diff coverage details
Coverable lines Covered lines Diff coverage
Pull request (#17769) 257 55 21.40%

Diff coverage is the percentage of lines that are covered by tests out of the coverable lines that the pull request added or modified: <covered lines added or modified>/<coverable lines added or modified> * 100%

See your quality gate settings    Change summary preferences

Codacy stopped sending the deprecated coverage status on June 5th, 2024. Learn more

Signed-off-by: Petar Tonev <[email protected]>
@petreze petreze requested review from a team as code owners February 14, 2025 13:15
Signed-off-by: Petar Tonev <[email protected]>
# Conflicts:
#	hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
#	hedera-node/test-clients/src/main/java/com/hedera/services/bdd/spec/utilops/pauses/HapiSpecWaitUntilNextBlock.java
Copy link
Contributor

@mhess-swl mhess-swl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I had a lot of thoughts while reviewing this, but I also recognize that 1) this is not the final production code we'll be using, and 2) I haven't been very involved in writing/coordinating this code for the past couple weeks. As long as this code is working we're probably fine to push off quite a few of my comments, and revisit them later.

Signed-off-by: Petar Tonev <[email protected]>
# Conflicts:
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/GrpcBlockItemWriter.java
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/FileBlockItemWriter.java
Signed-off-by: Petar Tonev <[email protected]>
@petreze petreze requested a review from mhess-swl February 20, 2025 19:38
mhess-swl
mhess-swl previously approved these changes Feb 21, 2025
Copy link
Contributor

@mhess-swl mhess-swl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a few more comments as a reminder, but they can be addressed later (feel free to resolve them whenever). Thanks @petreze @MiroslavGatsanoga and @derektriley, this is a lot of work! 🙌

# Conflicts:
#	hedera-node/hedera-app/src/main/java/com/hedera/node/app/blocks/impl/streaming/FileBlockItemWriter.java
#	hedera-node/hedera-config/src/main/java/com/hedera/node/config/data/BlockStreamConfig.java
Signed-off-by: Petar Tonev <[email protected]>
@petreze petreze requested a review from mhess-swl February 21, 2025 15:36
@derektriley derektriley merged commit e9306d3 into main Feb 21, 2025
44 checks passed
@derektriley derektriley deleted the wip-cn-to-bn branch February 21, 2025 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Initial Block Node Communication
10 participants