Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the ZeroMQ transport #27

Open
wants to merge 64 commits into
base: master
Choose a base branch
from

Conversation

SamuelSarle
Copy link
Contributor

No description provided.

@SamuelSarle SamuelSarle force-pushed the samuel/impl-zmq-transport branch from 4ebc035 to 85af8a6 Compare November 29, 2024 12:15
@SamuelSarle SamuelSarle force-pushed the samuel/impl-zmq-transport branch 2 times, most recently from 50647e8 to 0f5c529 Compare November 29, 2024 12:29
When a `Connected` input port receives a `CloseOutput` event for for the last
output port ID switch back to the `Open` state.
The conditions for the `loop { tokio::select!( ... ) }` to exit are:

1. All the references to `req_send` channel are dropped. This channel is
   used by the public transport interface to close the port. It clones
   the `Sender` and `send().awaits` on it. Once all of those have been
   processed, and the port is in a `Closed` state, there are no other
   references to the `req_send` port than the one removed in this
   commit.

2. All the references to `to_worker_send` channel are dropped. When
   receiving a `Message`, the `SubSocket` worker clones this `Sender`
   from an input worker's **Connected** state. Therefore if the input
   worker is in the **Closed** state, all references to the `Sender`
   will be dropped once the outstanding messages from the socket have
   been sent.

Thus, by dropping the reference to the `req_send` channel inside the
input port worker's thread, and after it processes the outstanding
sends, no other thread is available to get a reference to the `Sender`
channels and the loop is able to exit cleanly
@SamuelSarle
Copy link
Contributor Author

To run tests with full tracing output:

cd lib/protoflow-zeromq
RUST_LOG="trace" cargo test tests::run_transport -- --nocapture

@artob artob changed the title Implement ZeroMQ transport. Implement ZeroMQ transport Dec 4, 2024
@artob artob changed the title Implement ZeroMQ transport Implement the ZeroMQ transport Dec 4, 2024
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ CI failed

Platform Name Status Details
All Formatting Details
Linux Build library Details
Linux Build binaries Details
Linux Build tests Details
Linux Build examples Details
Linux Run tests Details

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ CI failed

Platform Name Status Details
All Formatting Details
Linux Build library Details
Linux Build binaries Details
Linux Build tests Details
Linux Build examples Details
Linux Run tests Details

@SamuelSarle SamuelSarle marked this pull request as ready for review December 4, 2024 20:19
@SamuelSarle SamuelSarle requested a review from artob as a code owner December 4, 2024 20:19
@User344
Copy link
Member

User344 commented Dec 5, 2024

Currently CI fails due to this PR's new dependency on protoc. Fixed in PR #28

@User344 User344 mentioned this pull request Dec 5, 2024
Copy link
Member

@User344 User344 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some nit-picks that I've noticed while I was fixing CI. Didn't look thoroughly at the rest of the code but at the first glance looks good to me.

lib/protoflow-zeromq/src/transport_event.proto Outdated Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is auto-generated. Maybe it makes more sense to put it in generated/-like directory and add it to .gitignore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explicitly configure prost-build in build.rs to output the generated files to src/ as by default they go to the build directory (target/.../) and are imported with include!(concat!(env!("OUT_DIR"), "/your.proto.package.rs")).

So unless we have a preference against it, I'd place this in src/ and commit the generated file.

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ CI failed

Platform Name Status Details
All Formatting Details
Linux Build library Details
Linux Build binaries Details
Linux Build tests Details
Linux Build examples Details
Linux Run tests Details

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ CI failed

Platform Name Status Details
All Formatting Details
Linux Build library Details
Linux Build binaries Details
Linux Build tests Details
Linux Build examples Details
Linux Run tests Details

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants