Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

filterx pubsub message #456

Merged
merged 4 commits into from
Jan 21, 2025
Merged

filterx pubsub message #456

merged 4 commits into from
Jan 21, 2025

Conversation

bshifter
Copy link
Member

Add pubsub_message() function to FilterX and extend Pub/Sub destination with protovar

This PR introduces new functionality for managing Google Pub/Sub messages in syslog-ng:

  1. pubsub_message() Function:
  • Adds a FilterX representation of the Google Pub/Sub message Protobuf descriptor.
  • Enables users to define and manipulate Pub/Sub message content directly within the FilterX scripting module (source-side).
  1. protovar Option in Pub/Sub Destination:
  • Extends the Pub/Sub destination module with the protovar option.
  • Allows referencing the FilterX-defined Pub/Sub Protobuf variable for setting the outgoing message payload.
    Together, these changes provide greater flexibility and control over Pub/Sub destination messages, supporting advanced customization directly from FilterX.

example:

filterx {
    # Adding data and attributes 
    $x = pubsub_message($MESSAGE, {"foo":{"bar":"baz"}});
};

log {
    destination {
        google-pubsub-grpc(
            ...
            proto_var($x)
        );
    };
};


Copy link
Contributor

github-actions bot commented Jan 15, 2025

This Pull Request introduces config grammar changes

axoflow/e814ffbb404072b7e293fbfca97989c1796b11f9 -> bshifter/fx-pubsub

--- a/destination
+++ b/destination

 google-pubsub-grpc(
+    proto-var(
+        <template-content>
+        <template-reference>
+    )
 )

@bshifter bshifter force-pushed the fx-pubsub branch 3 times, most recently from f7510c9 to d15fbc7 Compare January 16, 2025 10:27
@alltilla alltilla self-requested a review January 16, 2025 12:45
@bshifter bshifter force-pushed the fx-pubsub branch 2 times, most recently from 318dbda to 471948f Compare January 17, 2025 11:16
@bshifter bshifter force-pushed the fx-pubsub branch 2 times, most recently from 2fe1e2c to ff70e44 Compare January 20, 2025 18:10
@bshifter bshifter requested a review from alltilla January 20, 2025 18:35
@alltilla
Copy link
Member

Thanks for the updates on the PR. I found a couple more things that I unfortunately missed the last time.

The implementation follows the existing patterns and designs of the OpenTelemetry (OTel) filterx modules for consistency.

Signed-off-by: shifter <[email protected]>
The protovar option in the Pub/Sub destination allows users to directly access the Protobuf byte representation created by a FilterX function on the source side.

The Protobuf serialized variable is used to set the Pub/Sub message's content.
This enables users to fully control and manage the outgoing messages of the Pub/Sub destination from within FilterX.
By leveraging this option, users can customize the message payload dynamically, based on their specific requirements.
The data() and attributes() options cannot be used with protovar(). These options are mutually exclusive.

Signed-off-by: shifter <[email protected]>
@alltilla alltilla merged commit 7a1ef06 into axoflow:main Jan 21, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants