KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #1873

PratRanj07 · 2024-12-17T20:31:36Z

This PR intends to add support for DescribeConsumerGroup for the new consumer protocol.
The tests will fail till the changes in the librdkafka is merged to master.
Librdkafka PR: [https://github.com/https://github.com/https://github.com/confluentinc/librdkafka/pull/4941](KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups)

confluent-cla-assistant · 2024-12-17T20:31:47Z

🎉 All Contributor License Agreements have been signed. Ready to merge.
_{Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.}

pranavrth

Comment on Code part.

pranavrth · 2025-04-16T19:11:03Z

tests/integration/admin/test_describe_consumer_groups_compatability.py

+            'group.protocol': Protocol,
+            'enable.auto.commit': False,
+            'auto.offset.reset': 'earliest',
+            'debug': 'all'}


Remove debug logging.

pranavrth · 2025-04-16T19:11:18Z

tests/integration/admin/test_describe_consumer_groups_compatability.py

+topic_prefix = "test-topic"
+
+
+def create_consumers(kafka_cluster, topic, group_id, client_id, Protocol):


Suggested change

def create_consumers(kafka_cluster, topic, group_id, client_id, Protocol):

def create_consumers(kafka_cluster, topic, group_id, client_id, group_protocol):

pranavrth · 2025-04-17T06:13:52Z

examples/adminapi.py

+                if member.target_assignment:
+                    print("    Target Assignments:")
+                    for toppar in member.target_assignment.topic_partitions:
+                        print("      {} [{}]".format(toppar.topic, toppar.partition))


Suggested change

if member.target_assignment:

print(" Target Assignments:")

for toppar in member.target_assignment.topic_partitions:

print(" {} [{}]".format(toppar.topic, toppar.partition))

if member.target_assignment:

print(" Target Assignments:")

for toppar in member.target_assignment.topic_partitions:

print(f" {toppar.topic} [{toppar.partition)}]"

pranavrth · 2025-04-17T06:14:32Z

src/confluent_kafka/admin/_group.py

    group_instance_id : str
        The instance id of the group member.
    """

-    def __init__(self, member_id, client_id, host, assignment, group_instance_id=None):
+    def __init__(self, member_id, client_id, host, assignment, target_assignment, group_instance_id=None):


Don't change the API contract.

Suggested change

def __init__(self, member_id, client_id, host, assignment, target_assignment, group_instance_id=None):

def __init__(self, member_id, client_id, host, assignment, group_instance_id=None, target_assignment=None):

pranavrth · 2025-04-17T06:24:17Z

src/confluent_kafka/admin/_group.py

    coordinator: Node
        Consumer group coordinator.
    authorized_operations: list(AclOperation)
        AclOperations allowed for the consumer group.
    """

    def __init__(self, group_id, is_simple_consumer_group, members, partition_assignor, state,
-                 coordinator, authorized_operations=None):
+                 type, coordinator, authorized_operations=None):


Don't change API contract.

Suggested change

type, coordinator, authorized_operations=None):

coordinator, authorized_operations=None, type=ConsumerGroupType.UNKNOWN):

pranavrth · 2025-04-17T06:26:23Z

src/confluent_kafka/src/Admin.c

@@ -3892,6 +3893,15 @@ static PyObject *Admin_c_MemberDescription_to_py(const rd_kafka_MemberDescriptio

        PyDict_SetItemString(kwargs, "assignment", assignment);

+        c_target_assignment = rd_kafka_MemberDescription_target_assignment(c_member);
+        if(c_target_assignment) {
+                PyObject *target_assignment = Admin_c_MemberAssignment_to_py(c_target_assignment);


declare at top with assignment

Decrease refcount.

pranavrth · 2025-04-17T06:28:09Z

src/confluent_kafka/src/Admin.c

+        c_target_assignment = rd_kafka_MemberDescription_target_assignment(c_member);
+        if(c_target_assignment) {
+                PyObject *target_assignment = Admin_c_MemberAssignment_to_py(c_target_assignment);
+                if (!target_assignment) {


Check NULL case is working or not. target_assignment should be None in that case.

Yes it is working

emasab · 2025-04-17T10:55:30Z

tests/integration/admin/test_describe_consumer_groups_compatability.py

+        consumer.close()
+
+
+def test_describe_consumer_groups_compatability(kafka_cluster):


Suggested change

def test_describe_consumer_groups_compatability(kafka_cluster):

def test_describe_consumer_groups_compatibility(kafka_cluster):

We should change the test test_describe_operations.py instead of adding this one. There's a test that is disabled with KIP-848 there.
Here it seems the verify_describe_consumer_groups test isn't run as it doesn't start with test_ and isn't called by test_describe_consumer_groups_compatibility.

Better to move this tests to a different PR and merge them for GA.

Removed it. Will raise a seperate PR for that

emasab · 2025-04-17T12:31:39Z

CHANGELOG.md

+
+- [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#1856)
+- [KIP-848] `describe_consumer_groups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).
+


Suggested change

confluent-kafka-python v2.10.0 is based on librdkafka v2.10.0, see the

[librdkafka release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.10.0)

for a complete list of changes, enhancements, fixes and upgrade considerations.

emasab · 2025-04-17T12:32:35Z

CHANGELOG.md

+v2.10.0 is a feature release with the following fixes and enhancements:
+
+- [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#1856)
+- [KIP-848] `describe_consumer_groups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).


Suggested change

- [KIP-848] `describe_consumer_groups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

- [KIP-848] `describe_consumer_groups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#1873).

sonarqube-confluent · 2025-04-17T13:19:52Z

Analysis Details

0 Issues

0 Bugs
0 Vulnerabilities
0 Code Smells

Coverage and Duplications

12.50% Coverage (62.00% Estimated after merge)
No duplication information (1.00% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube

emasab

It's ok for me, reviewing the tests separately. Did some manual tests for the moment.

pranavrth

LGTM!.

PratRanj07 requested review from a team as code owners December 17, 2024 20:31

airlock-confluentinc bot force-pushed the dev_add_support_describegroup branch 2 times, most recently from bb27e73 to 91c560a Compare April 16, 2025 18:15

PratRanj07 changed the base branch from master to dev_kip848-preview-push April 16, 2025 18:16

This comment has been minimized.

Sign in to view

airlock-confluentinc bot force-pushed the dev_add_support_describegroup branch from 98bc0bc to a1e887a Compare April 17, 2025 05:37

This comment has been minimized.

Sign in to view

pranavrth requested changes Apr 17, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

airlock-confluentinc bot force-pushed the dev_add_support_describegroup branch from 0fd0377 to 8437ffd Compare April 17, 2025 08:56

This comment has been minimized.

Sign in to view

Base automatically changed from dev_kip848-preview-push to master April 17, 2025 10:03

airlock-confluentinc bot force-pushed the dev_add_support_describegroup branch from 8437ffd to 3cb4080 Compare April 17, 2025 10:14

This comment has been minimized.

Sign in to view

emasab reviewed Apr 17, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

PratRanj07 added 7 commits April 17, 2025 17:50

Added support for describe consumer group for new protocol

7fe5b90

style fix

d45971f

removed session timeout

137dd1d

style fix

0e696d3

Simplifying test code

e52fcc5

target assignment declaration

94a9df0

Protocol->group_protocol

55d7f08

PratRanj07 added 3 commits April 17, 2025 17:50

name change

80a1e3a

Changelog added

1d2716e

removed test

dd43a7d

airlock-confluentinc bot force-pushed the dev_add_support_describegroup branch from d5b6ed0 to dd43a7d Compare April 17, 2025 12:20

This comment has been minimized.

Sign in to view

emasab requested changes Apr 17, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

Changelog change

5808fc1

This comment has been minimized.

Sign in to view

v2.10.0rc3 release changes (#1969)

933e813

emasab approved these changes Apr 17, 2025

View reviewed changes

pranavrth approved these changes Apr 17, 2025

View reviewed changes

PratRanj07 merged commit 5a1f136 into master Apr 17, 2025
3 checks passed

PratRanj07 deleted the dev_add_support_describegroup branch April 17, 2025 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #1873

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #1873

PratRanj07 commented Dec 17, 2024 •

edited

Loading

confluent-cla-assistant bot commented Dec 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pranavrth left a comment

pranavrth Apr 16, 2025

pranavrth Apr 16, 2025

pranavrth Apr 17, 2025

pranavrth Apr 17, 2025

pranavrth Apr 17, 2025

pranavrth Apr 17, 2025

pranavrth Apr 17, 2025

PratRanj07 Apr 17, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

emasab Apr 17, 2025

emasab Apr 17, 2025 •

edited

Loading

PratRanj07 Apr 17, 2025

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

emasab Apr 17, 2025

emasab Apr 17, 2025

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented Apr 17, 2025

emasab left a comment

pranavrth left a comment

		topic_prefix = "test-topic"


		def create_consumers(kafka_cluster, topic, group_id, client_id, Protocol):

	def __init__(self, member_id, client_id, host, assignment, target_assignment, group_instance_id=None):
	def __init__(self, member_id, client_id, host, assignment, group_instance_id=None, target_assignment=None):

	type, coordinator, authorized_operations=None):
	coordinator, authorized_operations=None, type=ConsumerGroupType.UNKNOWN):

		consumer.close()


		def test_describe_consumer_groups_compatability(kafka_cluster):


		- [KIP-848] Group Config is now supported in AlterConfigs, IncrementalAlterConfigs and DescribeConfigs. (#1856)
		- [KIP-848] `describe_consumer_groups()` now supports KIP-848 introduced `consumer` groups. Two new fields for consumer group type and target assignment have also been added. Type defines whether this group is a `classic` or `consumer` group. Target assignment is only valid for the `consumer` protocol and its defaults to NULL. (#4922).

+confluent-kafka-python v2.10.0 is based on librdkafka v2.10.0, see the
+[librdkafka release notes](https://github.com/confluentinc/librdkafka/releases/tag/v2.10.0)
+for a complete list of changes, enhancements, fixes and upgrade considerations.

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #1873

KIP 848: Added support for DescribeConsumerGroup for consumer protocol groups #1873

Conversation

PratRanj07 commented Dec 17, 2024 • edited Loading

confluent-cla-assistant bot commented Dec 17, 2024

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

pranavrth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

emasab Apr 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented Apr 17, 2025

Analysis Details

0 Issues

Coverage and Duplications

emasab left a comment

Choose a reason for hiding this comment

pranavrth left a comment

Choose a reason for hiding this comment

PratRanj07 commented Dec 17, 2024 •

edited

Loading

emasab Apr 17, 2025 •

edited

Loading