rpk group

Describe, list, and delete consumer groups and manage their offsets.

Consumer groups allow you to horizontally scale consuming from topics. A non-group consumer consumes all records from all partitions you assign it. In contrast, consumer groups allow many consumers to coordinate and divide work. If you have two members in a group consuming topics A and B, each with three partitions, then both members consume three partitions. If you add another member to the group, then each of the three members will consume two partitions. This allows you to horizontally scale consuming of topics.

The unit of scaling is a single partition. If you add more consumers to a group than there are total partitions to consume, then some consumers will be idle. More commonly, you have many more partitions than consumer group members and each member consumes a chunk of available partitions. One scenario where you may want more members than partitions is if you want active standby’s to take over load immediately if any consuming member dies.

How group members divide work is entirely client driven (the "partition assignment strategy" or balancer depending on the client). Brokers know nothing about how consumers are assigning partitions. A broker’s role in group consuming is to choose which member is the leader of a group, forward that member’s assignment to every other member, and ensure all members are alive through heartbeats.

Consumers periodically commit their progress when consuming partitions. Through these commits, you can monitor just how far behind a consumer is from the latest messages in a partition. This is called lag. Large lag implies that the client is having problems, which could be from the server being too slow, or the client being oversubscribed in the number of partitions it is consuming, or the server being in a bad state that requires restarting or removing from the server pool, and so on.

You can manually manage offsets for a group, which allows you to rewind or forward commits. If you notice that a recent deploy of your consumers had a bug, you may want to stop all members, rewind the commits to before the latest deploy, and restart the members with a patch.

This command allows you to list all groups, describe a group (to view the members and their lag), and manage offsets.