Here is a plan for system tests. For this feature, there would a new system test suite : 'offset_mgmt_testsuite'
Test case #1:
3 brokers; 1 producer; 4 consumers
2 topics: "offsets" topic (replication : 3, partitions: 1) and "test" topic (replication : 2, partitions: 4)
Have 4 consumers consuming from the "test" topic. Fail two consumers one by one (graceful kill). This would trigger rebalance and consumers would commit and fetch offsets thus verifying these basic methods of offset manager.
After some time, shutdown the leader broker for the partition 0 of offsets topic so that the leadership moves to some follower and it will load the offsets. Kill a consumer to trigger rebalance thus verifying if the loading went correctly and offsets are served from this new leader.
Bring up 2 new consumers for "test" topic to test commit and fetch offset functionality.
Compare the messages produced and messages consumed. There should not be any duplication (or lost messages) indicating that consumers could store offsets and read from where they left off.
Test case #2:
Same as #1 but now we would do a force kill of consumers. Validation would involve verifying messages produced and unique messages consumed. Those should be same.
Test case #3:
1 broker, 1 consumers
2 topics: "offsets" topic (replication : 1, partitions: 1) and "test" topic (replication : 1, partitions: 4)
There would be pre-populated logs for the broker having some offsets information. Few entries will have null payload.
Broker comes up and starts loading logs of offsets topic. Consumer comes up and sends offset fetch request. This verifies if the consumer could fetch its offset correctly.
Let the broker run log cleaner on the logs of offsets topic. This would verify if the log cleaner is able to dedupe the offsets logs correctly and also if the deletion of offsets from the offset logs works correctly.
Dump the log of offsets topic. It must have lesser entries than before. Also, there must no entries corresponding to key with null payload.
Dump the messages consumed by consumer. This should not contain messages before the offset saved on the offsets log.