[KAFKA-16862] Refactor ConsumerTaskTest to be deterministic and avoid tight loops

• Type: Task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.8.0
• Fix Version/s: 3.9.0
• Component/s: Tiered-Storage
• Labels: newbie
• Assignee: Wang Xiaoqing
• Reporter: Greg Harris
• Created: 29/May/24 18:03
• Updated: 20/Jun/24 19:35
• Resolved: 20/Jun/24 19:35

Description

The ConsumerTaskTest instantiates a MockConsumer, and uses this MockConsumer instance in the ConsumerTask, which is run in a background thread.

• This causes the background thread to tight loop on MockConsumer#poll, which has no sleep or other delay mechanism. This wastes CPU cycles and makes it impossible to use MockConsumer#schedulePollTask to meaningfully mock out the poll behavior.

• The test thread then needs to use TestUtils.waitForCondition, which repeatedly polls a result until it is satisfactory, wasting CPU cycles and introducing opportunities for timeout errors. (The test is not currently flaky in CI, so this is less of a concern).

Instead, the ConsumerTaskTest can be rewritten to not utilize a background thread, and make all calls to the MockConsumer on the same thread. This is the model that the DistributedHerderTest uses with DistributedHerder#tick, and WorkerSinkTaskTest uses with WorkerSinkTask#iteration. AbstractWorkerSourceTaskTest uses a similar model with multiple methods, the most notable being AbstractWorkerSourceTask#sendRecords.

Issue Links

• links to: GitHub Pull Request #16303