TopologyTestDriver has the feature that it processes each input synchronously, resolving one of the most significant challenges with verifying the correctness of streaming applications.
When processing an input, it feeds that record to the source node, which then synchronously (it's always synchronous within a task) gets passed through the subtopology via Context#forward calls. Ultimately, outputs from that input are forwarded into the RecordCollector, which converts it to Producer.send calls. In TopologyTestDriver, this Producer is a special one that actually just captures the records.
Some output topics from one subtopology are inputs to another subtopology. For example, repartition topics. Immediately after the synchronous subtopology process() invocation, TopologyTestDriver iterates over the collected outputs from the special Producer. If they are purely output records, it just enqueues them for later retrieval by testing code. If they are records for internal topics, though, TopologyTestDriver immediately processes them as inputs for the relevant subtopology.
The problem, and this is very subtle, is that TopologyTestDriver does this recursively, which with some (apparently rare) programs can cause the output to be observed in an invalid order.
One such program is the one I wrote to test the fix for
KAFKA-9487 . It involves a foreign-key join whose result is joined back to one of its inputs.
This is what I did in https://github.com/apache/kafka/pull/8015