[SPARK-43414] Fix flakiness in Kafka RDD suites due to port binding configuration issue

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1
• Component/s: Structured Streaming
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 08/May/23 21:21
• Updated: 08/May/23 23:06
• Resolved: 08/May/23 23:06

Description

In SPARK-36837 we updated Kafka to 3.10, which uses a different set of configuration options for configuring the broker listener port. That PR only updated one of two KafkaTestUtils files (the SQL one), so the other one (used by Core tests) had an ineffective port binding configuration and would bind to the default 9092 port. This could lead to flakiness if multiple suites binding to that port ran in parallel.

To fix this, we just need to copy the updated port binding configuration from the other suite.