[SPARK-28367] Kafka connector infinite wait because metadata never updated

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1.3, 2.2.3, 2.3.3, 2.4.3, 3.0.0, 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Structured Streaming
• Assignee: Gabor Somogyi
• Reporter: Gabor Somogyi
• Created: 12/Jul/19 12:02
• Updated: 09/Feb/21 01:51
• Resolved: 11/Dec/20 10:18

Description

Spark uses an old and deprecated API named poll(long) which never returns and stays in live lock if metadata is not updated (for instance when broker disappears at consumer creation).

Issue Links

• Is contained by: SPARK-30169 Eliminate warnings in Kafka connector
• is duplicated by: SPARK-30311 Use KafkaConsumer.poll(Duration) instead of deprecated one
• links to: GitHub Pull Request #25135

Sub-Tasks

• Eliminate deprecated poll(long) API calls to avoid infinite wait in tests: Resolved
• Eliminate deprecated poll(long) API calls to avoid infinite wait in driver: Resolved
• Remove UninterruptibleThread usage from KafkaOffsetReader: Resolved
• Use new poll API in Kafka connector executor side to avoid infinite wait: Resolved