[SPARK-28142] KafkaContinuousStream ignores user configuration on CONSUMER_POLL_TIMEOUT

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 24/Jun/19 01:06
• Updated: 12/Dec/22 18:11
• Resolved: 24/Jun/19 13:21

Description

KafkaContinuousStream has a bug where pollTimeoutMs is always set to default value, as the value of KafkaSourceProvider.CONSUMER_POLL_TIMEOUT is kafkaConsumer.pollTimeoutMs which key-lowercased map has been provided as sourceOptions.

This is due to the missing spot, which Map should be passed as CaseInsensitiveStringMap as  SPARK-27106.

Issue Links

• links to: [Github] Pull Request #24942 (HeartSaVioR)
• links to: GitHub Pull Request #24942
• links to: GitHub Pull Request #24999