[FLINK-10774] connection leak when partition discovery is disabled and open throws exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.2, 1.5.5, 1.6.2
• Fix Version/s: 1.6.4, 1.7.2, 1.8.0
• Component/s: Connectors / Kafka
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Steven Zhen Wu
• Created: 04/Nov/18 18:31
• Updated: 14/Feb/19 16:15
• Resolved: 31/Jan/19 10:49

Description

Here is the scenario to reproduce the issue

• partition discovery is disabled
• open method throws an exception (e.g. when broker SSL authorization denies request)

In this scenario, run method won't be executed. As a result, partitionDiscoverer.close() won't be called. that caused the connection leak, because KafkaConsumer is initialized but not closed. That has caused outage that brought down our Kafka cluster, when a high-parallelism job got into a restart/failure loop.

Issue Links

• contains: FLINK-10721 Kafka discovery-loop exceptions may be swallowed
• links to: GitHub Pull Request #7020
• links to: GitHub Pull Request #7606