[NIFI-6501] Add config to limit buffer queue size in CaptureChangeMySQL

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.2, 1.14.0, 1.13.1, 1.13.2
• Fix Version/s: 1.20.0
• Component/s: Configuration
• Labels: easyfix
• Assignee: Matt Burgess
• Reporter: Purushotham Pushpavanthar
• Created: 29/Jul/19 19:23
• Updated: 17/Jan/23 22:20
• Resolved: 17/Jan/23 22:20

Description

CaptureChangeMySQL processor registers a listener with Blocking Queue as buffer with the BinLogClient. When the thread polling from the Queue is slower compared to writer, the queue grows uncontrollably and brings down the node.

Since the flow writing to listeners in [mysql-binlog-connector-java|https://github.com/shyiko/mysql-binlog-connector-java] is blocking, we should initialize the queue with initial size and queue offer timeout specified by the user based in cluster configuration.

http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html

Issue Links

• links to: GitHub Pull Request #3616
• links to: GitHub Pull Request #5153
• links to: GitHub Pull Request #6791