Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6501

Add config to limit buffer queue size in CaptureChangeMySQL

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: In Progress
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.9.2
    • Fix Version/s: None
    • Component/s: Configuration
    • Labels:

      Description

      CaptureChangeMySQL processor registers a listener with Blocking Queue as buffer with the BinLogClient. When the thread polling from the Queue is slower compared to writer, the queue grows uncontrollably and brings down the node.

      Since the flow writing to listeners in [mysql-binlog-connector-java|https://github.com/shyiko/mysql-binlog-connector-java] is blocking, we should initialize the queue with initial size and queue offer timeout specified by the user based in cluster configuration.

      http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pushpavanthar Purushotham Pushpavanthar
                Reporter:
                pushpavanthar Purushotham Pushpavanthar
              • Votes:
                2 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h