Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3086

KafkaChannel commits consume offset to Zookeeper every time causing performance problem

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.7.0
    • Fix Version/s: None
    • Component/s: Kafka Channel
    • Labels:
      None

      Description

      Currently, the AUTO_COMMIT_ENABLED setting is hard corded to false(KafkaChannel.java line 285), and the commit is done every transaction(KafkaChannel.java line 567). This will cause large overhead to Kafka Zookeeper when consuming very large amount of data because every consumer in every node will be writing to Zookeeper.
      I actually had this problem and affected other running consumers' offsets on the Zookeeper. So I suggest to change the hard coded setting AUTO_COMMIT_ENABLED to true, and set a default AUTO_COMMIT_INTERVAL value, and remove the commitOffsets() method.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              OGCheeze JiYeon OH
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: