Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5718

Add native offset management for ReliableKafkaReceiver

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 2.0.0
    • Component/s: DStreams
    • Labels:
      None

      Description

      Kafka 0.8.2 supports native offsets management instead of ZK, this will get better performance, for now in ReliableKafkaReceiver, we rely on ZK to manage the offsets, this potentially will be a bottleneck if the injection rate is high (once per 200ms by default), so here in order to get better performance as well as keeping consistent with Kafka, add native offset management for ReliableKafkaReceiver.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                koeninger Cody Koeninger
                Reporter:
                jerryshao Saisai Shao
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: