Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15406

Structured streaming support for consuming from Kafka

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Done
    • None
    • 2.1.0
    • Structured Streaming
    • None

    Description

      This is the parent JIRA to track all the work for the building a Kafka source for Structured Streaming. Here is the design doc for an initial version of the Kafka Source.
      https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit?usp=sharing

      ================== Old description =========================
      Structured streaming doesn't have support for kafka yet. I personally feel like time based indexing would make for a much better interface, but it's been pushed back to kafka 0.10.1

      https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index

      Attachments

        Activity

          People

            Unassigned Unassigned
            koeninger Cody Koeninger
            Votes:
            13 Vote for this issue
            Watchers:
            45 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: