Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Done
-
None
-
None
Description
This is the parent JIRA to track all the work for the building a Kafka source for Structured Streaming. Here is the design doc for an initial version of the Kafka Source.
https://docs.google.com/document/d/19t2rWe51x7tq2e5AOfrsM9qb8_m7BRuv9fel9i0PqR8/edit?usp=sharing
================== Old description =========================
Structured streaming doesn't have support for kafka yet. I personally feel like time based indexing would make for a much better interface, but it's been pushed back to kafka 0.10.1
https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
Attachments
1.
|
Prerequisites for Kafka 0.8 support in Structured Streaming |
|
Closed | Unassigned |
2.
|
Prerequisites for Kafka 0.10 support in Structured Streaming |
|
Closed | Unassigned |
3.
|
Kafka 0.10 support in Structured Streaming |
|
Resolved | Shixiong Zhu |
4.
|
More granular control of starting offsets (assign) |
|
Resolved | Cody Koeninger |
5.
|
Maximum data per trigger |
|
Resolved | Cody Koeninger |
6.
|
Fetch the earliest offsets manually in KafkaSource instead of counting on KafkaConsumer |
|
Resolved | Shixiong Zhu |
7.
|
Disaster recovery of offsets from WAL |
|
Closed | Unassigned |
8.
|
Add a test to make sure the default starting offset is latest |
|
Resolved | Tathagata Das |