Details
-
Task
-
Status: Closed
-
Major
-
Resolution: Implemented
-
None
-
None
-
None
Description
Changes in 0.10.0.0 version of Kafka
-------------------------------------------------
Message format in 0.10.0 has changed. Messages now include a time-stamp field and compressed messages include relative offsets.
Please refer the below link about the changes in Kafka 0.10.0.0 version:
https://kafka.apache.org/documentation/#upgrade_10_breaking.
Design
---------
I'm proposing the below design for new Kafka connectors by refactoring the existing Kafka connector(0.9.* version)
1) Split up the malhar-kafka to malhar-kafka-common, malhar-kafka_0.9, malhar-kafka_0.10. Directory structure would be as follows:
malhar-kafka
| --malhar-kafka-common |
| --malhar-kafka_0.9 |
| --malhar-kafka_0.10 If possible, will add the connectors for 0.8.x version. 2) Convert the KafkaConsumerWrapper to AbstractKafkaConsumerWrapper by adding the abstract methods over the consumer API's. 3) "malhar-kafka-common" package consists of the abstract classes and by default it builds over the 0.9.x version of Kafka. 4) "malhar-kafka-0.*" consists of concrete classes and its corresponding KafkaConsumerWrapper. |
Attachments
Issue Links
- links to
