Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1793

Upgrade to recent API for kafka consumer / producer.

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Kafka consumer: 
      1.  Samza currently uses SimpleConsumer to consume messages from the KafkaBroker. However, the SimpleConsumer API is deprecated and is slated to be removed in future kafka-clients release. It's essential to migrate to KafkaConsumer api sooner to continue kafka connector support in samza. 

      2. Internally within linkedIn, we've been using an implementation of the new consumer/produce for a while with some success.

      Kafka Producer: 

      Since 0.10.1, Kafka’s messages can carry timestamps, indicating the time the event has occurred or the time when the message has been written to the Kafka broker. This is essential to support rewinding of checkpoints based upon time.

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                boryas Boris Shkolnik
                Reporter:
                spvenkat Shanthoosh Venkataraman
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: