Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1793

Upgrade to recent API for kafka consumer / producer.

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • None
    • None
    • None

    Description

      Kafka consumer: 
      1.  Samza currently uses SimpleConsumer to consume messages from the KafkaBroker. However, the SimpleConsumer API is deprecated and is slated to be removed in future kafka-clients release. It's essential to migrate to KafkaConsumer api sooner to continue kafka connector support in samza. 

      2. Internally within linkedIn, we've been using an implementation of the new consumer/produce for a while with some success.

      Kafka Producer: 

      Since 0.10.1, Kafka’s messages can carry timestamps, indicating the time the event has occurred or the time when the message has been written to the Kafka broker. This is essential to support rewinding of checkpoints based upon time.

      Attachments

        Issue Links

          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1371 Some Samza Containers get stuck at "Starting BrokerProxy for hostname:portnum" while others seem to be fine

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1906 remove ExponentialSleepStrategy from KafkaSystemAdmin code.

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1887 Fix and re-enable TestKafkaSystemAdmin:testClearStream test

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1912 improve code style for KafkaSystemConsumer(Proxy) and KafkaAdminClient

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1944 refactor KafkaSystemAdmin not to keep Coordinator/Changelog/Intermediate streams info.

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1945 get rid of deleteMessageCalled from KafkaSystemAdmin

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1888 minor improvements for the after commit for KafkaSystemAdmin and KafkaSystemConsumer.

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1899 remove KafkaConfig.getKafkaSystemConsumerConfig

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1921 Upgrade SamzaKafkaSystemAdmin to use latest AdminClient

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1922 Create tests for KafkaSystemConsumerMetrics

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. SAMZA-1943 get rid of ExtendedSystemAdmin

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              boryas Boris Shkolnik
              spvenkat Shanthoosh Venkataraman
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: