Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-8749

Connect SinkTask.put(...) to specify ArrayList<SinkRecord> in Signature

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: KafkaConnect
    • Labels:

      Description

      Detailed in KIP-502
       
      The key class in a Sink Connector is the `SinkTask` implementation. The `SinkTask.put(...)` method delivers the records from the KafkaConsumer to the Connector-specific code through this entrypoint, and its current abstract class specifies:

      public abstract void put(Collection<SinkRecord> records);

      Usually, the implementer will use a for-each loop to iterate through the records one by one and act on them. There is only one caller, WorkerSinkTask which always passes an ArrayList.

      There are two shortcomings with this signature:

      1. The Connector implementation is not provide with any ordering guarantees of Records.
      2. The decision values giving flexibility to the caller of the method over the implementer of the method.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cvafadari Cyrus Vafadari
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: