Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31413

Accessing the sequence number and partition id for records in Kinesis adapter

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Critical
    • Resolution: Invalid
    • Affects Version/s: 2.4.3
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:
      None

      Description

      We are using spark 2.4.3 in java. We would like to log the partition key and the sequence number of every event. The overloaded create stream function of the kinesis utils always throws a compilation error.

       

      Function<Record,Record> printSeq = s -> s;
      KinesisUtils.createStream(
      jssc,
      appName,
      streamName,
      endPointUrl,
      regionName,
      InitialPositionInStream.TRIM_HORIZON,
      kinesisCheckpointInterval,
      StorageLevel.MEMORY_AND_DISK_SER(),
      printSeq,
      Record.class);

      Record is of type com.amazonaws.services.kinesis.model.Record

       

      The exception is as follows:

      no suitable method found for createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)

      JAVA DOCS : https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-

      Is there a way out?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              varun_senthilnathan varun senthilnathan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: