[SPARK-31413] Accessing the sequence number and partition id for records in Kinesis adapter

• Type: Question
• Status: Resolved
• Priority: Critical
• Resolution: Invalid
• Affects Version/s: 2.4.3
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: varun senthilnathan
• Created: 10/Apr/20 09:25
• Updated: 12/Dec/22 18:10
• Resolved: 13/Apr/20 04:25

Description

We are using spark 2.4.3 in java. We would like to log the partition key and the sequence number of every event. The overloaded create stream function of the kinesis utils always throws a compilation error.

 

Function<Record,Record> printSeq = s -> s;
KinesisUtils.createStream(
jssc,
appName,
streamName,
endPointUrl,
regionName,
InitialPositionInStream.TRIM_HORIZON,
kinesisCheckpointInterval,
StorageLevel.MEMORY_AND_DISK_SER(),
printSeq,
Record.class);

Record is of type com.amazonaws.services.kinesis.model.Record

 

The exception is as follows:

no suitable method found for createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)

JAVA DOCS : https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-

Is there a way out?