The Spark Streaming Kinesis Example (kinesis-asl) is broken due to wrong AWS Java SDK version (1.9.16) referenced with the AWS KCL version (1.3.0).
AWS KCL 1.3.0 references AWS Java SDK version 1.9.37.
Using 1.9.16 in combination with 1.3.0 does fail to get data out of the stream.
I tested Spark Streaming with 1.9.37 and it works fine.
Testing a simple KCL client outside of Spark with 1.3.0 and 1.9.16 also fails, so it is due to the specific versions used in 1.5.2 and not a Spark related implementation.
- duplicates
-
SPARK-12269 Update aws-java-sdk version
-
- Resolved
-
- links to