[SPARK-12453] Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.2
• Fix Version/s: 1.5.3, 1.6.1, 2.0.0
• Component/s: DStreams
• Labels: easyfix
• Assignee: Brian London
• Reporter: Martin Schade
• Created: 21/Dec/15 18:03
• Updated: 05/Jan/16 23:20
• Resolved: 05/Jan/16 23:19

Description

The Spark Streaming Kinesis Example (kinesis-asl) is broken due to wrong AWS Java SDK version (1.9.16) referenced with the AWS KCL version (1.3.0).

AWS KCL 1.3.0 references AWS Java SDK version 1.9.37.
Using 1.9.16 in combination with 1.3.0 does fail to get data out of the stream.
I tested Spark Streaming with 1.9.37 and it works fine.
Testing a simple KCL client outside of Spark with 1.3.0 and 1.9.16 also fails, so it is due to the specific versions used in 1.5.2 and not a Spark related implementation.

Issue Links

• duplicates: SPARK-12269 Update aws-java-sdk version
• links to: [Github] Pull Request #10416 (Schadix)
• links to: [Github] Pull Request #10492 (BrianLondon)