[SPARK-12269] Update aws-java-sdk version

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: DStreams
• Assignee: Brian London
• Reporter: Brian London
• Created: 10/Dec/15 21:00
• Updated: 13/Jan/16 19:12
• Resolved: 11/Jan/16 09:32

Description

The current Spark Streaming kinesis connector references a quite old version 1.9.40 of the AWS Java SDK (1.10.40 is current). Numerous AWS features including Kinesis Firehose are unavailable in 1.9. Those two versions of the AWS SDK in turn require conflicting versions of Jackson (2.4.4 and 2.5.3 respectively) such that one cannot include the current AWS SDK in a project that also uses the Spark Streaming Kinesis ASL.

Bumping the version of Jackson and the AWS library solves this problem and will allow Firehose integrations.

Issue Links

• is duplicated by: SPARK-12453 Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version
• links to: [Github] Pull Request #10256 (BrianLondon)