Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12453

Spark Streaming Kinesis Example broken due to wrong AWS Java SDK version

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.2
    • Fix Version/s: 1.5.3, 1.6.1, 2.0.0
    • Component/s: DStreams
    • Labels:

      Description

      The Spark Streaming Kinesis Example (kinesis-asl) is broken due to wrong AWS Java SDK version (1.9.16) referenced with the AWS KCL version (1.3.0).

      AWS KCL 1.3.0 references AWS Java SDK version 1.9.37.
      Using 1.9.16 in combination with 1.3.0 does fail to get data out of the stream.
      I tested Spark Streaming with 1.9.37 and it works fine.
      Testing a simple KCL client outside of Spark with 1.3.0 and 1.9.16 also fails, so it is due to the specific versions used in 1.5.2 and not a Spark related implementation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brianlondon Brian London
                Reporter:
                schadix Martin Schade
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: