Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11413

Java 8 build has problem with joda-time and s3 request, should bump joda-time version

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: Build
    • Labels:
      None

      Description

      Joda-time has problems with formatting time zones starting with Java 1.8u60, and this will cause s3 request to fail. It is said to have been fixed at joda-time 2.8.1.
      Spark is still using joda-time 2.5 by fault, if java8 is used to build spark, should set -Djoda.version=2.8.1 or above.
      I was hit by this problem, and -Djoda.version=2.9 worked.
      I don't see any reason not to bump up joda-time version in pom.xml
      Should I create a pull request for this? It is trivial.

      https://github.com/aws/aws-sdk-java/issues/484
      https://github.com/aws/aws-sdk-java/issues/444
      http://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header

        Issue Links

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note that joda-time is only explicitly pulled in with Hive builds; any version of spark built without -Phive will omit it. This gives whoever compiled their private version of Spark the opportunity and duty to put the version of Joda time compatible with the AWS SDK/JVM on their own classpath.

          Show
          stevel@apache.org Steve Loughran added a comment - Note that joda-time is only explicitly pulled in with Hive builds; any version of spark built without -Phive will omit it. This gives whoever compiled their private version of Spark the opportunity and duty to put the version of Joda time compatible with the AWS SDK/JVM on their own classpath.
          Hide
          srowen Sean Owen added a comment -

          Issue resolved by pull request 9379
          https://github.com/apache/spark/pull/9379

          Show
          srowen Sean Owen added a comment - Issue resolved by pull request 9379 https://github.com/apache/spark/pull/9379
          Hide
          yongjiaw Yongjia Wang added a comment -

          There is no more transitive dependencies from joda-time. So as long as it passes the tests, it should be good?

          Show
          yongjiaw Yongjia Wang added a comment - There is no more transitive dependencies from joda-time. So as long as it passes the tests, it should be good?
          Hide
          apachespark Apache Spark added a comment -

          User 'yongjiaw' has created a pull request for this issue:
          https://github.com/apache/spark/pull/9379

          Show
          apachespark Apache Spark added a comment - User 'yongjiaw' has created a pull request for this issue: https://github.com/apache/spark/pull/9379
          Hide
          srowen Sean Owen added a comment -

          I imagine it will be as well to update to 2.9, but yes that would be good to investigate while you're having a look. I don't expect any problems, it's just good practice.

          Show
          srowen Sean Owen added a comment - I imagine it will be as well to update to 2.9, but yes that would be good to investigate while you're having a look. I don't expect any problems, it's just good practice.
          Hide
          yongjiaw Yongjia Wang added a comment -

          I can follow up with a PR first. The latest joda-time release is 2.9. Do you suggest just using 2.8.1 as the fix with minimum changes?

          Show
          yongjiaw Yongjia Wang added a comment - I can follow up with a PR first. The latest joda-time release is 2.9. Do you suggest just using 2.8.1 as the fix with minimum changes?
          Hide
          srowen Sean Owen added a comment -

          To assume? no, that's why I'm suggesting you try it by submitting a PR to run the tests, but also investigate the list of changes from 2.5 -> 2.8.1 and to diff the dependency trees.

          Show
          srowen Sean Owen added a comment - To assume? no, that's why I'm suggesting you try it by submitting a PR to run the tests, but also investigate the list of changes from 2.5 -> 2.8.1 and to diff the dependency trees.
          Hide
          yongjiaw Yongjia Wang added a comment -

          I see.
          I don't know, is it safe to assume joda-time is backward compatible as far as Spark is concerned?

          Show
          yongjiaw Yongjia Wang added a comment - I see. I don't know, is it safe to assume joda-time is backward compatible as far as Spark is concerned?
          Hide
          srowen Sean Owen added a comment -

          I understand all that, but my question is about the side-effects of the update. Does it cause any other potential incompatibility, or change transitive dependencies? That's always the question when someone wants to update a dependency.

          Show
          srowen Sean Owen added a comment - I understand all that, but my question is about the side-effects of the update. Does it cause any other potential incompatibility, or change transitive dependencies? That's always the question when someone wants to update a dependency.
          Hide
          yongjiaw Yongjia Wang added a comment - - edited

          My last statement was wrong.
          It's a problem when using jre 8u60 or above, and when joda-time 2.8.0 or below was loaded at run time.
          The following error was reproduced when I use the official spark release and run with JRE 1.8.0_65 (Oracle Corporation)
          Note that the aws-java-sdk-s3 library already depends on joda-time-2.8.1. It's due to Spark assembly's inclusion of joda-time-2.5 which was loaded by JVM that caused this problem.

          com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E0D714AF923221DD), S3 Extended Request ID: 8OJlBMYKVzwrEXHANpBTsiLRmaFn2lHQ4sAvkU/HF66wWGVj8VR2/Jh4Wl8QuFUt
          at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182)
          at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770)
          at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
          at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
          at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
          at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738)
          at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653)

          Show
          yongjiaw Yongjia Wang added a comment - - edited My last statement was wrong. It's a problem when using jre 8u60 or above, and when joda-time 2.8.0 or below was loaded at run time. The following error was reproduced when I use the official spark release and run with JRE 1.8.0_65 (Oracle Corporation) Note that the aws-java-sdk-s3 library already depends on joda-time-2.8.1. It's due to Spark assembly's inclusion of joda-time-2.5 which was loaded by JVM that caused this problem. com.amazonaws.services.s3.model.AmazonS3Exception: AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: E0D714AF923221DD), S3 Extended Request ID: 8OJlBMYKVzwrEXHANpBTsiLRmaFn2lHQ4sAvkU/HF66wWGVj8VR2/Jh4Wl8QuFUt at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1182) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:770) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785) at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3738) at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:653)
          Hide
          yongjiaw Yongjia Wang added a comment -

          yea, the fix is to update the joda.version number from 2.5 to 2.8.1 or to the newest 2.9 in Spark's pom.xml.
          This PR would just be a 1 line change in pom.xml
          It's only an issue when compiling with java 1.8u60, due to the changes in time format, and s3 request in some way relies on the time format in its header.
          Those links in my comment should explain better. It's no new dependency, but can be considered a java8 s3 combo bug.
          So this only affects the combination of java 8u60 + using s3a, which should become quite common and hits more people.
          If I understand correctly, building with java7 or java8 before u60 should be fine, even running with jre after java 8u60.

          Show
          yongjiaw Yongjia Wang added a comment - yea, the fix is to update the joda.version number from 2.5 to 2.8.1 or to the newest 2.9 in Spark's pom.xml. This PR would just be a 1 line change in pom.xml It's only an issue when compiling with java 1.8u60, due to the changes in time format, and s3 request in some way relies on the time format in its header. Those links in my comment should explain better. It's no new dependency, but can be considered a java8 s3 combo bug. So this only affects the combination of java 8u60 + using s3a, which should become quite common and hits more people. If I understand correctly, building with java7 or java8 before u60 should be fine, even running with jre after java 8u60.
          Hide
          srowen Sean Owen added a comment -

          How about updating joda time to 2.8.1 everywhere? what are the compatibility issues, new dependencies, etc?
          It doesn't seem to make sense to only do it for java 8.

          Show
          srowen Sean Owen added a comment - How about updating joda time to 2.8.1 everywhere? what are the compatibility issues, new dependencies, etc? It doesn't seem to make sense to only do it for java 8.

            People

            • Assignee:
              yongjiaw Yongjia Wang
              Reporter:
              yongjiaw Yongjia Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development