Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      The Maven dependency on aws-sdk has been changed to aws-sdk-s3 and the version bumped. Applications depending on transitive dependencies pulled in by aws-sdk and not aws-sdk-s3 might not work.

      Description

      This was originally part of HADOOP-11684, pulling out to this separate subtask as requested by Steve Loughran

      1. HADOOP-12269-001.patch
        8 kB
        Thomas Demoor
      2. HADOOP-12269-002.patch
        13 kB
        Thomas Demoor

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yes, update from 1.7.4 to 1.10.6..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yes, update from 1.7.4 to 1.10.6..
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          Patch bumps from aws-sdk-1.7.4 to aws-sdk-s3-1.10.6.

          • Only S3 library so smaller binary (possible since 1.9).
          • Multipart threshold changed from int to long as the corresponding change (bugfix) was made in the aws sdk
          • Added config setting to make overriding the signing algorithm possible. This to keep object stores that still use the previous signing algorithm functional with s3. Set this config setting to S3Signer to get v2 authentication.
          Show
          Thomas Demoor Thomas Demoor added a comment - Patch bumps from aws-sdk-1.7.4 to aws-sdk-s3-1.10.6. Only S3 library so smaller binary (possible since 1.9). Multipart threshold changed from int to long as the corresponding change (bugfix) was made in the aws sdk Added config setting to make overriding the signing algorithm possible. This to keep object stores that still use the previous signing algorithm functional with s3. Set this config setting to S3Signer to get v2 authentication.
          Hide
          busbey Sean Busbey added a comment -

          What's the target Hadoop version(s)?

          Show
          busbey Sean Busbey added a comment - What's the target Hadoop version(s)?
          Hide
          fabbri Aaron Fabbri added a comment -

          Can we use HADOOP-12267 for 2.6.x and 2.7.x and do this latest-greatest sdk for trunk? Eager to get this upstream as customers are hitting it.

          Show
          fabbri Aaron Fabbri added a comment - Can we use HADOOP-12267 for 2.6.x and 2.7.x and do this latest-greatest sdk for trunk? Eager to get this upstream as customers are hitting it.
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          As soon as possible (just like all my other patches ). For "outsiders" it's not really trivial to find out what the next version will be. I see discussions on the dev lists on 2.8 vs. 3.0 ... I kinda sense 2.8 is winning the front. Targeting 2.8 for now.

          Concerning the 2.6-7.x branches: Aaron Fabbri, I think your patch moves past 1.7.6, which has the int -> long bugfix on multipart threshold in the aws sdk. So you need to backport those changes as well or verify that that does not cause issues.

          Show
          Thomas Demoor Thomas Demoor added a comment - As soon as possible (just like all my other patches ). For "outsiders" it's not really trivial to find out what the next version will be. I see discussions on the dev lists on 2.8 vs. 3.0 ... I kinda sense 2.8 is winning the front. Targeting 2.8 for now. Concerning the 2.6-7.x branches: Aaron Fabbri , I think your patch moves past 1.7.6, which has the int -> long bugfix on multipart threshold in the aws sdk. So you need to backport those changes as well or verify that that does not cause issues.
          Hide
          fabbri Aaron Fabbri added a comment -

          I tested my HADOOP-12267 patches w/o the multipart config patch and it does behave differently, so I'm porting those changes back.

          Thomas Demoor in your patch, what is the thinking of having DEFAULT_MIN_MULTIPART_THRESHOLD == DEFAULT_MULTIPART_SIZE (100MB).

          I assume you changed it from Integer.MAX_VALUE to enable multipart uploads for smaller files. Should we make it DEFAULT_MULTIPART_SIZE * 2 instead? I'll probably go with that for HADOOP-12267 unless I hear otherwise.

          Show
          fabbri Aaron Fabbri added a comment - I tested my HADOOP-12267 patches w/o the multipart config patch and it does behave differently, so I'm porting those changes back. Thomas Demoor in your patch, what is the thinking of having DEFAULT_MIN_MULTIPART_THRESHOLD == DEFAULT_MULTIPART_SIZE (100MB). I assume you changed it from Integer.MAX_VALUE to enable multipart uploads for smaller files. Should we make it DEFAULT_MULTIPART_SIZE * 2 instead? I'll probably go with that for HADOOP-12267 unless I hear otherwise.
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          Setting defaults is difficult as the "correct value depends greatly on the bandwidth between your Hadoop cluster and the S3(-compatible) backend and the reason you use multipartupload: limit size of "unit that gets lost" on a bad connection (WAN scenario), starting upload before all data is in (s3a.fast.upload=true), ...

          I'm setting it back to 2GB (although that's certainly a lot too high for WAN). Adjusting the defaults merits its own JIRA as the other s3a defaults are also, mmh, not fantastic.

          Show
          Thomas Demoor Thomas Demoor added a comment - Setting defaults is difficult as the "correct value depends greatly on the bandwidth between your Hadoop cluster and the S3(-compatible) backend and the reason you use multipartupload: limit size of "unit that gets lost" on a bad connection (WAN scenario), starting upload before all data is in (s3a.fast.upload=true), ... I'm setting it back to 2GB (although that's certainly a lot too high for WAN). Adjusting the defaults merits its own JIRA as the other s3a defaults are also, mmh, not fantastic.
          Hide
          fabbri Aaron Fabbri added a comment -

          Talked with Lei (Eddy) Xu and we agreed it is better to get trunk onto latest SDK and then backport the same to branch-2 etc.

          Thomas Demoor, what can I do to help this get committed to trunk? Do you want me to roll another patch with DEFAULT_MIN_MULTIPART_THRESHOLD reverted to ~2GB and do some testing?

          Show
          fabbri Aaron Fabbri added a comment - Talked with Lei (Eddy) Xu and we agreed it is better to get trunk onto latest SDK and then backport the same to branch-2 etc. Thomas Demoor , what can I do to help this get committed to trunk? Do you want me to roll another patch with DEFAULT_MIN_MULTIPART_THRESHOLD reverted to ~2GB and do some testing?
          Hide
          Thomas Demoor Thomas Demoor added a comment -

          Patch 2 sets multipart threshold back to ~ 2GB and replaces calls that were deprecated in the aws-sdk (between 1.7 and 1.10) by their up-to-date counterparts

          I ran it through test-patch.sh vs AWS to check. Jenkins does not run tests, s3a needs to be tested manually vs AWS ('by / to convince' the committer), see the index.md doc for info on how to test with auth-keys.xml . Steve Loughran has been commiting s3a patches in his spare time ...

          Show
          Thomas Demoor Thomas Demoor added a comment - Patch 2 sets multipart threshold back to ~ 2GB and replaces calls that were deprecated in the aws-sdk (between 1.7 and 1.10) by their up-to-date counterparts I ran it through test-patch.sh vs AWS to check. Jenkins does not run tests, s3a needs to be tested manually vs AWS ('by / to convince' the committer), see the index.md doc for info on how to test with auth-keys.xml . Steve Loughran has been commiting s3a patches in his spare time ...
          Hide
          fabbri Aaron Fabbri added a comment -

          Thomas Demoor Steve Loughran I applied Patch 2 to trunk and did some testing.. Looks good on my end.

          Did the following via s3a:// URIs to my amazon S3 bucket:

          • hdfs put and get with md5 checking of integrity
          • Test multipart upload on/off, e.g.

          hdfs fs -Dfs.s3a.multipart.threshold=5242880 -Dfs.s3a.multipart.size=5242880 -put hadoop-client2.tgz s3a://$ACCESS:$SECRET@fabbri-dev/test-folder

          then `lsof -i 4tcp | grep java` to confirm single/multiple TCP connections to S3 storage

          • Download the multipart-uploaded tar file (hdfs fs -get) and run md5 to check round-trip integrity versus original local file.

          Hope that helps. Eager to get this upstream for the critical overflow bug I explained in HADOOP-12267

          Show
          fabbri Aaron Fabbri added a comment - Thomas Demoor Steve Loughran I applied Patch 2 to trunk and did some testing.. Looks good on my end. Did the following via s3a:// URIs to my amazon S3 bucket: hdfs put and get with md5 checking of integrity Test multipart upload on/off, e.g. hdfs fs -Dfs.s3a.multipart.threshold=5242880 -Dfs.s3a.multipart.size=5242880 -put hadoop-client2.tgz s3a://$ACCESS:$SECRET@fabbri-dev/test-folder then `lsof -i 4tcp | grep java` to confirm single/multiple TCP connections to S3 storage Download the multipart-uploaded tar file (hdfs fs -get) and run md5 to check round-trip integrity versus original local file. Hope that helps. Eager to get this upstream for the critical overflow bug I explained in HADOOP-12267
          Hide
          fabbri Aaron Fabbri added a comment -

          +1 latest patch (in case that wasn't clear)

          Show
          fabbri Aaron Fabbri added a comment - +1 latest patch (in case that wasn't clear)
          Hide
          eddyxu Lei (Eddy) Xu added a comment - - edited

          +1, Ran hdfs dfs -put to s3a.

          I will commit shortly.

          Thanks a lot for working on this, Thomas Demoor], and thanks for the reviews, Aaron Fabbri.

          Show
          eddyxu Lei (Eddy) Xu added a comment - - edited +1, Ran hdfs dfs -put to s3a. I will commit shortly. Thanks a lot for working on this, Thomas Demoor ], and thanks for the reviews, Aaron Fabbri .
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          Committed to trunk and branch-2.

          Show
          eddyxu Lei (Eddy) Xu added a comment - Committed to trunk and branch-2.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thomas Demoor thanks a lot for working on this..Lei (Eddy) Xu thanks for commiting...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thomas Demoor thanks a lot for working on this.. Lei (Eddy) Xu thanks for commiting...
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8261 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8261/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8261 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8261/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-project/pom.xml
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Yarn-trunk #1008 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1008/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Yarn-trunk #1008 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1008/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-Yarn-trunk-Java8 #278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/278/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-Yarn-trunk-Java8 #278 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/278/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-tools/hadoop-aws/pom.xml hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #267 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/267/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #267 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/267/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-project/pom.xml
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2205/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2205 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2205/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #275 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/275/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-project/pom.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #275 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/275/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-project/pom.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2224 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2224/)
          HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java
          • hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-project/pom.xml
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java
          • hadoop-tools/hadoop-aws/pom.xml
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2224 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2224/ ) HADOOP-12269 . Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) (lei: rev d5403747b57b1e294e533ce17f197e7be8f5339c) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFastOutputStream.java hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-project/pom.xml hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AOutputStream.java hadoop-tools/hadoop-aws/pom.xml hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          Hide
          stevel@apache.org Steve Loughran added a comment -

          editing subject to make it easier to work out which patch changed JARs

          Show
          stevel@apache.org Steve Loughran added a comment - editing subject to make it easier to work out which patch changed JARs
          Hide
          stevel@apache.org Steve Loughran added a comment -

          tagging this as incompatible as there's now link-time problems; we can fix the binding to make it less brittle

          Show
          stevel@apache.org Steve Loughran added a comment - tagging this as incompatible as there's now link-time problems; we can fix the binding to make it less brittle
          Hide
          stevel@apache.org Steve Loughran added a comment -

          we can't fix the binding, and with the explosion of dependency JARs, anything which depends on the full SDK is going to be in trouble. Leaving as incompatible

          Show
          stevel@apache.org Steve Loughran added a comment - we can't fix the binding, and with the explosion of dependency JARs, anything which depends on the full SDK is going to be in trouble. Leaving as incompatible

            People

            • Assignee:
              Thomas Demoor Thomas Demoor
              Reporter:
              Thomas Demoor Thomas Demoor
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development