Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13154

S3AFileSystem printAmazonServiceException/printAmazonClientException appear copy & paste of AWS examples

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.7.2
    • Fix Version/s: 2.7.3
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The logging code in S3AFileSystem.printAmazonServiceException() and printAmazonClientException appear to be paste + edits of the example code in the amazon SDK, such as http://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingJava.html]

      Either we review the license to validate it, and add credits to the code if compatible, or we rework. HADOOP-13130 would be the place to do that, as it is changing exception handling anyway.

      tagging as blocker as it is license related

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
          Hide
          cnauroth Chris Nauroth added a comment -

          Before committing this, I had a successful full test run against my S3 buckets in US-west-2.

          Show
          cnauroth Chris Nauroth added a comment - Before committing this, I had a successful full test run against my S3 buckets in US-west-2.
          Hide
          cnauroth Chris Nauroth added a comment -

          +1 for the patch. I committed this to branch-2.7. Steve Loughran, thank you.

          Show
          cnauroth Chris Nauroth added a comment - +1 for the patch. I committed this to branch-2.7. Steve Loughran , thank you.
          Hide
          cnauroth Chris Nauroth added a comment -

          The patch looks good. I'll take it through a test run on branch-2.7 later today.

          Show
          cnauroth Chris Nauroth added a comment - The patch looks good. I'll take it through a test run on branch-2.7 later today.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 46s branch-2.7 passed
          +1 compile 0m 10s branch-2.7 passed with JDK v1.8.0_91
          +1 compile 0m 12s branch-2.7 passed with JDK v1.7.0_101
          +1 checkstyle 0m 21s branch-2.7 passed
          +1 mvnsite 0m 21s branch-2.7 passed
          +1 mvneclipse 0m 40s branch-2.7 passed
          +1 findbugs 0m 41s branch-2.7 passed
          +1 javadoc 0m 14s branch-2.7 passed with JDK v1.8.0_91
          +1 javadoc 0m 14s branch-2.7 passed with JDK v1.7.0_101
          +1 mvninstall 0m 14s the patch passed
          +1 compile 0m 12s the patch passed with JDK v1.8.0_91
          +1 javac 0m 12s the patch passed
          +1 compile 0m 10s the patch passed with JDK v1.7.0_101
          +1 javac 0m 10s the patch passed
          +1 checkstyle 0m 14s hadoop-tools/hadoop-aws: The patch generated 0 new + 205 unchanged - 8 fixed = 205 total (was 213)
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          -1 whitespace 0m 0s The patch has 1102 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 26s The patch 70 line(s) with tabs.
          +1 findbugs 0m 43s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91
          +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91.
          +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          16m 50s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:c420dfe
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804918/HADOOP-13154-branch-2.7-001.patch
          JIRA Issue HADOOP-13154
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6004a2fbc034 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2.7 / 0ee36fd
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/testReport/
          modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/console
          Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 46s branch-2.7 passed +1 compile 0m 10s branch-2.7 passed with JDK v1.8.0_91 +1 compile 0m 12s branch-2.7 passed with JDK v1.7.0_101 +1 checkstyle 0m 21s branch-2.7 passed +1 mvnsite 0m 21s branch-2.7 passed +1 mvneclipse 0m 40s branch-2.7 passed +1 findbugs 0m 41s branch-2.7 passed +1 javadoc 0m 14s branch-2.7 passed with JDK v1.8.0_91 +1 javadoc 0m 14s branch-2.7 passed with JDK v1.7.0_101 +1 mvninstall 0m 14s the patch passed +1 compile 0m 12s the patch passed with JDK v1.8.0_91 +1 javac 0m 12s the patch passed +1 compile 0m 10s the patch passed with JDK v1.7.0_101 +1 javac 0m 10s the patch passed +1 checkstyle 0m 14s hadoop-tools/hadoop-aws: The patch generated 0 new + 205 unchanged - 8 fixed = 205 total (was 213) +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 10s the patch passed -1 whitespace 0m 0s The patch has 1102 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 26s The patch 70 line(s) with tabs. +1 findbugs 0m 43s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_91 +1 javadoc 0m 13s the patch passed with JDK v1.7.0_101 +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.8.0_91. +1 unit 0m 11s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 16m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:c420dfe JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12804918/HADOOP-13154-branch-2.7-001.patch JIRA Issue HADOOP-13154 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6004a2fbc034 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / 0ee36fd Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/testReport/ modules C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9519/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          BTW, that commit you found with git blame isn't in branch-2.7; the code came in HADOOP-10400, the original S3A donation.

          Show
          stevel@apache.org Steve Loughran added a comment - BTW, that commit you found with git blame isn't in branch-2.7; the code came in HADOOP-10400 , the original S3A donation.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Address AWS example code in error logging by (a) removing one of the logging options, (b) switching to the logging in branch-2, while culling the covering text from amazon's code

          Show
          stevel@apache.org Steve Loughran added a comment - Address AWS example code in error logging by (a) removing one of the logging options, (b) switching to the logging in branch-2, while culling the covering text from amazon's code
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Oh, it's been along for a while, I was the one who put in the contributed code. I only realised yesterday that this is the case.

          in HADOOP-13130 I pull the methods out, but that is part of a bigger diff.

          I think for 2.7 we can/should just add a comment in the credits of the file (and other license file) saying "from Amazon"

          Show
          stevel@apache.org Steve Loughran added a comment - Oh, it's been along for a while, I was the one who put in the contributed code. I only realised yesterday that this is the case. in HADOOP-13130 I pull the methods out, but that is part of a bigger diff. I think for 2.7 we can/should just add a comment in the credits of the file (and other license file) saying "from Amazon"
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Steve Loughran, looks like they've been around for a while. git blame on both these methods point to the following commit, back to you

          commit 27c4e90efce04e1b1302f668b5eb22412e00d033
          Author: Steve Loughran <stevel@apache.org>
          Date:   Thu May 12 19:24:20 2016 +0100
          
              HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
              patch includes
              HADOOP-12844 Recover when S3A fails on IOException in read()
              HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
              HADOOP-13047 S3a Forward seek in stream length to be configurable
          
          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Steve Loughran , looks like they've been around for a while. git blame on both these methods point to the following commit, back to you commit 27c4e90efce04e1b1302f668b5eb22412e00d033 Author: Steve Loughran <stevel@apache.org> Date: Thu May 12 19:24:20 2016 +0100 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel patch includes HADOOP-12844 Recover when S3A fails on IOException in read() HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge HADOOP-13047 S3a Forward seek in stream length to be configurable

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development