Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3
    • Labels:
      None

      Description

      It should be possible to provide hints about send/receive buffers to AmazonS3Client via ClientConfiguration. It would be good to expose these parameters in S3AFileSystem for perf.

      1. HADOOP-13212-branch-2-004.patch
        5 kB
        Steve Loughran
      2. HADOOP-13212-branch-2-003.patch
        1 kB
        Steve Loughran
      3. HADOOP-13212-branch-2-002.patch
        5 kB
        Rajesh Balamohan
      4. HADOOP-13212-branch-2-001.patch
        5 kB
        Rajesh Balamohan

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Hadoop-trunk-Commit #10124 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10124/)
          HADOOP-13212 Provide an option to set the socket buffers in (stevel: rev 37362c2f922b8d038002e61132b110ae4dd6d5ba)

          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java
          • hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md
          • hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3AInputStreamPerformance.java
          • hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
          • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Hadoop-trunk-Commit #10124 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10124/ ) HADOOP-13212 Provide an option to set the socket buffers in (stevel: rev 37362c2f922b8d038002e61132b110ae4dd6d5ba) hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/Constants.java hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/scale/TestS3AInputStreamPerformance.java hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          Hide
          stevel@apache.org Steve Loughran added a comment -

          +1, committed. Thanks!

          Show
          stevel@apache.org Steve Loughran added a comment - +1, committed. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 25s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 3m 5s Maven dependency ordering for branch
          +1 mvninstall 6m 37s branch-2 passed
          +1 compile 5m 30s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 31s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 22s branch-2 passed
          +1 mvnsite 1m 18s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 2m 16s branch-2 passed
          +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 1s the patch passed
          +1 compile 5m 42s the patch passed with JDK v1.8.0_91
          +1 javac 5m 42s the patch passed
          +1 compile 6m 35s the patch passed with JDK v1.7.0_101
          +1 javac 6m 35s the patch passed
          +1 checkstyle 1m 26s the patch passed
          +1 mvnsite 1m 24s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 48s the patch passed
          +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101
          +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 30s The patch does not generate ASF License warnings.
          93m 39s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818858/HADOOP-13212-branch-2-004.patch
          JIRA Issue HADOOP-13212
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux fb2b4a4704bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / ad8180f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 25s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 3m 5s Maven dependency ordering for branch +1 mvninstall 6m 37s branch-2 passed +1 compile 5m 30s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 31s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 22s branch-2 passed +1 mvnsite 1m 18s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 2m 16s branch-2 passed +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 16s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 5m 42s the patch passed with JDK v1.8.0_91 +1 javac 5m 42s the patch passed +1 compile 6m 35s the patch passed with JDK v1.7.0_101 +1 javac 6m 35s the patch passed +1 checkstyle 1m 26s the patch passed +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 48s the patch passed +1 javadoc 1m 7s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101 +1 unit 7m 39s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 93m 39s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818858/HADOOP-13212-branch-2-004.patch JIRA Issue HADOOP-13212 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux fb2b4a4704bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / ad8180f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10026/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 29s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 32s Maven dependency ordering for branch
          +1 mvninstall 6m 28s branch-2 passed
          +1 compile 5m 26s branch-2 passed with JDK v1.8.0_91
          +1 compile 6m 22s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 22s branch-2 passed
          +1 mvnsite 1m 20s branch-2 passed
          +1 mvneclipse 1m 42s branch-2 passed
          +1 findbugs 2m 14s branch-2 passed
          +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 1s the patch passed
          +1 compile 5m 29s the patch passed with JDK v1.8.0_91
          +1 javac 5m 29s the patch passed
          +1 compile 6m 40s the patch passed with JDK v1.7.0_101
          +1 javac 6m 40s the patch passed
          +1 checkstyle 1m 25s the patch passed
          +1 mvnsite 1m 22s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 50s the patch passed
          +1 javadoc 1m 10s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101
          +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          92m 10s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818858/HADOOP-13212-branch-2-004.patch
          JIRA Issue HADOOP-13212
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 231f90fef63d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / ad8180f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 29s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 32s Maven dependency ordering for branch +1 mvninstall 6m 28s branch-2 passed +1 compile 5m 26s branch-2 passed with JDK v1.8.0_91 +1 compile 6m 22s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 22s branch-2 passed +1 mvnsite 1m 20s branch-2 passed +1 mvneclipse 1m 42s branch-2 passed +1 findbugs 2m 14s branch-2 passed +1 javadoc 1m 3s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 15s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 1s the patch passed +1 compile 5m 29s the patch passed with JDK v1.8.0_91 +1 javac 5m 29s the patch passed +1 compile 6m 40s the patch passed with JDK v1.7.0_101 +1 javac 6m 40s the patch passed +1 checkstyle 1m 25s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 34s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 50s the patch passed +1 javadoc 1m 10s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101 +1 unit 7m 29s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 92m 10s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818858/HADOOP-13212-branch-2-004.patch JIRA Issue HADOOP-13212 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 231f90fef63d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / ad8180f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10025/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Patch 004: the full diff

          Show
          stevel@apache.org Steve Loughran added a comment - Patch 004: the full diff
          Hide
          stevel@apache.org Steve Loughran added a comment -

          patch 003 is only my diff, not the full patch

          Show
          stevel@apache.org Steve Loughran added a comment - patch 003 is only my diff, not the full patch
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Tentative +1, pending Jenkins review (local tests pass)

          this is the patch I'm going to apply.

          It's patch 002 with the use of the new options performed in the test case by referring to their definition in Constants

          Show
          stevel@apache.org Steve Loughran added a comment - Tentative +1, pending Jenkins review (local tests pass) this is the patch I'm going to apply. It's patch 002 with the use of the new options performed in the test case by referring to their definition in Constants
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 27s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 1m 43s Maven dependency ordering for branch
          +1 mvninstall 7m 41s branch-2 passed
          +1 compile 7m 20s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 33s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 33s branch-2 passed
          +1 mvnsite 1m 28s branch-2 passed
          +1 mvneclipse 0m 35s branch-2 passed
          +1 findbugs 2m 39s branch-2 passed
          +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 2s the patch passed
          +1 compile 6m 48s the patch passed with JDK v1.8.0_91
          +1 javac 6m 48s the patch passed
          +1 compile 7m 35s the patch passed with JDK v1.7.0_101
          +1 javac 7m 35s the patch passed
          +1 checkstyle 1m 29s the patch passed
          +1 mvnsite 1m 32s the patch passed
          +1 mvneclipse 0m 37s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 xml 0m 0s The patch has no ill-formed XML file.
          +1 findbugs 2m 51s the patch passed
          +1 javadoc 1m 18s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 24s the patch passed with JDK v1.7.0_101
          +1 unit 9m 21s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          103m 27s



          Reason Tests
          JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818042/HADOOP-13212-branch-2-002.patch
          JIRA Issue HADOOP-13212
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 13cc67b1d984 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / c9f5348
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/artifact/patchprocess/whitespace-eol.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 27s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 43s Maven dependency ordering for branch +1 mvninstall 7m 41s branch-2 passed +1 compile 7m 20s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 33s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 33s branch-2 passed +1 mvnsite 1m 28s branch-2 passed +1 mvneclipse 0m 35s branch-2 passed +1 findbugs 2m 39s branch-2 passed +1 javadoc 1m 12s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 2s the patch passed +1 compile 6m 48s the patch passed with JDK v1.8.0_91 +1 javac 6m 48s the patch passed +1 compile 7m 35s the patch passed with JDK v1.7.0_101 +1 javac 7m 35s the patch passed +1 checkstyle 1m 29s the patch passed +1 mvnsite 1m 32s the patch passed +1 mvneclipse 0m 37s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 2m 51s the patch passed +1 javadoc 1m 18s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 24s the patch passed with JDK v1.7.0_101 +1 unit 9m 21s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 21s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 103m 27s Reason Tests JDK v1.8.0_91 Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12818042/HADOOP-13212-branch-2-002.patch JIRA Issue HADOOP-13212 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 13cc67b1d984 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / c9f5348 Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/artifact/patchprocess/whitespace-eol.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10005/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          From my environment, i have been always getting timeout issues for those tests. The default buffer size has been retained to 8192 which is the default in httpclient as well. So it does not change the buffer in OOTB config.

          Attaching .2 version with indentation fixes.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - From my environment, i have been always getting timeout issues for those tests. The default buffer size has been retained to 8192 which is the default in httpclient as well. So it does not change the buffer in OOTB config. Attaching .2 version with indentation fixes.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          -1, doesn't merge without conflicts in index.md. Can you make the indentation consistent with the other entries too...we need four spaces for the text to go preformatted

          Show
          stevel@apache.org Steve Loughran added a comment - -1, doesn't merge without conflicts in index.md. Can you make the indentation consistent with the other entries too...we need four spaces for the text to go preformatted
          Hide
          stevel@apache.org Steve Loughran added a comment -

          yes, looks like you are hitting scale problems, aren't you? I was worried it was somehow related to the buffer size making things worse

          Show
          stevel@apache.org Steve Loughran added a comment - yes, looks like you are hitting scale problems, aren't you? I was worried it was somehow related to the buffer size making things worse
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Without patch, here is the report which are mainly due to timeout issues. Are you concerned on the timeout issues?.

          Tests in error:
            TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesToRemote:96->AbstractContractDistCpTest.largeFiles:172->AbstractContractDistCpTest.runDistCp:188 »
            TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesFromRemote:108->AbstractContractDistCpTest.largeFiles:174 » FileNotFound
            TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »
            TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »  test timed out after 1800...
            TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 »
          
          Tests run: 261, Failures: 0, Errors: 4, Skipped: 7
          
          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Without patch, here is the report which are mainly due to timeout issues. Are you concerned on the timeout issues?. Tests in error: TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesToRemote:96->AbstractContractDistCpTest.largeFiles:172->AbstractContractDistCpTest.runDistCp:188 » TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesFromRemote:108->AbstractContractDistCpTest.largeFiles:174 » FileNotFound TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » test timed out after 1800... TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 » Tests run: 261, Failures: 0, Errors: 4, Skipped: 7
          Hide
          stevel@apache.org Steve Loughran added a comment -

          that's not good. what test results do you get without the patch being applied?

          Show
          stevel@apache.org Steve Loughran added a comment - that's not good. what test results do you get without the patch being applied?
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          This is based on us-west-2 region. Ran the tests with "-DtestsThreadCount=2".

          Tests in error:
            TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesToRemote:96->AbstractContractDistCpTest.largeFiles:176 »
            TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »
            TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 »  test timed out after 1800...
            TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 »
          
          Tests run: 261, Failures: 0, Errors: 3, Skipped: 7
          
          Show
          rajesh.balamohan Rajesh Balamohan added a comment - This is based on us-west-2 region. Ran the tests with "-DtestsThreadCount=2". Tests in error: TestS3AContractDistCp>AbstractContractDistCpTest.largeFilesToRemote:96->AbstractContractDistCpTest.largeFiles:176 » TestS3ADeleteFilesOneByOne>TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » TestS3ADeleteManyFiles.testBulkRenameAndDelete:99 » test timed out after 1800... TestS3ADirectoryPerformance.testTimeToStatNonEmptyDirectory:153->timeToStatPath:179 » Tests run: 261, Failures: 0, Errors: 3, Skipped: 7
          Hide
          stevel@apache.org Steve Loughran added a comment -

          as usual: which s3a infra did you run the hadoop-aws test suite against?

          Show
          stevel@apache.org Steve Loughran added a comment - as usual: which s3a infra did you run the hadoop-aws test suite against?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          0 mvndep 0m 52s Maven dependency ordering for branch
          +1 mvninstall 7m 41s branch-2 passed
          +1 compile 7m 31s branch-2 passed with JDK v1.8.0_91
          +1 compile 7m 23s branch-2 passed with JDK v1.7.0_101
          +1 checkstyle 1m 30s branch-2 passed
          +1 mvnsite 1m 31s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 2m 25s branch-2 passed
          +1 javadoc 1m 9s branch-2 passed with JDK v1.8.0_91
          +1 javadoc 1m 26s branch-2 passed with JDK v1.7.0_101
          0 mvndep 0m 16s Maven dependency ordering for patch
          +1 mvninstall 1m 8s the patch passed
          +1 compile 7m 1s the patch passed with JDK v1.8.0_91
          +1 javac 7m 1s the patch passed
          +1 compile 7m 29s the patch passed with JDK v1.7.0_101
          +1 javac 7m 29s the patch passed
          +1 checkstyle 1m 28s the patch passed
          +1 mvnsite 1m 29s the patch passed
          +1 mvneclipse 0m 38s the patch passed
          -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 whitespace 0m 2s The patch 10 line(s) with tabs.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 46s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_91
          +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101
          +1 unit 8m 31s hadoop-common in the patch passed with JDK v1.7.0_101.
          +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          100m 0s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816770/HADOOP-13212-branch-2-001.patch
          JIRA Issue HADOOP-13212
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
          uname Linux 0952d5c049f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 687185f
          Default Java 1.7.0_101
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101
          findbugs v3.0.0
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/artifact/patchprocess/whitespace-eol.txt
          whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/artifact/patchprocess/whitespace-tabs.txt
          JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 52s Maven dependency ordering for branch +1 mvninstall 7m 41s branch-2 passed +1 compile 7m 31s branch-2 passed with JDK v1.8.0_91 +1 compile 7m 23s branch-2 passed with JDK v1.7.0_101 +1 checkstyle 1m 30s branch-2 passed +1 mvnsite 1m 31s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 2m 25s branch-2 passed +1 javadoc 1m 9s branch-2 passed with JDK v1.8.0_91 +1 javadoc 1m 26s branch-2 passed with JDK v1.7.0_101 0 mvndep 0m 16s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 7m 1s the patch passed with JDK v1.8.0_91 +1 javac 7m 1s the patch passed +1 compile 7m 29s the patch passed with JDK v1.7.0_101 +1 javac 7m 29s the patch passed +1 checkstyle 1m 28s the patch passed +1 mvnsite 1m 29s the patch passed +1 mvneclipse 0m 38s the patch passed -1 whitespace 0m 0s The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 2s The patch 10 line(s) with tabs. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 46s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_91 +1 javadoc 1m 23s the patch passed with JDK v1.7.0_101 +1 unit 8m 31s hadoop-common in the patch passed with JDK v1.7.0_101. +1 unit 0m 22s hadoop-aws in the patch passed with JDK v1.7.0_101. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 100m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12816770/HADOOP-13212-branch-2-001.patch JIRA Issue HADOOP-13212 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 0952d5c049f6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 687185f Default Java 1.7.0_101 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_101 findbugs v3.0.0 whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_101 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9941/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          HttpClient internally would set it to 8192 when these options were not available. Having this patch enables to change it at runtime for performance benchmarks and experiments.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - HttpClient internally would set it to 8192 when these options were not available. Having this patch enables to change it at runtime for performance benchmarks and experiments.

            People

            • Assignee:
              rajesh.balamohan Rajesh Balamohan
              Reporter:
              rajesh.balamohan Rajesh Balamohan
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development