Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12303

Change default EC cell size to 1MB for better performance

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-beta1
    • Component/s: None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      The cell size of the provided HDFS erasure coding policies has been changed from 64k to 1024k for better performance. The policy names have all been changed accordingly, i.e. RS-6-3.1024k.

      Description

      As discussed in HDFS-11814, 1MB cell size shows better performance than others during the tests.

      1. HDFS-12303.02.patch
        34 kB
        Wei Zhou
      2. HDFS-12303.01.patch
        28 kB
        Wei Zhou
      3. HDFS-12303.00.patch
        24 kB
        Wei Zhou

        Activity

        Hide
        zhouwei Wei Zhou added a comment -

        Initial patch.

        Show
        zhouwei Wei Zhou added a comment - Initial patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 30s Maven dependency ordering for branch
        +1 mvninstall 18m 13s trunk passed
        +1 compile 1m 53s trunk passed
        +1 checkstyle 0m 56s trunk passed
        +1 mvnsite 1m 53s trunk passed
        +1 findbugs 4m 0s trunk passed
        +1 javadoc 1m 21s trunk passed
              Patch Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 44s the patch passed
        +1 compile 1m 49s the patch passed
        +1 javac 1m 49s the patch passed
        +1 checkstyle 0m 51s the patch passed
        +1 mvnsite 1m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 4s The patch has no ill-formed XML file.
        +1 findbugs 3m 56s the patch passed
        +1 javadoc 1m 7s the patch passed
              Other Tests
        +1 unit 1m 29s hadoop-hdfs-client in the patch passed.
        -1 unit 103m 3s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        147m 2s



        Reason Tests
        Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100
          hadoop.hdfs.TestReadStripedFileWithMissingBlocks
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy
          hadoop.hdfs.tools.TestDFSZKFailoverController
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210
          hadoop.hdfs.TestDFSInputStream
          hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150
          hadoop.hdfs.TestReconstructStripedFile
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030
        Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130
          org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12303
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881906/HDFS-12303.00.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux e45c982d5cd5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2e43c28
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20700/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20700/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 18m 13s trunk passed +1 compile 1m 53s trunk passed +1 checkstyle 0m 56s trunk passed +1 mvnsite 1m 53s trunk passed +1 findbugs 4m 0s trunk passed +1 javadoc 1m 21s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 44s the patch passed +1 compile 1m 49s the patch passed +1 javac 1m 49s the patch passed +1 checkstyle 0m 51s the patch passed +1 mvnsite 1m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 3m 56s the patch passed +1 javadoc 1m 7s the patch passed       Other Tests +1 unit 1m 29s hadoop-hdfs-client in the patch passed. -1 unit 103m 3s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 147m 2s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure100   hadoop.hdfs.TestReadStripedFileWithMissingBlocks   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy   hadoop.hdfs.tools.TestDFSZKFailoverController   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure180   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure050   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure190   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure210   hadoop.hdfs.TestDFSInputStream   hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure040   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150   hadoop.hdfs.TestReconstructStripedFile   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 Timed out junit tests org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130   org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881906/HDFS-12303.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e45c982d5cd5 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e43c28 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20700/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20700/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20700/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        zhouwei Wei Zhou added a comment -

        The junit test failures reported can not be reproduced in my local environment. Resubmit the patch to trigger another round Hadoop QA check. Thanks!

        Show
        zhouwei Wei Zhou added a comment - The junit test failures reported can not be reproduced in my local environment. Resubmit the patch to trigger another round Hadoop QA check. Thanks!
        Hide
        andrew.wang Andrew Wang added a comment -

        Hi Wei, thanks for working on this! A single review comment:

        • I notice there's still an old reference to XOR-64k in TestDistCpUtils

        Toggling the JIRA status doesn't retrigger precommit, you'd need to re-upload the same patch. I ran a few of the failed tests and saw that they succeeded. Hoping it's just a flake.

        Show
        andrew.wang Andrew Wang added a comment - Hi Wei, thanks for working on this! A single review comment: I notice there's still an old reference to XOR-64k in TestDistCpUtils Toggling the JIRA status doesn't retrigger precommit, you'd need to re-upload the same patch. I ran a few of the failed tests and saw that they succeeded. Hoping it's just a flake.
        Hide
        zhouwei Wei Zhou added a comment -

        Thanks Andrew Wang for reviewing the patch and clarifying the JIRA details. The old references were removed from TestDistCpUtils and patch rebased. Thanks!

        Show
        zhouwei Wei Zhou added a comment - Thanks Andrew Wang for reviewing the patch and clarifying the JIRA details. The old references were removed from TestDistCpUtils and patch rebased. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 30s Maven dependency ordering for branch
        +1 mvninstall 13m 52s trunk passed
        +1 compile 14m 12s trunk passed
        +1 checkstyle 2m 6s trunk passed
        +1 mvnsite 2m 9s trunk passed
        +1 findbugs 3m 47s trunk passed
        +1 javadoc 1m 33s trunk passed
              Patch Compile Tests
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 39s the patch passed
        +1 compile 10m 41s the patch passed
        +1 javac 10m 41s the patch passed
        +1 checkstyle 2m 4s the patch passed
        +1 mvnsite 2m 7s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 4s The patch has no ill-formed XML file.
        +1 findbugs 4m 11s the patch passed
        +1 javadoc 1m 35s the patch passed
              Other Tests
        +1 unit 1m 21s hadoop-hdfs-client in the patch passed.
        -1 unit 87m 17s hadoop-hdfs in the patch failed.
        +1 unit 13m 54s hadoop-distcp in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        166m 51s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestReadStripedFileWithDecoding
          hadoop.hdfs.server.blockmanagement.TestBlockManager
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010
          hadoop.hdfs.TestPipelines
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12303
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882807/HDFS-12303.01.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 412b28b69b4b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 7a82d7b
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20772/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20772/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.       trunk Compile Tests 0 mvndep 1m 30s Maven dependency ordering for branch +1 mvninstall 13m 52s trunk passed +1 compile 14m 12s trunk passed +1 checkstyle 2m 6s trunk passed +1 mvnsite 2m 9s trunk passed +1 findbugs 3m 47s trunk passed +1 javadoc 1m 33s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 39s the patch passed +1 compile 10m 41s the patch passed +1 javac 10m 41s the patch passed +1 checkstyle 2m 4s the patch passed +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 4m 11s the patch passed +1 javadoc 1m 35s the patch passed       Other Tests +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 87m 17s hadoop-hdfs in the patch failed. +1 unit 13m 54s hadoop-distcp in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 166m 51s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestReadStripedFileWithDecoding   hadoop.hdfs.server.blockmanagement.TestBlockManager   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010   hadoop.hdfs.TestPipelines   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882807/HDFS-12303.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 412b28b69b4b 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 7a82d7b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20772/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20772/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20772/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        drankye Kai Zheng added a comment -

        Just took a look at the failed unit tests, finding most of them reported the following line:

        at org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780)
        

        pointing to this line of code

        Preconditions.checkState(ackedBytes <= sentBytes);
        

        which is being handled under HDFS-11882.

        Show
        drankye Kai Zheng added a comment - Just took a look at the failed unit tests, finding most of them reported the following line: at org.apache.hadoop.hdfs.DFSStripedOutputStream.updatePipeline(DFSStripedOutputStream.java:780) pointing to this line of code Preconditions.checkState(ackedBytes <= sentBytes); which is being handled under HDFS-11882 .
        Hide
        zhouwei Wei Zhou added a comment -

        Thanks Kai Zheng for review the patch. Those cases that has connection with updatePipeline are all passed in my testing, so it should have no connection with issue. This patch fixes an issue in TestBlockManager and mainly for another round of checking. Thanks!

        Show
        zhouwei Wei Zhou added a comment - Thanks Kai Zheng for review the patch. Those cases that has connection with updatePipeline are all passed in my testing, so it should have no connection with issue. This patch fixes an issue in TestBlockManager and mainly for another round of checking. Thanks!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
              trunk Compile Tests
        0 mvndep 1m 47s Maven dependency ordering for branch
        +1 mvninstall 15m 26s trunk passed
        +1 compile 14m 20s trunk passed
        +1 checkstyle 2m 7s trunk passed
        +1 mvnsite 2m 8s trunk passed
        +1 findbugs 3m 48s trunk passed
        +1 javadoc 1m 36s trunk passed
              Patch Compile Tests
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 1m 40s the patch passed
        +1 compile 10m 45s the patch passed
        +1 javac 10m 45s the patch passed
        +1 checkstyle 2m 8s the patch passed
        +1 mvnsite 2m 7s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 4s The patch has no ill-formed XML file.
        +1 findbugs 4m 10s the patch passed
        +1 javadoc 1m 34s the patch passed
              Other Tests
        +1 unit 1m 21s hadoop-hdfs-client in the patch passed.
        -1 unit 90m 44s hadoop-hdfs in the patch failed.
        +1 unit 14m 26s hadoop-distcp in the patch passed.
        +1 asflicense 0m 36s The patch does not generate ASF License warnings.
        173m 6s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy
          hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020
          hadoop.hdfs.TestLeaseRecoveryStriped
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
        Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-12303
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883267/HDFS-12303.02.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 486a492d3218 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 4249172
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/20817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20817/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20817/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.       trunk Compile Tests 0 mvndep 1m 47s Maven dependency ordering for branch +1 mvninstall 15m 26s trunk passed +1 compile 14m 20s trunk passed +1 checkstyle 2m 7s trunk passed +1 mvnsite 2m 8s trunk passed +1 findbugs 3m 48s trunk passed +1 javadoc 1m 36s trunk passed       Patch Compile Tests 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 1m 40s the patch passed +1 compile 10m 45s the patch passed +1 javac 10m 45s the patch passed +1 checkstyle 2m 8s the patch passed +1 mvnsite 2m 7s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 4s The patch has no ill-formed XML file. +1 findbugs 4m 10s the patch passed +1 javadoc 1m 34s the patch passed       Other Tests +1 unit 1m 21s hadoop-hdfs-client in the patch passed. -1 unit 90m 44s hadoop-hdfs in the patch failed. +1 unit 14m 26s hadoop-distcp in the patch passed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 173m 6s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes   hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy   hadoop.hdfs.tools.TestDFSHAAdminMiniCluster   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure020   hadoop.hdfs.TestLeaseRecoveryStriped   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure120   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure Timed out junit tests org.apache.hadoop.hdfs.TestWriteReadStripedFile   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12303 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883267/HDFS-12303.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 486a492d3218 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4249172 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-HDFS-Build/20817/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20817/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs hadoop-tools/hadoop-distcp U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20817/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        +1 LGTM, will check this in.

        Show
        andrew.wang Andrew Wang added a comment - +1 LGTM, will check this in.
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk. Thanks Wei for the nice contribution and Kai for reviewing!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk. Thanks Wei for the nice contribution and Kai for reviewing!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12246/)
        HDFS-12303. Change default EC cell size to 1MB for better performance. (wang: rev f29a0fc288a625522ba910e61b63fd5f10418b3d)

        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml
        • (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/TestErasureCodingCLI.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/test_ec_policies.xml
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12246 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12246/ ) HDFS-12303 . Change default EC cell size to 1MB for better performance. (wang: rev f29a0fc288a625522ba910e61b63fd5f10418b3d) (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSErasureCoding.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testErasureCodingConf.xml (edit) hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/util/TestDistCpUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/cli/TestErasureCodingCLI.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/SystemErasureCodingPolicies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/resources/test_ec_policies.xml

          People

          • Assignee:
            zhouwei Wei Zhou
            Reporter:
            zhouwei Wei Zhou
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development