Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.1.0
    • erasure-coding, test
    • None
    • Reviewed

    Attachments

      1. HDFS-12677.1.patch
        6 kB
        Takanobu Asanuma
      2. HDFS-12677.002.patch
        7 kB
        Ekanth Sethuramalingam

      Activity

        Uploaded the 1st patch. The new test class with a random ec policy extends TestReconstructStripedFile with a few changes.

        When the ec policy is XOR-2-1-1024k, this assertion fails.

        TestReconstructStripedFile#testNNSendsErasureCodingTasks
             assertTrue(policy.getNumParityUnits() >= deadDN);
        

        I checked the code and this assertion seems not to make sense. So the 1st patch removes it. I confirmed that all EC policies pass the all tests with the patch in my local computer.

        tasanuma Takanobu Asanuma added a comment - Uploaded the 1st patch. The new test class with a random ec policy extends TestReconstructStripedFile with a few changes. When the ec policy is XOR-2-1-1024k , this assertion fails. TestReconstructStripedFile#testNNSendsErasureCodingTasks assertTrue(policy.getNumParityUnits() >= deadDN); I checked the code and this assertion seems not to make sense. So the 1st patch removes it. I confirmed that all EC policies pass the all tests with the patch in my local computer.
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        -1 docker 0m 11s Docker failed to build yetus/hadoop:0de40f0.



        This message was automatically generated.

        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 docker 0m 11s Docker failed to build yetus/hadoop:0de40f0. Subsystem Report/Notes JIRA Issue HDFS-12677 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12893020/HDFS-12677.1.patch Console output https://builds.apache.org/job/PreCommit-HDFS-Build/21746/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        ekanth Ekanth Sethuramalingam added a comment - - edited

        The first patch failed for me in a couple of other places (where the default policy was set/enabled and not the randomly chosen policy). I have uploaded a new patch that fixes this issue and I also updated the patch to change the assertTrue to assumeTrue to make sure tests (for XOR-2-1-1024k or a future EC policy that matches this condition) do not skip the check.

        ekanth Ekanth Sethuramalingam added a comment - - edited The first patch failed for me in a couple of other places (where the default policy was set/enabled and not the randomly chosen policy). I have uploaded a new patch that fixes this issue and I also updated the patch to change the assertTrue to assumeTrue to make sure tests (for XOR-2-1-1024k or a future EC policy that matches this condition) do not skip the check.
        ywskycn Wei Yan added a comment -

        Thanks for the patchs tasanuma0829, ekanth . LGTM.

        xiaochen eddyxu , mind take a look?

        ywskycn Wei Yan added a comment - Thanks for the patchs tasanuma0829 , ekanth . LGTM. xiaochen eddyxu , mind take a look?
        eddyxu Lei (Eddy) Xu added a comment -

        Thanks ekanth, ywskycn, tasanuma0829, the 002 patch LGTM. +1

        eddyxu Lei (Eddy) Xu added a comment - Thanks ekanth , ywskycn , tasanuma0829 , the 002 patch LGTM. +1

        ekanth, I had to do similar changes for HDFS-12919.
        HDFS-12677.002.patch LGTM.

        elgoiri Íñigo Goiri added a comment - ekanth , I had to do similar changes for HDFS-12919 . HDFS-12677.002.patch LGTM.

        I committed this. Thanks tasanuma0829 for the patch and ekanth for the revision.

        ekanth I apologize, the commit message should have cited both of you.

        cdouglas Christopher Douglas added a comment - I committed this. Thanks tasanuma0829 for the patch and ekanth for the revision. ekanth I apologize, the commit message should have cited both of you.
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13816 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13816/)
        HDFS-12677. Extend TestReconstructStripedFile with a random EC policy. (cdouglas: rev 39a5fbae479ecee3a563e2f4eb937471fbf666f8)

        • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFileWithRandomECPolicy.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13816 (See https://builds.apache.org/job/Hadoop-trunk-Commit/13816/ ) HDFS-12677 . Extend TestReconstructStripedFile with a random EC policy. (cdouglas: rev 39a5fbae479ecee3a563e2f4eb937471fbf666f8) (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFileWithRandomECPolicy.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestReconstructStripedFile.java
        genericqa genericqa added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 1m 7s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              trunk Compile Tests
        +1 mvninstall 29m 16s trunk passed
        +1 compile 0m 50s trunk passed
        +1 checkstyle 0m 40s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 shadedclient 9m 58s branch has no errors when building and testing our client artifacts.
        +1 findbugs 1m 49s trunk passed
        +1 javadoc 0m 54s trunk passed
              Patch Compile Tests
        +1 mvninstall 1m 1s the patch passed
        +1 compile 0m 50s the patch passed
        -1 javac 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 394 unchanged - 3 fixed = 397 total (was 397)
        +1 checkstyle 0m 41s the patch passed
        +1 mvnsite 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 shadedclient 9m 55s patch has no errors when building and testing our client artifacts.
        +1 findbugs 1m 50s the patch passed
        +1 javadoc 0m 49s the patch passed
              Other Tests
        -1 unit 149m 53s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        211m 30s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.TestDFSStripedOutputStreamWithFailure
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.ha.TestBootstrapStandby



        Subsystem Report/Notes
        Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f
        JIRA Issue HDFS-12677
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12914101/HDFS-12677.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle
        uname Linux 023d8701f494 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/patchprocess/precommit/personality/provided.sh
        git revision trunk / ac627f5
        maven version: Apache Maven 3.3.9
        Default Java 1.8.0_151
        findbugs v3.1.0-RC1
        javac https://builds.apache.org/job/PreCommit-HDFS-Build/23414/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/23414/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/23414/testReport/
        Max. process+thread count 3471 (vs. ulimit of 10000)
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/23414/console
        Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        genericqa genericqa added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 7s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       trunk Compile Tests +1 mvninstall 29m 16s trunk passed +1 compile 0m 50s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 0m 53s trunk passed +1 shadedclient 9m 58s branch has no errors when building and testing our client artifacts. +1 findbugs 1m 49s trunk passed +1 javadoc 0m 54s trunk passed       Patch Compile Tests +1 mvninstall 1m 1s the patch passed +1 compile 0m 50s the patch passed -1 javac 0m 50s hadoop-hdfs-project_hadoop-hdfs generated 3 new + 394 unchanged - 3 fixed = 397 total (was 397) +1 checkstyle 0m 41s the patch passed +1 mvnsite 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 shadedclient 9m 55s patch has no errors when building and testing our client artifacts. +1 findbugs 1m 50s the patch passed +1 javadoc 0m 49s the patch passed       Other Tests -1 unit 149m 53s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 211m 30s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.TestDFSStripedOutputStreamWithFailure   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.ha.TestBootstrapStandby Subsystem Report/Notes Docker Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f JIRA Issue HDFS-12677 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12914101/HDFS-12677.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle uname Linux 023d8701f494 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/patchprocess/precommit/personality/provided.sh git revision trunk / ac627f5 maven version: Apache Maven 3.3.9 Default Java 1.8.0_151 findbugs v3.1.0-RC1 javac https://builds.apache.org/job/PreCommit-HDFS-Build/23414/artifact/out/diff-compile-javac-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/23414/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/23414/testReport/ Max. process+thread count 3471 (vs. ulimit of 10000) modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/23414/console Powered by Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.

        Thanks for revising, reviewing and committing it, everyone!

        tasanuma Takanobu Asanuma added a comment - Thanks for revising, reviewing and committing it, everyone!

        People

          tasanuma Takanobu Asanuma
          tasanuma Takanobu Asanuma
          Votes:
          0 Vote for this issue
          Watchers:
          8 Start watching this issue

          Dates

            Created:
            Updated:
            Resolved: