Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9597

BaseReplicationPolicyTest should update data node stats after adding a data node

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: datanode, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Looks like HDFS-9034 broke TestReplicationPolicyConsiderLoad#testChooseTargetWithDecomNodes.
      This test has been failing since yesterday.

      1. HDFS-9597.001.patch
        0.8 kB
        Wei-Chiu Chuang

        Issue Links

          Activity

          Hide
          benoyantony Benoy Antony added a comment -

          committed to branch-2.8 also.

          Show
          benoyantony Benoy Antony added a comment - committed to branch-2.8 also.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Benoy Antony, there is a branch-2.8 where you need to land this patch for it to be in 2.8.0.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Benoy Antony , there is a branch-2.8 where you need to land this patch for it to be in 2.8.0.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Thanks! Phil Yang and Surendra Singh Lilhore for reviews and Benoy Antony for committing this patch.
          Happy holiday!

          Show
          jojochuang Wei-Chiu Chuang added a comment - Thanks! Phil Yang and Surendra Singh Lilhore for reviews and Benoy Antony for committing this patch. Happy holiday!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9025/)
          HDFS-9597. BaseReplicationPolicyTest should update data node stats after (benoy: rev 8cfd672397efd91c471c417afef3aff85f64b506)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BaseReplicationPolicyTest.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9025 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9025/ ) HDFS-9597 . BaseReplicationPolicyTest should update data node stats after (benoy: rev 8cfd672397efd91c471c417afef3aff85f64b506) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BaseReplicationPolicyTest.java
          Hide
          benoyantony Benoy Antony added a comment -

          Committed in trunk and branch-2. Thanks Wei-Chiu Chuang for the fix and Phil Yang for the review.

          Show
          benoyantony Benoy Antony added a comment - Committed in trunk and branch-2. Thanks Wei-Chiu Chuang for the fix and Phil Yang for the review.
          Hide
          benoyantony Benoy Antony added a comment -

          +1. The fix looks good to me. Thanks for taking care of it.
          Will commit the patch after the tests.

          Show
          benoyantony Benoy Antony added a comment - +1. The fix looks good to me. Thanks for taking care of it. Will commit the patch after the tests.
          Hide
          surendrasingh Surendra Singh Lilhore added a comment -

          Thanks Wei-Chiu Chuang for fixing this issue..
          +1 LGTM

          Show
          surendrasingh Surendra Singh Lilhore added a comment - Thanks Wei-Chiu Chuang for fixing this issue.. +1 LGTM
          Hide
          yangzhe1991 Phil Yang added a comment -

          Sorry for duplicating, this patch looks good to me

          Show
          yangzhe1991 Phil Yang added a comment - Sorry for duplicating, this patch looks good to me
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Failed test are unrelated.
          The ASF license warning appears to be a false positive.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Failed test are unrelated. The ASF license warning appears to be a false positive.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 12s trunk passed
          +1 compile 0m 46s trunk passed with JDK v1.8.0_66
          +1 compile 0m 45s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 2s trunk passed
          +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 44s the patch passed with JDK v1.8.0_66
          +1 javac 0m 44s the patch passed
          +1 compile 0m 46s the patch passed with JDK v1.7.0_91
          +1 javac 0m 46s the patch passed
          +1 checkstyle 0m 17s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66
          +1 javadoc 1m 55s the patch passed with JDK v1.7.0_91
          -1 unit 58m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_66.
          -1 unit 54m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 26s Patch generated 1 ASF License warnings.
          141m 43s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDecommission
            hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
            hadoop.hdfs.server.datanode.TestBlockScanner
            hadoop.hdfs.server.namenode.TestNNThroughputBenchmark
          JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779377/HDFS-9597.001.patch
          JIRA Issue HDFS-9597
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4d6786eae801 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 0af492b
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13988/testReport/
          asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13988/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 12s trunk passed +1 compile 0m 46s trunk passed with JDK v1.8.0_66 +1 compile 0m 45s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 2s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 1m 54s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 52s the patch passed +1 compile 0m 44s the patch passed with JDK v1.8.0_66 +1 javac 0m 44s the patch passed +1 compile 0m 46s the patch passed with JDK v1.7.0_91 +1 javac 0m 46s the patch passed +1 checkstyle 0m 17s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 11s the patch passed with JDK v1.8.0_66 +1 javadoc 1m 55s the patch passed with JDK v1.7.0_91 -1 unit 58m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_66. -1 unit 54m 34s hadoop-hdfs in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 26s Patch generated 1 ASF License warnings. 141m 43s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.hdfs.TestDecommission   hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits   hadoop.hdfs.server.datanode.TestBlockScanner   hadoop.hdfs.server.namenode.TestNNThroughputBenchmark JDK v1.7.0_91 Failed junit tests hadoop.hdfs.TestRecoverStripedFile Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779377/HDFS-9597.001.patch JIRA Issue HDFS-9597 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4d6786eae801 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0af492b Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/13988/testReport/ asflicense https://builds.apache.org/job/PreCommit-HDFS-Build/13988/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HDFS-Build/13988/console This message was automatically generated.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Rev01: call HeartbeatManager.updateDnStat() to add stats after adding a datanode.

          I suspect other tests may also be affected by HDFS-9034 though.

          Show
          jojochuang Wei-Chiu Chuang added a comment - Rev01: call HeartbeatManager.updateDnStat() to add stats after adding a datanode. I suspect other tests may also be affected by HDFS-9034 though.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          The test case failed, because after HDFS-9034, the data node storage are considered failed, thus not included in the stats.

          Show
          jojochuang Wei-Chiu Chuang added a comment - The test case failed, because after HDFS-9034 , the data node storage are considered failed, thus not included in the stats.

            People

            • Assignee:
              jojochuang Wei-Chiu Chuang
              Reporter:
              jojochuang Wei-Chiu Chuang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development