Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9919

TestStandbyCheckpoints#testNonPrimarySBNUploadFSImage waitForCheckpoint incorrectly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Target Version/s:

      Description

      In HDFS-9787, it sloved the problem that standby NNs can upload FSImage to ANN after become non-primary standby NN. But in its unit test, it seems there is a small problem. when ANN change state to standby and standby nns should do a checkpoint. And in test, it makes a checpoint check as follow:

          for (int i = 0; i < NUM_NNS; i++) {
            // Once the standby catches up, it should do a checkpoint
            // and save to local directories.
            HATestUtil.waitForCheckpoint(cluster, 1, ImmutableList.of(12));
          }
      

      And in these code, the nnIdx for waitForCheckpoint is always 1, not i. It seems there is no need to check one standby nn's checkpoint NUM_NNS times.

        Activity

        Hide
        linyiqun Yiqun Lin added a comment -

        Attach a simple patch for this.

        Show
        linyiqun Yiqun Lin added a comment - Attach a simple patch for this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 20s trunk passed
        +1 compile 1m 2s trunk passed with JDK v1.8.0_74
        +1 compile 0m 49s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 22s trunk passed
        +1 mvnsite 0m 56s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 2m 21s trunk passed
        +1 javadoc 1m 32s trunk passed with JDK v1.8.0_74
        +1 javadoc 1m 58s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 57s the patch passed
        +1 compile 1m 0s the patch passed with JDK v1.8.0_74
        +1 javac 1m 0s the patch passed
        +1 compile 0m 48s the patch passed with JDK v1.7.0_95
        +1 javac 0m 48s the patch passed
        +1 checkstyle 0m 20s the patch passed
        +1 mvnsite 0m 57s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 2m 28s the patch passed
        +1 javadoc 1m 29s the patch passed with JDK v1.8.0_74
        +1 javadoc 2m 2s the patch passed with JDK v1.7.0_95
        -1 unit 75m 46s hadoop-hdfs in the patch failed with JDK v1.8.0_74.
        -1 unit 81m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 1m 10s Patch does not generate ASF License warnings.
        189m 1s



        Reason Tests
        JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.server.mover.TestStorageMover
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.web.TestWebHDFS
          hadoop.hdfs.TestDFSUpgradeFromImage
        JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager
          org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.hdfs.TestAppendDifferentChecksum
          org.apache.hadoop.hdfs.TestGetBlocks
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.TestEncryptionZones
          hadoop.hdfs.server.namenode.TestSecureNameNode
          hadoop.hdfs.server.namenode.TestDecommissioningStatus
          hadoop.hdfs.server.namenode.TestFileTruncate
          hadoop.hdfs.server.datanode.TestDataNodeMXBean
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider
        JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.server.balancer.TestBalancer
          org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding
          org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791984/HDFS-9919.001.patch
        JIRA Issue HDFS-9919
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 88b34f3f3f61 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f86850b
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14741/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14741/console
        Powered by Apache Yetus 0.2.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 20s trunk passed +1 compile 1m 2s trunk passed with JDK v1.8.0_74 +1 compile 0m 49s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 22s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 2m 21s trunk passed +1 javadoc 1m 32s trunk passed with JDK v1.8.0_74 +1 javadoc 1m 58s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 57s the patch passed +1 compile 1m 0s the patch passed with JDK v1.8.0_74 +1 javac 1m 0s the patch passed +1 compile 0m 48s the patch passed with JDK v1.7.0_95 +1 javac 0m 48s the patch passed +1 checkstyle 0m 20s the patch passed +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 28s the patch passed +1 javadoc 1m 29s the patch passed with JDK v1.8.0_74 +1 javadoc 2m 2s the patch passed with JDK v1.7.0_95 -1 unit 75m 46s hadoop-hdfs in the patch failed with JDK v1.8.0_74. -1 unit 81m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 1m 10s Patch does not generate ASF License warnings. 189m 1s Reason Tests JDK v1.8.0_74 Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.server.mover.TestStorageMover   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.web.TestWebHDFS   hadoop.hdfs.TestDFSUpgradeFromImage JDK v1.8.0_74 Timed out junit tests org.apache.hadoop.hdfs.server.blockmanagement.TestBlockManager   org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   org.apache.hadoop.hdfs.TestDecommission   org.apache.hadoop.hdfs.TestAppendDifferentChecksum   org.apache.hadoop.hdfs.TestGetBlocks JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.server.namenode.TestSecureNameNode   hadoop.hdfs.server.namenode.TestDecommissioningStatus   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestRequestHedgingProxyProvider JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2   org.apache.hadoop.hdfs.TestDFSClientRetries   org.apache.hadoop.hdfs.server.balancer.TestBalancer   org.apache.hadoop.hdfs.TestReadStripedFileWithDecoding   org.apache.hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12791984/HDFS-9919.001.patch JIRA Issue HDFS-9919 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 88b34f3f3f61 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f86850b Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_74 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_74.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14741/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14741/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14741/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
        Hide
        andrew.wang Andrew Wang added a comment -

        LGTM +1, will commit shortly

        Show
        andrew.wang Andrew Wang added a comment - LGTM +1, will commit shortly
        Hide
        andrew.wang Andrew Wang added a comment -

        Committed to trunk, thanks for the contribution Lin Yiqun!

        Show
        andrew.wang Andrew Wang added a comment - Committed to trunk, thanks for the contribution Lin Yiqun!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9444 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9444/)
        HDFS-9919. TestStandbyCheckpoints#testNonPrimarySBNUploadFSImage (wang: rev a14a6f08ee9404168affe91affd095e349630971)

        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9444 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9444/ ) HDFS-9919 . TestStandbyCheckpoints#testNonPrimarySBNUploadFSImage (wang: rev a14a6f08ee9404168affe91affd095e349630971) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
        Hide
        linyiqun Yiqun Lin added a comment -

        Thanks Andrew Wang for quick review and commit!

        Show
        linyiqun Yiqun Lin added a comment - Thanks Andrew Wang for quick review and commit!

          People

          • Assignee:
            linyiqun Yiqun Lin
            Reporter:
            linyiqun Yiqun Lin
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development