Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: namenode, test
    • Labels:
      None

      Description

      This can be placed in org.apache.hadoop.hdfs.server.namenode.TestStartup test class.

      UPDATE: this JIRA is for name dir only; for the edit log directories, we have unit test TestInitializeSharedEdits#testInitializeSharedEdits, which tests that in HA mode, we should not have been able to start any NN without shared dir.

      1. HDFS-11085.001.patch
        4 kB
        Xiaobing Zhou
      2. HDFS-11085.000.patch
        4 kB
        Xiaobing Zhou

        Activity

        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Posted initial patch v000, please kindly review it, thx.

        Show
        xiaobingo Xiaobing Zhou added a comment - Posted initial patch v000, please kindly review it, thx.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 7m 30s trunk passed
        +1 compile 0m 47s trunk passed
        +1 checkstyle 0m 29s trunk passed
        +1 mvnsite 1m 0s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 54s trunk passed
        +1 javadoc 0m 42s trunk passed
        +1 mvninstall 0m 47s the patch passed
        +1 compile 0m 44s the patch passed
        +1 javac 0m 44s the patch passed
        -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 73 unchanged - 0 fixed = 74 total (was 73)
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 57s the patch passed
        +1 javadoc 0m 39s the patch passed
        +1 unit 57m 16s hadoop-hdfs in the patch passed.
        +1 asflicense 0m 18s The patch does not generate ASF License warnings.
        77m 20s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11085
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836933/HDFS-11085.000.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 42b99361cfc1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 20c4d8e
        Default Java 1.8.0_101
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17411/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17411/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17411/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 30s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 54s trunk passed +1 javadoc 0m 42s trunk passed +1 mvninstall 0m 47s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 73 unchanged - 0 fixed = 74 total (was 73) +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 57s the patch passed +1 javadoc 0m 39s the patch passed +1 unit 57m 16s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 77m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11085 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836933/HDFS-11085.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 42b99361cfc1 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 20c4d8e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17411/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17411/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17411/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment - - edited

        The patch looks good to me overall.

        Some minor comments:

        1. Checkstyle warning is related.
        2. MiniDFSCluster can be used in try-with-resource. Shorter and neater.
        3. I think we can use Paths.get() for joining path strings. Assuming file path separate is / is not platform independent probably. Also if there are methods in JDK or Apache component that work just fine, I hesitate to use third party libraries.
          661	    final String nnDirStr = Joiner.on("/").join(
          662	        hdfsDir.toString(),
          663	        GenericTestUtils.getMethodName(), "name");
          
        4. I don't think we need the implementation details about the Collection type (List). We just need the only element in the collections, don't we?
          673	      /* get and verify NN dir */
          674	      List<URI> nameDirs = (List<URI>) FSNamesystem.getNamespaceDirs(config);
          

          Alternatively:

          final Collection<URI> nameDirs = FSNamesystem.getNamespaceDirs(config);
          assertNotNull(nameDirs);
          assertTrue(nameDirs.iterator().hasNext());
          final URI nameDir = nameDirs.iterator().next();
          ...
          
        Show
        liuml07 Mingliang Liu added a comment - - edited The patch looks good to me overall. Some minor comments: Checkstyle warning is related. MiniDFSCluster can be used in try-with-resource. Shorter and neater. I think we can use Paths.get() for joining path strings. Assuming file path separate is / is not platform independent probably. Also if there are methods in JDK or Apache component that work just fine, I hesitate to use third party libraries. 661 final String nnDirStr = Joiner.on( "/" ).join( 662 hdfsDir.toString(), 663 GenericTestUtils.getMethodName(), "name" ); I don't think we need the implementation details about the Collection type (List). We just need the only element in the collections, don't we? 673 /* get and verify NN dir */ 674 List<URI> nameDirs = (List<URI>) FSNamesystem.getNamespaceDirs(config); Alternatively: final Collection<URI> nameDirs = FSNamesystem.getNamespaceDirs(config); assertNotNull(nameDirs); assertTrue(nameDirs.iterator().hasNext()); final URI nameDir = nameDirs.iterator().next(); ...
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        v001 addressed your comments, thanks Mingliang Liu.

        Show
        xiaobingo Xiaobing Zhou added a comment - v001 addressed your comments, thanks Mingliang Liu .
        Hide
        liuml07 Mingliang Liu added a comment -

        +1 pending on Jenkins.

        Show
        liuml07 Mingliang Liu added a comment - +1 pending on Jenkins.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 9m 10s trunk passed
        +1 compile 1m 1s trunk passed
        +1 checkstyle 0m 32s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 40s trunk passed
        +1 javadoc 0m 39s trunk passed
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 42s the patch passed
        +1 javac 0m 42s the patch passed
        +1 checkstyle 0m 23s the patch passed
        +1 mvnsite 0m 48s the patch passed
        +1 mvneclipse 0m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 47s the patch passed
        +1 javadoc 0m 35s the patch passed
        -1 unit 60m 30s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        81m 33s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.server.datanode.TestDirectoryScanner
          hadoop.hdfs.server.namenode.TestStartup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Issue HDFS-11085
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836996/HDFS-11085.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 810c59746e44 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5cad93d
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/17417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17417/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17417/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 9m 10s trunk passed +1 compile 1m 1s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 35s the patch passed -1 unit 60m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 81m 33s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.server.datanode.TestDirectoryScanner   hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-11085 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12836996/HDFS-11085.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 810c59746e44 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5cad93d Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17417/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17417/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17417/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        liuml07 Mingliang Liu added a comment -

        It's interesting hadoop.hdfs.server.namenode.TestStartup is failing. Would you have a look at this, Xiaobing Zhou?

        Show
        liuml07 Mingliang Liu added a comment - It's interesting hadoop.hdfs.server.namenode.TestStartup is failing. Would you have a look at this, Xiaobing Zhou ?
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        Mingliang Liu I filed a ticket HDFS-11107. This is a flaky failure, it's passed with and without my patch v001.

        Show
        xiaobingo Xiaobing Zhou added a comment - Mingliang Liu I filed a ticket HDFS-11107 . This is a flaky failure, it's passed with and without my patch v001.
        Hide
        liuml07 Mingliang Liu added a comment -

        Committed to trunk through branch-2.8 branches. I resolved trivial conflicts when committing. Thanks Xiaobing Zhou for contributing.

        Show
        liuml07 Mingliang Liu added a comment - Committed to trunk through branch-2.8 branches. I resolved trivial conflicts when committing. Thanks Xiaobing Zhou for contributing.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10774 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10774/)
        HDFS-11085. Add unit test for NameNode failing to start when name dir is (liuml07: rev 0c0ab102ab392ba07ed2aa8d8a67eef4c20cad9b)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10774 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10774/ ) HDFS-11085 . Add unit test for NameNode failing to start when name dir is (liuml07: rev 0c0ab102ab392ba07ed2aa8d8a67eef4c20cad9b) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java

          People

          • Assignee:
            xiaobingo Xiaobing Zhou
            Reporter:
            liuml07 Mingliang Liu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development