Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11112

Journal Nodes should refuse to format non-empty directories

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Journal Nodes should reject the format RPC request if a storage directory is non-empty. The relevant code is in JNStorage#format.

        void format(NamespaceInfo nsInfo) throws IOException {
          setStorageInfo(nsInfo);
      ...
          unlockAll();
          sd.clearDirectory();
          writeProperties(sd);
          createPaxosDir();
          analyzeStorage();
      

      This would make the behavior similar to namenode -format -nonInteractive.

      1. HDFS-11112.001.patch
        3 kB
        Yiqun Lin
      2. HDFS-11112.002.patch
        3 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Sounds like similar to HDFS-10457?

          Show
          jojochuang Wei-Chiu Chuang added a comment - Sounds like similar to HDFS-10457 ?
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Indeed, thanks for the pointer Wei-Chiu Chuang.

          Show
          arpitagarwal Arpit Agarwal added a comment - Indeed, thanks for the pointer Wei-Chiu Chuang .
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Arpit Agarwal for reporting this. Will attach a patch soon.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Arpit Agarwal for reporting this. Will attach a patch soon.
          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a initial patch. Please have a review. Thanks!

          Show
          linyiqun Yiqun Lin added a comment - Attach a initial patch. Please have a review. Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 3s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          +1 checkstyle 0m 24s the patch passed
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 36s the patch passed
          +1 unit 53m 29s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          71m 56s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837981/HDFS-11112.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a61d24f7e610 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 026b39a
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17471/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17471/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 3s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 36s the patch passed +1 unit 53m 29s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 71m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837981/HDFS-11112.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a61d24f7e610 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 026b39a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17471/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17471/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thank you for taking this up Yiqun Lin. Couple of nitpicks with the patch:

          1. This comment can be removed. Since we call unlockAll() and analyzeStorage(), it follows that we expect the storage can be non-empty and locked.
                // Unlock the directory since the storage maybe not empty and
                // locked before formatting sometimes.
            
          2. We should wrap the sd.analyzeStorage(StartupOption.FORMAT, this, true); call in try-finally and release the lock in the finally block, so we don't return with the lock held if analyze failed.

          This change may be incompatible as there is a good chance of breaking someone's test setup. I'd commit it to trunk only, just to be safe.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thank you for taking this up Yiqun Lin . Couple of nitpicks with the patch: This comment can be removed. Since we call unlockAll() and analyzeStorage(), it follows that we expect the storage can be non-empty and locked. // Unlock the directory since the storage maybe not empty and // locked before formatting sometimes. We should wrap the sd.analyzeStorage(StartupOption.FORMAT, this, true); call in try-finally and release the lock in the finally block, so we don't return with the lock held if analyze failed. This change may be incompatible as there is a good chance of breaking someone's test setup. I'd commit it to trunk only, just to be safe.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Arpit Agarwal for the comments. Attach a new patch to address the comments.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Arpit Agarwal for the comments. Attach a new patch to address the comments.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 33s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 54s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 0m 40s the patch passed
          -1 unit 94m 50s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          115m 28s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestStartup
            hadoop.hdfs.TestDataTransferKeepalive
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestDFSStripedOutputStream
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HDFS-11112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838099/HDFS-11112.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6863068bda1c 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / e1c6ef2
          Default Java 1.8.0_101
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17478/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17478/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 33s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 0m 40s the patch passed -1 unit 94m 50s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 115m 28s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestStartup   hadoop.hdfs.TestDataTransferKeepalive   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSStripedOutputStream Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HDFS-11112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838099/HDFS-11112.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6863068bda1c 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e1c6ef2 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/17478/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17478/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17478/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          The failed tests are not related, hi Arpit Agarwal, could you please have a review on the latest patch? I have addressed your comments in the latest patch. Thanks!

          Show
          linyiqun Yiqun Lin added a comment - The failed tests are not related, hi Arpit Agarwal , could you please have a review on the latest patch? I have addressed your comments in the latest patch. Thanks!
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          Hi Arpit Agarwal
          a while back I also studied the behavior of JN in this situation (with minor deviation) after finished HDFS-10360 to deal with the similar issue in DataNodes. Here's what I saw. Could you comment if this is also what you saw? Maybe the situation is different, and from what I saw JN didn't automatically format the directory.

          if current/VERSION is missing, it will start with no problem like nothing has happened. No auto-format. But when NN tries to send edits to the JN, the JN will respond with JournalNotFormattedException. Cloudera Manager dashboard will see JN and NN in bad health.

          2016-06-07 11:14:18,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournal
          Protocol.getJournalState from 172.26.10.80:36663 Call#7 Retry#0
          org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal Storage Directory /data/jn/nameservice1 not formatted
          at org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472)
          at org.apache.hadoop.hdfs.qjournal.server.Journal.getLastPromisedEpoch(Journal.java:245)
          at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:124)
          at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslat
          orPB.java:118)
          at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.
          java:25415)
          at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
          at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
          at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:415)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
          at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)

          If I remove current/VERSION from all three JNs,
          NN will shutdown because it's unabel to get a quorum.

          If I put the file back to an active JN, it will still complain about it. But if I restart the JN with the file in place, the JN will function normally.

          So in the case of JN, there is no permanent harm and the damage can be reverted.

          Yiqun Lin I think the patch looks good to me overall. Do you know the behavior of JN after your patch? Does JN simply ignores format? Does JN continues to function normally?

          Show
          jojochuang Wei-Chiu Chuang added a comment - Hi Arpit Agarwal a while back I also studied the behavior of JN in this situation (with minor deviation) after finished HDFS-10360 to deal with the similar issue in DataNodes. Here's what I saw. Could you comment if this is also what you saw? Maybe the situation is different, and from what I saw JN didn't automatically format the directory. if current/VERSION is missing, it will start with no problem like nothing has happened. No auto-format. But when NN tries to send edits to the JN, the JN will respond with JournalNotFormattedException. Cloudera Manager dashboard will see JN and NN in bad health. 2016-06-07 11:14:18,656 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8485, call org.apache.hadoop.hdfs.qjournal.protocol.QJournal Protocol.getJournalState from 172.26.10.80:36663 Call#7 Retry#0 org.apache.hadoop.hdfs.qjournal.protocol.JournalNotFormattedException: Journal Storage Directory /data/jn/nameservice1 not formatted at org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:472) at org.apache.hadoop.hdfs.qjournal.server.Journal.getLastPromisedEpoch(Journal.java:245) at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:124) at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslat orPB.java:118) at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos. java:25415) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) If I remove current/VERSION from all three JNs, NN will shutdown because it's unabel to get a quorum. If I put the file back to an active JN, it will still complain about it. But if I restart the JN with the file in place, the JN will function normally. So in the case of JN, there is no permanent harm and the damage can be reverted. Yiqun Lin I think the patch looks good to me overall. Do you know the behavior of JN after your patch? Does JN simply ignores format? Does JN continues to function normally?
          Hide
          linyiqun Yiqun Lin added a comment - - edited

          Hi Wei-Chiu Chuang, there are some comments from me:

          if current/VERSION is missing, it will start with no problem like nothing has happened. No auto-format. But when NN tries to send edits to the JN, the JN will respond with JournalNotFormattedException.

          Can I understand this as that we should start JN failed if current/VERSION is missing and avoid the other exceptions that will be threw in the subsequent operations? I haven't tested for this special case and I think this should not be the situation that addressed in the patch of this JIRA. If this is a problem, one way we can to do a additional "empty check" similar HDFS-10360. Now the method JNStorage.analyzeAndRecoverStorage doesn't do the check for missing current/VERSION file when initialize the JNStorage instance.

          Do you know the behavior of JN after your patch? Does JN simply ignores format? Does JN continues to function normally?

          My patch makes JN format failed if there is already storage directories. I'm sure It will ignores the format operations since it will throws the exception and terminate the subsequent format operations.

          Correct me if I am wrong.Thanks.

          Show
          linyiqun Yiqun Lin added a comment - - edited Hi Wei-Chiu Chuang , there are some comments from me: if current/VERSION is missing, it will start with no problem like nothing has happened. No auto-format. But when NN tries to send edits to the JN, the JN will respond with JournalNotFormattedException. Can I understand this as that we should start JN failed if current/VERSION is missing and avoid the other exceptions that will be threw in the subsequent operations? I haven't tested for this special case and I think this should not be the situation that addressed in the patch of this JIRA. If this is a problem, one way we can to do a additional "empty check" similar HDFS-10360 . Now the method JNStorage.analyzeAndRecoverStorage doesn't do the check for missing current/VERSION file when initialize the JNStorage instance. Do you know the behavior of JN after your patch? Does JN simply ignores format? Does JN continues to function normally? My patch makes JN format failed if there is already storage directories. I'm sure It will ignores the format operations since it will throws the exception and terminate the subsequent format operations. Correct me if I am wrong.Thanks.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          IIUC, JNStorage#format may be called when starting NameNode with -format or -initializeSharedEdits arguments. So this is a different scenario than what I described (starting JournalNode).

          Show
          jojochuang Wei-Chiu Chuang added a comment - IIUC, JNStorage#format may be called when starting NameNode with -format or -initializeSharedEdits arguments. So this is a different scenario than what I described (starting JournalNode).
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Sorry I couldn't respond earlier. Wei-Chiu Chuang that is correct, if the format request comes in via RPC then the JN skips the check for non-empty directories.

          Show
          arpitagarwal Arpit Agarwal added a comment - Sorry I couldn't respond earlier. Wei-Chiu Chuang that is correct, if the format request comes in via RPC then the JN skips the check for non-empty directories.
          Hide
          jojochuang Wei-Chiu Chuang added a comment -

          I see, Arpit Agarwal. In that case the patch looks good to me.

          Show
          jojochuang Wei-Chiu Chuang added a comment - I see, Arpit Agarwal . In that case the patch looks good to me.
          Hide
          linyiqun Yiqun Lin added a comment -

          HI Arpit Agarwal, would you have a quick look for the latest patch, I see this the state of this JIRA has not been updated for many days. Thanks.

          Show
          linyiqun Yiqun Lin added a comment - HI Arpit Agarwal , would you have a quick look for the latest patch, I see this the state of this JIRA has not been updated for many days. Thanks.
          Hide
          arpitagarwal Arpit Agarwal added a comment - - edited

          Sorry I missed this one Yiqun Lin.

          +1 pending Jenkins. (retriggered Jenkins as the previous run had a couple of test failures which may be caused by the change in behavior).

          Show
          arpitagarwal Arpit Agarwal added a comment - - edited Sorry I missed this one Yiqun Lin . +1 pending Jenkins. (retriggered Jenkins as the previous run had a couple of test failures which may be caused by the change in behavior).
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 10s trunk passed
          +1 compile 0m 48s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 59s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 49s the patch passed
          +1 javac 0m 49s the patch passed
          +1 checkstyle 0m 27s the patch passed
          +1 mvnsite 0m 56s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 38s the patch passed
          -1 unit 99m 58s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          127m 25s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMXBean
          Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11112
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838099/HDFS-11112.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 520934c90454 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b6f290d
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18309/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18309/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 10s trunk passed +1 compile 0m 48s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 49s the patch passed +1 javac 0m 49s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 38s the patch passed -1 unit 99m 58s hadoop-hdfs in the patch failed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 127m 25s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMXBean Timed out junit tests org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11112 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12838099/HDFS-11112.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 520934c90454 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b6f290d Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/18309/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18309/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18309/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          I've committed this. The test failures look unrelated.

          Thank you for this nice improvement Yiqun Lin.

          Show
          arpitagarwal Arpit Agarwal added a comment - I've committed this. The test failures look unrelated. Thank you for this nice improvement Yiqun Lin .
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11197 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11197/)
          HDFS-11112. Journal Nodes should refuse to format non-empty directories. (arp: rev 6aa09dc28adc3e7c81258568ac08996eee0356da)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournal.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11197 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11197/ ) HDFS-11112 . Journal Nodes should refuse to format non-empty directories. (arp: rev 6aa09dc28adc3e7c81258568ac08996eee0356da) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Storage.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournal.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JNStorage.java
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Arpit Agarwal for the commit!

          Show
          linyiqun Yiqun Lin added a comment - Thanks Arpit Agarwal for the commit!

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              arpitagarwal Arpit Agarwal
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development