Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: hdfs-client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently the APIs for safe-mode are part of DistributedFileSystem, which is supposed to be a private interface. However, dependent software often wants to wait until the NN is out of safemode. Though it could poll trying to create a file and catching SafeModeException, we should consider making some of these APIs public.

      1. HDFS-2413.patch
        6 kB
        Harsh J
      2. HDFS-2413.patch
        5 kB
        Harsh J
      3. HDFS-2413.patch
        5 kB
        Harsh J
      4. HDFS-2413.patch
        4 kB
        Harsh J

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/)
          HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1032 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/ ) HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #997 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/)
          HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #997 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/ ) HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1944 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1944/)
          HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632)

          Result = FAILURE
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1944 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1944/ ) HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #741 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/741/)
          svn merge -c 1305632 from trunk for HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #741 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/741/ ) svn merge -c 1305632 from trunk for HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/)
          HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/ ) HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #721 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/)
          svn merge -c 1305632 from trunk for HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #721 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/ ) svn merge -c 1305632 from trunk for HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1932 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1932/)
          HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1932 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1932/ ) HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. Contributed by harsh (Revision 1305632) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305632 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #731 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/731/)
          svn merge -c 1305632 from trunk for HDFS-2413. Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633)

          Result = SUCCESS
          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #731 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/731/ ) svn merge -c 1305632 from trunk for HDFS-2413 . Add an API DistributedFileSystem.isInSafeMode() and change DistributedFileSystem to @InterfaceAudience.LimitedPrivate. (Revision 1305633) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305633 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestSafeMode.java
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Harsh!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Harsh!
          Hide
          Aaron T. Myers added a comment -

          Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment.

          Works for me. Thanks for thinking about it.

          Nicholas, would you like to commit this patch or shall I? We should also probably update the target version to 0.23.3 instead of 0.23.2.

          Show
          Aaron T. Myers added a comment - Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment. Works for me. Thanks for thinking about it. Nicholas, would you like to commit this patch or shall I? We should also probably update the target version to 0.23.3 instead of 0.23.2.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 Thanks Harsh, the patch looks good.

          > Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable?

          Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 Thanks Harsh, the patch looks good. > Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable? Since this is a new API, we are not sure if want to change it. Let's leave it as-is for the moment.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519909/HDFS-2413.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.TestGetBlocks
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519909/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2097//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Thanks, Harsh. The latest patch looks good to me. Nicholas, does it seem OK to you?

          Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable?

          Show
          Aaron T. Myers added a comment - Thanks, Harsh. The latest patch looks good to me. Nicholas, does it seem OK to you? Also, might it be reasonable to mark just isInSafeMode @InterfaceStability.Stable?
          Hide
          Harsh J added a comment -

          Silly me (Applied changes onto the wrong class). Here's a new patch that cleans the additions up, and applies Nicholas' comments as well, properly this time.

          Show
          Harsh J added a comment - Silly me (Applied changes onto the wrong class). Here's a new patch that cleans the additions up, and applies Nicholas' comments as well, properly this time.
          Hide
          Aaron T. Myers added a comment -

          Hey Harsh, the patch looks like it will work to me, and I agree that the test failures are unrelated: the HAAdmin ones should now be fixed, the other two are HDFS-3142 and HDFS-3143. But, per Nicholas, I thought we were going to change the InterfaceAudience of DistributedFileSystem to LimitedPrivate from Private?

          Show
          Aaron T. Myers added a comment - Hey Harsh, the patch looks like it will work to me, and I agree that the test failures are unrelated: the HAAdmin ones should now be fixed, the other two are HDFS-3142 and HDFS-3143 . But, per Nicholas, I thought we were going to change the InterfaceAudience of DistributedFileSystem to LimitedPrivate from Private?
          Hide
          Harsh J added a comment -

          Per https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/, all reported failures appear unrelated. The HA tests have been failing for a while now (possible misconfig?) and the other ones have failed since 3 runs before (And are unrelated to this safe mode wrapper change, as noticeable in org.apache.hadoop.hdfs.TestGetBlocks.getBlocksWithException).

          Show
          Harsh J added a comment - Per https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/ , all reported failures appear unrelated. The HA tests have been failing for a while now (possible misconfig?) and the other ones have failed since 3 runs before (And are unrelated to this safe mode wrapper change, as noticeable in org.apache.hadoop.hdfs.TestGetBlocks.getBlocksWithException).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519869/HDFS-2413.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.tools.TestDFSHAAdmin
          org.apache.hadoop.hdfs.TestGetBlocks
          org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster
          org.apache.hadoop.cli.TestHDFSCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519869/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.TestDFSHAAdmin org.apache.hadoop.hdfs.TestGetBlocks org.apache.hadoop.hdfs.tools.TestDFSHAAdminMiniCluster org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2095//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Sounds good Nicholas.

          I updated the patch to add that in.

          Show
          Harsh J added a comment - Sounds good Nicholas. I updated the patch to add that in.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          After some more thought, we should change DistributedFileSystem to LimitedPrivate + Unstable since there are other methods (e.g. concat(..)) that other projects are using. Sounds good?

          Show
          Tsz Wo Nicholas Sze added a comment - After some more thought, we should change DistributedFileSystem to LimitedPrivate + Unstable since there are other methods (e.g. concat(..)) that other projects are using. Sounds good?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          My first thought is that you may simply annotate only DistributedFileSystem.isInSafeMode(), i.e.

          @InterfaceAudience.LimitedPrivate({ "MapReduce", "HBase" })
          public boolean isInSafeMode() throws IOException {
          

          However, I am not sure if it is okay since DistributedFileSystem is annotated as private. Let me check this.

          Show
          Tsz Wo Nicholas Sze added a comment - My first thought is that you may simply annotate only DistributedFileSystem.isInSafeMode(), i.e. @InterfaceAudience.LimitedPrivate({ "MapReduce" , "HBase" }) public boolean isInSafeMode() throws IOException { However, I am not sure if it is okay since DistributedFileSystem is annotated as private. Let me check this.
          Hide
          Harsh J added a comment -

          Nicholas,

          Agreed that changing DFSAdmin does appear weird.

          Should we make DistributedFileSystem LimitedPrivate for HBase and take care in updating it? AFAICT, use of DFS was never encouraged.

          Show
          Harsh J added a comment - Nicholas, Agreed that changing DFSAdmin does appear weird. Should we make DistributedFileSystem LimitedPrivate for HBase and take care in updating it? AFAICT, use of DFS was never encouraged.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Patch that makes DFSAdmin carry two utility functions ...

          It is not a good idea to change DFSAdmin to LimitedPrivate. The method setSafeMode(String[] argv, int idx) is not really a good API for other projects to use.

          The method isInSafeMode() added in DistributedFileSystem is sufficient.

          Show
          Tsz Wo Nicholas Sze added a comment - > Patch that makes DFSAdmin carry two utility functions ... It is not a good idea to change DFSAdmin to LimitedPrivate. The method setSafeMode(String[] argv, int idx) is not really a good API for other projects to use. The method isInSafeMode() added in DistributedFileSystem is sufficient.
          Hide
          Harsh J added a comment -

          All reportedly failing tests are unrelated to this method-addition change.

          They all seem to have (again) failed due to a bad cleanup:

          Cannot lock storage /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1. The directory is already locked.
          
          Show
          Harsh J added a comment - All reportedly failing tests are unrelated to this method-addition change. They all seem to have (again) failed due to a bad cleanup: Cannot lock storage /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/name1. The directory is already locked.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514218/HDFS-2413.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.TestParallelRead
          org.apache.hadoop.hdfs.TestBlockMissingException
          org.apache.hadoop.hdfs.TestLargeBlock
          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.TestReplication
          org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery
          org.apache.hadoop.hdfs.TestMissingBlocksAlert
          org.apache.hadoop.hdfs.TestFileStatus
          org.apache.hadoop.hdfs.TestDFSClientExcludedNodes
          org.apache.hadoop.hdfs.TestMultiThreadedHflush
          org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
          org.apache.hadoop.hdfs.TestRenameWhileOpen
          org.apache.hadoop.hdfs.TestWriteRead
          org.apache.hadoop.hdfs.TestLeaseRecovery
          org.apache.hadoop.hdfs.TestDFSMkdirs
          org.apache.hadoop.hdfs.TestHDFSTrash
          org.apache.hadoop.hdfs.TestFileAppend4
          org.apache.hadoop.hdfs.TestQuota
          org.apache.hadoop.hdfs.security.token.block.TestBlockToken
          org.apache.hadoop.hdfs.TestFileAppend2
          org.apache.hadoop.hdfs.TestListFilesInDFS
          org.apache.hadoop.hdfs.TestFileAppend3
          org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure
          org.apache.hadoop.hdfs.TestDecommission
          org.apache.hadoop.hdfs.TestFileCorruption
          org.apache.hadoop.hdfs.TestFileCreationEmpty
          org.apache.hadoop.hdfs.TestSetTimes
          org.apache.hadoop.hdfs.TestBalancerBandwidth
          org.apache.hadoop.hdfs.TestFileCreationClient
          org.apache.hadoop.hdfs.TestDFSShell
          org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.hdfs.security.TestDelegationToken
          org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
          org.apache.hadoop.hdfs.tools.TestGetGroups
          org.apache.hadoop.hdfs.TestPipelines
          org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser
          org.apache.hadoop.hdfs.TestBlocksScheduledCounter
          org.apache.hadoop.hdfs.TestHDFSFileSystemContract
          org.apache.hadoop.hdfs.TestDFSClientRetries
          org.apache.hadoop.hdfs.TestAppendDifferentChecksum
          org.apache.hadoop.hdfs.TestReadWhileWriting

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1864//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1864//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514218/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.TestParallelRead org.apache.hadoop.hdfs.TestBlockMissingException org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestReplication org.apache.hadoop.hdfs.TestClientProtocolForPipelineRecovery org.apache.hadoop.hdfs.TestMissingBlocksAlert org.apache.hadoop.hdfs.TestFileStatus org.apache.hadoop.hdfs.TestDFSClientExcludedNodes org.apache.hadoop.hdfs.TestMultiThreadedHflush org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestWriteRead org.apache.hadoop.hdfs.TestLeaseRecovery org.apache.hadoop.hdfs.TestDFSMkdirs org.apache.hadoop.hdfs.TestHDFSTrash org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestQuota org.apache.hadoop.hdfs.security.token.block.TestBlockToken org.apache.hadoop.hdfs.TestFileAppend2 org.apache.hadoop.hdfs.TestListFilesInDFS org.apache.hadoop.hdfs.TestFileAppend3 org.apache.hadoop.hdfs.TestReplaceDatanodeOnFailure org.apache.hadoop.hdfs.TestDecommission org.apache.hadoop.hdfs.TestFileCorruption org.apache.hadoop.hdfs.TestFileCreationEmpty org.apache.hadoop.hdfs.TestSetTimes org.apache.hadoop.hdfs.TestBalancerBandwidth org.apache.hadoop.hdfs.TestFileCreationClient org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.hdfs.security.TestDelegationToken org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract org.apache.hadoop.hdfs.tools.TestGetGroups org.apache.hadoop.hdfs.TestPipelines org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser org.apache.hadoop.hdfs.TestBlocksScheduledCounter org.apache.hadoop.hdfs.TestHDFSFileSystemContract org.apache.hadoop.hdfs.TestDFSClientRetries org.apache.hadoop.hdfs.TestAppendDifferentChecksum org.apache.hadoop.hdfs.TestReadWhileWriting +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1864//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1864//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Patch addresses Eli's comments.

          Show
          Harsh J added a comment - Patch addresses Eli's comments.
          Hide
          Eli Collins added a comment -

          I don't think we want to wait inside the client, just expose the API so a user/management tool can see if the NN is in safemode (the tool can wait and re-query if it wants).

          Show
          Eli Collins added a comment - I don't think we want to wait inside the client, just expose the API so a user/management tool can see if the NN is in safemode (the tool can wait and re-query if it wants).
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12508636/HDFS-2413.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 20 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings).

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12508636/HDFS-2413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 20 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1739//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Patch that makes DFSAdmin carry two utility functions that lets its users check safemode and wait upon it.

          Show
          Harsh J added a comment - Patch that makes DFSAdmin carry two utility functions that lets its users check safemode and wait upon it.
          Hide
          Harsh J added a comment -

          Uma Maheswara Rao G - You may want to report that downstream as well – I believe they can surely catch SafeModeExceptions or so, and do better?

          [~steve@apache.org] - The JMX output at /jmx carries this state today. Is that insufficient? I'll consider this inclusion as well, after your reply.

          Show
          Harsh J added a comment - Uma Maheswara Rao G - You may want to report that downstream as well – I believe they can surely catch SafeModeExceptions or so, and do better? [~steve@apache.org] - The JMX output at /jmx carries this state today. Is that insufficient? I'll consider this inclusion as well, after your reply.
          Hide
          Steve Loughran added a comment -

          It'd be nice to have this served up over HTTP too, some URL like /islive that returns 503 if not; the #of live node if the system is up. This makes it trivial for management tools and things like curl to check the system

          Show
          Steve Loughran added a comment - It'd be nice to have this served up over HTTP too, some URL like /islive that returns 503 if not; the #of live node if the system is up. This makes it trivial for management tools and things like curl to check the system
          Hide
          Uma Maheswara Rao G added a comment -

          Harsh, This will really help in other systems also as you mention above.
          I remember one case in HIVE also, when starting the hive it needs to creates some table in HDFS and derby related to metastore. If NN is in safe mode, it used to fail the creatione of tables in dfs and succeeds in derby. Because of this partial table table creations, it shows metastore as currupt and fails the startup.

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - Harsh, This will really help in other systems also as you mention above. I remember one case in HIVE also, when starting the hive it needs to creates some table in HDFS and derby related to metastore. If NN is in safe mode, it used to fail the creatione of tables in dfs and succeeds in derby. Because of this partial table table creations, it shows metastore as currupt and fails the startup. Thanks Uma
          Hide
          Harsh J added a comment -

          (One thing that may benefit in the interim is HBASE-4510)

          Show
          Harsh J added a comment - (One thing that may benefit in the interim is HBASE-4510 )
          Hide
          Harsh J added a comment -

          I like Srivas's idea, but its gonna be a behavior change at the moment. I think we should pursue this on another ticket.

          The remaining question for this issue is if we should add the API to a new class (say, 'HDFSAdmin', since DFSAdmin seems limited for shell usage and is also marked for private audience), or if we should support the whole of DFSAdmin with a new addition, publicly?

          Show
          Harsh J added a comment - I like Srivas's idea, but its gonna be a behavior change at the moment. I think we should pursue this on another ticket. The remaining question for this issue is if we should add the API to a new class (say, 'HDFSAdmin', since DFSAdmin seems limited for shell usage and is also marked for private audience), or if we should support the whole of DFSAdmin with a new addition, publicly?
          Hide
          Uma Maheswara Rao G added a comment -

          I think we can expose below API.
          /** Is the system in safe mode? */
          public boolean isInSafeMode();

          So that, the depending systems can check this API to know whether NN is out of safemode or not.

          Todd, is this ok for you? or expecting some other APIs as well?

          Thanks
          Uma

          Show
          Uma Maheswara Rao G added a comment - I think we can expose below API. /** Is the system in safe mode? */ public boolean isInSafeMode(); So that, the depending systems can check this API to know whether NN is out of safemode or not. Todd, is this ok for you? or expecting some other APIs as well? Thanks Uma
          Hide
          M. C. Srivas added a comment -

          This should be normal behavior part of all file-system ops. It is not practical for a programmer to wrap all file access calls (eg, write, mkdir, open) with "wait for NN to leave safemode".

          Show
          M. C. Srivas added a comment - This should be normal behavior part of all file-system ops. It is not practical for a programmer to wrap all file access calls (eg, write, mkdir, open) with "wait for NN to leave safemode".

            People

            • Assignee:
              Harsh J
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development