Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-3084

FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.24.0, 0.23.3
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha
    • Labels:
      None

      Description

      The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs.

      The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId.

      1. hdfs-3084.txt
        10 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          This patch, in conjunction with HADOOP-8007, provides the necessary facilities:

          • the configuration used to instantiate the NodeFencer is now the configuration of the target node, rather than the fencing node. This means that all of the configs like RPC address have already had the nameservice and namenodeid chopped off
          • in addition to the variables described in HADOOP-8007, we add $target_namenodeid and $target_nameserviceid to the fencing environment
          • added getters for namenode id and nameservice id to NNHAServiceTarget. If a custom fencing implementation is HDFS specific, it can downcast to the implementation class to access these getters. Alternatively, it may call getFencingParameters() and then use the resulting string map to get at this info.

          The patch depends on HADOOP-8007. I'll submit it to QA once that other patch is committed.

          Show
          Todd Lipcon added a comment - This patch, in conjunction with HADOOP-8007 , provides the necessary facilities: the configuration used to instantiate the NodeFencer is now the configuration of the target node, rather than the fencing node. This means that all of the configs like RPC address have already had the nameservice and namenodeid chopped off in addition to the variables described in HADOOP-8007 , we add $target_namenodeid and $target_nameserviceid to the fencing environment added getters for namenode id and nameservice id to NNHAServiceTarget. If a custom fencing implementation is HDFS specific, it can downcast to the implementation class to access these getters. Alternatively, it may call getFencingParameters() and then use the resulting string map to get at this info. The patch depends on HADOOP-8007 . I'll submit it to QA once that other patch is committed.
          Hide
          Eli Collins added a comment -

          +1 looks good

          Show
          Eli Collins added a comment - +1 looks good
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. Good stuff, Todd.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. Good stuff, Todd.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521282/hdfs-3084.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521282/hdfs-3084.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I haven't been following all the fencing jiras, so qq, are tokens and/or their services involved?

          Show
          Daryn Sharp added a comment - I haven't been following all the fencing jiras, so qq, are tokens and/or their services involved?
          Hide
          Todd Lipcon added a comment -

          No, the scripts and failover controllers use keytab-based or straight user credentials.

          Show
          Todd Lipcon added a comment - No, the scripts and failover controllers use keytab-based or straight user credentials.
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 and trunk, thanks for reviews.

          Show
          Todd Lipcon added a comment - Committed to branch-2 and trunk, thanks for reviews.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2070 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2070/)
          HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2070 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2070/ ) HDFS-3084 . FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1995 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1995/)
          HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1995 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1995/ ) HDFS-3084 . FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2007 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2007/)
          HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2007 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2007/ ) HDFS-3084 . FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/)
          HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1006 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1006/ ) HDFS-3084 . FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/)
          HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1041 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1041/ ) HDFS-3084 . FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309526 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Philip Zeyliger
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development