Hadoop Common
  1. Hadoop Common
  2. HADOOP-8007

HA: use substitution token for fencing argument

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha
    • Labels:
      None

      Description

      Per HADOOP-7983 currently the fencer always passes the target host:port to fence as the first argument to the fence script, it would be better to use a substitution token. That is to say, the user would configure "myfence.sh $TARGETHOST foo bar" and Hadoop would substitute the target. This would allow use of pre-existing scripts that might have a different ordering of arguments without a wrapper.

      1. hadoop-8007.txt
        7 kB
        Todd Lipcon
      2. hadoop-8007.txt
        7 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Aaron T. Myers added a comment -

          Converting to top-level issue with commit of HADOOP-7454.

          Show
          Aaron T. Myers added a comment - Converting to top-level issue with commit of HADOOP-7454 .
          Hide
          Todd Lipcon added a comment -

          Attached patch changes the configuration of the fencing script as follows:

          By default, no arguments are passed to the script. This is different than the previous implementation, but since HA hasn't been in a release yet, I think it's fine to break this.

          Instead, users are expected to use the environment to choose whatever arguments they need. Since we pass the shell command through to bash, environment variables are interpolated.

          We already handled passing the configuration through. This patch also adds the ability for an HAServiceTarget to add additional key/value pairs, which are prefixed with target_ in the environment. The three passed by the Common code are $target_host, $target_port, and $target_address (which is the same as host:port).

          HDFS-3084 will add the NNID and NSID to this map.

          I will also file a separate JIRA to update the docs, which unfortunately live in the MapReduce project.

          Show
          Todd Lipcon added a comment - Attached patch changes the configuration of the fencing script as follows: By default, no arguments are passed to the script. This is different than the previous implementation, but since HA hasn't been in a release yet, I think it's fine to break this. Instead, users are expected to use the environment to choose whatever arguments they need. Since we pass the shell command through to bash, environment variables are interpolated. We already handled passing the configuration through. This patch also adds the ability for an HAServiceTarget to add additional key/value pairs, which are prefixed with target_ in the environment. The three passed by the Common code are $target_host , $target_port , and $target_address (which is the same as host:port). HDFS-3084 will add the NNID and NSID to this map. I will also file a separate JIRA to update the docs, which unfortunately live in the MapReduce project.
          Hide
          Aaron T. Myers added a comment -

          +1, the patch looks good to me. Thanks a lot for doing this, Todd.

          Please do file the separate JIRA for the docs, and I'll be happy to review those as well.

          Show
          Aaron T. Myers added a comment - +1, the patch looks good to me. Thanks a lot for doing this, Todd. Please do file the separate JIRA for the docs, and I'll be happy to review those as well.
          Hide
          Aaron T. Myers added a comment -

          Oh, one little thing: please add a method comment to HAServiceTarget#addFencingParameters. Subclass implementers will appreciate it.

          Show
          Aaron T. Myers added a comment - Oh, one little thing: please add a method comment to HAServiceTarget#addFencingParameters. Subclass implementers will appreciate it.
          Hide
          Eli Collins added a comment -

          +1 looks great

          Show
          Eli Collins added a comment - +1 looks great
          Hide
          Todd Lipcon added a comment -

          Updated patch adds the javadoc requested above. I filed MAPREDUCE-4103 to update the docs.

          Show
          Todd Lipcon added a comment - Updated patch adds the javadoc requested above. I filed MAPREDUCE-4103 to update the docs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521287/hadoop-8007.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests:
          org.apache.hadoop.fs.viewfs.TestViewFsTrash
          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521287/hadoop-8007.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.viewfs.TestViewFsTrash org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          org.apache.hadoop.ha.TestZKFailoverController

          This failure was the JMXEnv issue tracked in HADOOP-8245.

          I will commit this momentarily

          Show
          Todd Lipcon added a comment - org.apache.hadoop.ha.TestZKFailoverController This failure was the JMXEnv issue tracked in HADOOP-8245 . I will commit this momentarily
          Hide
          Todd Lipcon added a comment -

          Committed to trunk and branch-2. I am not marking this as Incompatible, since the previous incarnation was not in any release.

          Show
          Todd Lipcon added a comment - Committed to trunk and branch-2. I am not marking this as Incompatible, since the previous incarnation was not in any release.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1988 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1988/)
          HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1988 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1988/ ) HADOOP-8007 . Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2063 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2063/)
          HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2063 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2063/ ) HADOOP-8007 . Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2001/)
          HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2001/ ) HADOOP-8007 . Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/)
          HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/ ) HADOOP-8007 . Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/)
          HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/ ) HADOOP-8007 . Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Aaron T. Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development