Details
-
Type:
Improvement
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 2.0.0-alpha
-
Fix Version/s: 2.0.0-alpha
-
Component/s: ha
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
Per HADOOP-7983 currently the fencer always passes the target host:port to fence as the first argument to the fence script, it would be better to use a substitution token. That is to say, the user would configure "myfence.sh $TARGETHOST foo bar" and Hadoop would substitute the target. This would allow use of pre-existing scripts that might have a different ordering of arguments without a wrapper.
Issue Links
- is related to
-
HDFS-3084
FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
-
- Resolved
-
- relates to
-
MAPREDUCE-4103
Fix HA docs for changes to shell command fencer args
-
- Closed
-
Activity
| Parent |
|
|
| Reporter | Eli Collins [ eli ] | Aaron T. Myers [ atm ] |
| Issue Type | Sub-task [ 7 ] | Improvement [ 4 ] |
| Assignee | Eli Collins [ eli ] | Todd Lipcon [ tlipcon ] |
Attached patch changes the configuration of the fencing script as follows:
By default, no arguments are passed to the script. This is different than the previous implementation, but since HA hasn't been in a release yet, I think it's fine to break this.
Instead, users are expected to use the environment to choose whatever arguments they need. Since we pass the shell command through to bash, environment variables are interpolated.
We already handled passing the configuration through. This patch also adds the ability for an HAServiceTarget to add additional key/value pairs, which are prefixed with target_ in the environment. The three passed by the Common code are $target_host, $target_port, and $target_address (which is the same as host:port).
HDFS-3084 will add the NNID and NSID to this map.
I will also file a separate JIRA to update the docs, which unfortunately live in the MapReduce project.
| Attachment | hadoop-8007.txt [ 12521280 ] |
| Status | Open [ 1 ] | Patch Available [ 10002 ] |
| Affects Version/s | 2.0.0 [ 12320352 ] | |
| Affects Version/s | 0.24.0 [ 12317652 ] | |
| Target Version/s | 0.24.0 [ 12317652 ] | 2.0.0 [ 12320352 ] |
+1, the patch looks good to me. Thanks a lot for doing this, Todd.
Please do file the separate JIRA for the docs, and I'll be happy to review those as well.
Oh, one little thing: please add a method comment to HAServiceTarget#addFencingParameters. Subclass implementers will appreciate it.
+1 looks great
Updated patch adds the javadoc requested above. I filed MAPREDUCE-4103 to update the docs.
| Attachment | hadoop-8007.txt [ 12521287 ] |
| Link |
This issue relates to |
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521287/hadoop-8007.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.fs.viewfs.TestViewFsTrash
org.apache.hadoop.ha.TestZKFailoverController
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/818//console
This message is automatically generated.
org.apache.hadoop.ha.TestZKFailoverController
This failure was the JMXEnv issue tracked in HADOOP-8245.
I will commit this momentarily
Committed to trunk and branch-2. I am not marking this as Incompatible, since the previous incarnation was not in any release.
| Status | Patch Available [ 10002 ] | Resolved [ 5 ] |
| Hadoop Flags | Reviewed [ 10343 ] | |
| Fix Version/s | 2.0.0 [ 12320352 ] | |
| Resolution | Fixed [ 1 ] |
Integrated in Hadoop-Common-trunk-Commit #1988 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1988/)
HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
Integrated in Hadoop-Hdfs-trunk-Commit #2063 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2063/)
HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
Integrated in Hadoop-Mapreduce-trunk-Commit #2001 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2001/)
HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
Integrated in Hadoop-Hdfs-trunk #1005 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/)
HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
Integrated in Hadoop-Mapreduce-trunk #1040 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/)
HADOOP-8007. Use substitution tokens for fencing argument. Contributed by Todd Lipcon. (Revision 1309285)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309285
Files :
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceTarget.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java
- /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestShellCommandFencer.java
| Status | Resolved [ 5 ] | Closed [ 6 ] |
Converting to top-level issue with commit of
HADOOP-7454.