[HADOOP-13341] Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha2
• Component/s: scripts
• Target Version/s: 3.0.0-alpha2
• Hadoop Flags: Incompatible change

Release Note

<!-- markdown -->
Users:
* Ability to set per-command+sub-command options from the command line.
* Makes daemon environment variable options consistent across the project. (See deprecation list below)
* HADOOP\_CLIENT\_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it.

Developers:
* No longer need to do custom handling for options in the case section of the shell scripts.
* Consolidates all \_OPTS handling into hadoop-functions.sh to enable future projects.
* All daemons running with secure mode features now get \_SECURE\_EXTRA\_OPTS support.

\_OPTS Changes:

| Old | New |
|:---- |:---- |
| HADOOP\_BALANCER\_OPTS | HDFS\_BALANCER\_OPTS |
| HADOOP\_DATANODE\_OPTS | HDFS\_DATANODE\_OPTS |
| HADOOP\_DN\_SECURE_EXTRA_OPTS | HDFS\_DATANODE\_SECURE\_EXTRA\_OPTS |
| HADOOP\_JOB\_HISTORYSERVER\_OPTS | MAPRED\_HISTORYSERVER\_OPTS |
| HADOOP\_JOURNALNODE\_OPTS | HDFS\_JOURNALNODE\_OPTS |
| HADOOP\_MOVER\_OPTS | HDFS\_MOVER\_OPTS |
| HADOOP\_NAMENODE\_OPTS | HDFS\_NAMENODE\_OPTS |
| HADOOP\_NFS3\_OPTS | HDFS\_NFS3\_OPTS |
| HADOOP\_NFS3\_SECURE\_EXTRA\_OPTS | HDFS\_NFS3\_SECURE\_EXTRA\_OPTS |
| HADOOP\_PORTMAP\_OPTS | HDFS\_PORTMAP\_OPTS |
| HADOOP\_SECONDARYNAMENODE\_OPTS | HDFS\_SECONDARYNAMENODE\_OPTS |
| HADOOP\_ZKFC\_OPTS | HDFS\_ZKFC\_OPTS |

• Assignee: Allen Wittenauer
• Reporter: Allen Wittenauer
• Created: 05/Jul/16 06:22
• Updated: 22/Jun/17 19:45
• Resolved: 12/Sep/16 18:20

Description

Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPTS env var. We can replace all of the custom vars with generic handling just like we do for the username check.

For example, with generic handling in place:

Old Var	New Var
HADOOP_NAMENODE_OPTS	HDFS_NAMENODE_OPTS
YARN_RESOURCEMANAGER_OPTS	YARN_RESOURCEMANAGER_OPTS
n/a	YARN_TIMELINEREADER_OPTS
n/a	HADOOP_DISTCP_OPTS
n/a	MAPRED_DISTCP_OPTS
HADOOP_DN_SECURE_EXTRA_OPTS	HDFS_DATANODE_SECURE_EXTRA_OPTS
HADOOP_NFS3_SECURE_EXTRA_OPTS	HDFS_NFS3_SECURE_EXTRA_OPTS
HADOOP_JOB_HISTORYSERVER_OPTS	MAPRED_HISTORYSERVER_OPTS

This makes it:

a) consistent across the entire project
b) consistent for every subcommand
c) eliminates almost all of the custom appending in the case statements

It's worth pointing out that subcommands like distcp that sometimes need a higher than normal client-side heapsize or custom options are a huge win. Combined with .hadooprc and/or dynamic subcommands, it means users can easily do customizations based upon their needs without a lot of weirdo shell aliasing or one line shell scripts off to the side.

Issue Links

• breaks: MAPREDUCE-6904 HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh
• is required by: HADOOP-13365 Convert _OPTS to arrays to enable spaces in file paths
• relates to: HADOOP-13595 Rework hadoop_usage to be broken up by clients/daemons/etc.
• links to: GitHub Pull Request #126

Sub-Tasks

• Handle HADOOP_CLIENT_OPTS in a function: Resolved
• Add a function to handle command_subcommand_OPTS: Resolved
• Modify common to use hadoop_subcommand_opts: Resolved
• Modify HDFS to use hadoop_subcommand_opts: Resolved
• Modify YARN to use hadoop_subcommand_opts: Resolved
• Documentation for HADOOP_subcommand_OPTS: Resolved
• Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity): Resolved
• Add an equivalent of hadoop_subcmd_opts for secure opts: Resolved
• Change hadoop_subcommand_opts to use only uppercase: Resolved
• hadoop_subcommand_opts should print name not actual content during debug: Resolved
• modify mapred to use hadoop_subcommand_opts: Resolved