Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13341

Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: scripts
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      <!-- markdown -->
      Users:
      * Ability to set per-command+sub-command options from the command line.
      * Makes daemon environment variable options consistent across the project. (See deprecation list below)
      * HADOOP\_CLIENT\_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it.

      Developers:
      * No longer need to do custom handling for options in the case section of the shell scripts.
      * Consolidates all \_OPTS handling into hadoop-functions.sh to enable future projects.
      * All daemons running with secure mode features now get \_SECURE\_EXTRA\_OPTS support.

      \_OPTS Changes:

      | Old | New |
      |:---- |:---- |
      | HADOOP\_BALANCER\_OPTS | HDFS\_BALANCER\_OPTS |
      | HADOOP\_DATANODE\_OPTS | HDFS\_DATANODE\_OPTS |
      | HADOOP\_DN\_SECURE_EXTRA_OPTS | HDFS\_DATANODE\_SECURE\_EXTRA\_OPTS |
      | HADOOP\_JOB\_HISTORYSERVER\_OPTS | MAPRED\_HISTORYSERVER\_OPTS |
      | HADOOP\_JOURNALNODE\_OPTS | HDFS\_JOURNALNODE\_OPTS |
      | HADOOP\_MOVER\_OPTS | HDFS\_MOVER\_OPTS |
      | HADOOP\_NAMENODE\_OPTS | HDFS\_NAMENODE\_OPTS |
      | HADOOP\_NFS3\_OPTS | HDFS\_NFS3\_OPTS |
      | HADOOP\_NFS3\_SECURE\_EXTRA\_OPTS | HDFS\_NFS3\_SECURE\_EXTRA\_OPTS |
      | HADOOP\_PORTMAP\_OPTS | HDFS\_PORTMAP\_OPTS |
      | HADOOP\_SECONDARYNAMENODE\_OPTS | HDFS\_SECONDARYNAMENODE\_OPTS |
      | HADOOP\_ZKFC\_OPTS | HDFS\_ZKFC\_OPTS |
      Show
      <!-- markdown --> Users: * Ability to set per-command+sub-command options from the command line. * Makes daemon environment variable options consistent across the project. (See deprecation list below) * HADOOP\_CLIENT\_OPTS is now honored for every non-daemon sub-command. Prior to this change, many sub-commands did not use it. Developers: * No longer need to do custom handling for options in the case section of the shell scripts. * Consolidates all \_OPTS handling into hadoop-functions.sh to enable future projects. * All daemons running with secure mode features now get \_SECURE\_EXTRA\_OPTS support. \_OPTS Changes: | Old | New | |:---- |:---- | | HADOOP\_BALANCER\_OPTS | HDFS\_BALANCER\_OPTS | | HADOOP\_DATANODE\_OPTS | HDFS\_DATANODE\_OPTS | | HADOOP\_DN\_SECURE_EXTRA_OPTS | HDFS\_DATANODE\_SECURE\_EXTRA\_OPTS | | HADOOP\_JOB\_HISTORYSERVER\_OPTS | MAPRED\_HISTORYSERVER\_OPTS | | HADOOP\_JOURNALNODE\_OPTS | HDFS\_JOURNALNODE\_OPTS | | HADOOP\_MOVER\_OPTS | HDFS\_MOVER\_OPTS | | HADOOP\_NAMENODE\_OPTS | HDFS\_NAMENODE\_OPTS | | HADOOP\_NFS3\_OPTS | HDFS\_NFS3\_OPTS | | HADOOP\_NFS3\_SECURE\_EXTRA\_OPTS | HDFS\_NFS3\_SECURE\_EXTRA\_OPTS | | HADOOP\_PORTMAP\_OPTS | HDFS\_PORTMAP\_OPTS | | HADOOP\_SECONDARYNAMENODE\_OPTS | HDFS\_SECONDARYNAMENODE\_OPTS | | HADOOP\_ZKFC\_OPTS | HDFS\_ZKFC\_OPTS |

      Description

      Big features like YARN-2928 demonstrate that even senior level Hadoop developers forget that daemons need a custom _OPTS env var. We can replace all of the custom vars with generic handling just like we do for the username check.

      For example, with generic handling in place:

      Old Var New Var
      HADOOP_NAMENODE_OPTS HDFS_NAMENODE_OPTS
      YARN_RESOURCEMANAGER_OPTS YARN_RESOURCEMANAGER_OPTS
      n/a YARN_TIMELINEREADER_OPTS
      n/a HADOOP_DISTCP_OPTS
      n/a MAPRED_DISTCP_OPTS
      HADOOP_DN_SECURE_EXTRA_OPTS HDFS_DATANODE_SECURE_EXTRA_OPTS
      HADOOP_NFS3_SECURE_EXTRA_OPTS HDFS_NFS3_SECURE_EXTRA_OPTS
      HADOOP_JOB_HISTORYSERVER_OPTS MAPRED_HISTORYSERVER_OPTS

      This makes it:

      a) consistent across the entire project
      b) consistent for every subcommand
      c) eliminates almost all of the custom appending in the case statements

      It's worth pointing out that subcommands like distcp that sometimes need a higher than normal client-side heapsize or custom options are a huge win. Combined with .hadooprc and/or dynamic subcommands, it means users can easily do customizations based upon their needs without a lot of weirdo shell aliasing or one line shell scripts off to the side.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                aw Allen Wittenauer
                Reporter:
                aw Allen Wittenauer
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: