Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8464

hadoop-env.sh is included twice: once via hadoop-config.sh the again explicitly via scripts

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: conf
    • Labels:
      None

      Description

      Many (most? all?) scripts include hadoop-env.sh twice. The first time is indirectly: scripts do ". hadoop-config.sh" and hadoop-config.sh then does ". hadoop-env.sh". Then scripts will later directly do ". hadoop-env.sh".

      This double-include is error prone.

      Note several Jira's reporting issues with HADOOP_LOG_DIR set to /user/user (e.g. /root/root, /hadoop/hadop, etc). This is because hadoop-env.sh has done "HADOOP_LOG_DIR=$HADOOP_LOG_DIR/$user". Many fixes for those issues are around addressing that one line, but it seems that this is going to be a recurring theme of breakages. It's probably also why services are routinely started with many command line arguments repeated multiple times.

      I'm not familiar enough with the "design" of all the scripts, so not sure what the solution should be.

        Issue Links

          Activity

          Hide
          vinayrpet Vinayakumar B added a comment -

          Ideally calling hadoop-env.sh multiple times should not create any problems.

          Because hadoop-config.sh itself called from so many files. Ex: hadoop-daemon.sh, slaves.sh, hdfs, yarn, etc.

          We can remove the explicit call of hadoop-env.sh from scripts other than hadoop-config.sh

          HADOOP-8476 will resolve all the issues.

          Show
          vinayrpet Vinayakumar B added a comment - Ideally calling hadoop-env.sh multiple times should not create any problems. Because hadoop-config.sh itself called from so many files. Ex: hadoop-daemon.sh, slaves.sh, hdfs, yarn, etc. We can remove the explicit call of hadoop-env.sh from scripts other than hadoop-config.sh HADOOP-8476 will resolve all the issues.
          Hide
          vinayrpet Vinayakumar B added a comment -

          And one more thing is, variables configured in hadoop-env.sh should not append to its own initial value. It should be absolute. Then calling multiple times will not be a problem at all.

          Show
          vinayrpet Vinayakumar B added a comment - And one more thing is, variables configured in hadoop-env.sh should not append to its own initial value. It should be absolute. Then calling multiple times will not be a problem at all.
          Hide
          daryn Daryn Sharp added a comment -

          Won't we lose the ability to override values if it does an absolute set? I'd think the script should only set the value if unset. Ie. something like:

          ${KEY:=<default value or expression>}

          I agree that technically it would be safe to include hadoop-env multiple times although it would be good to avoid redundant and unnecessary includes where possible.

          Show
          daryn Daryn Sharp added a comment - Won't we lose the ability to override values if it does an absolute set? I'd think the script should only set the value if unset. Ie. something like: ${KEY:=< default value or expression>} I agree that technically it would be safe to include hadoop-env multiple times although it would be good to avoid redundant and unnecessary includes where possible.
          Hide
          aw Allen Wittenauer added a comment -

          Duping this to HADOOP-9902 which fixed this.

          Show
          aw Allen Wittenauer added a comment - Duping this to HADOOP-9902 which fixed this.

            People

            • Assignee:
              Unassigned
              Reporter:
              jayf Jay Finger
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development