Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11022

User replaced functions get lost 2-3 levels deep (e.g., sbin)

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: scripts
    • Labels:
      None

      Description

      The code that protects hadoop-env.sh from being re-executed is also causing functions that the user replaced to get overridden with the defaults. This typically happens when running commands that nest, such as most of the content in sbin. Just running stuff out of bin (e.g., bin/hdfs --daemon start namenode) does not trigger this behavior.

      1. HADOOP-11022.patch
        9 kB
        Allen Wittenauer

        Issue Links

          Activity

          Hide
          aw Allen Wittenauer added a comment - - edited

          The basic problem is the export of HADOOP_ENV_PROCESSED. This is used to prevent hadoop-env.sh from being re-read. This protection makes sense if you have, e.g., FOO_OPT="$FOO_OPTS blah" in this file. But with the current state of the world, there's quite a few things going wrong here:

          1. We don't protect the other *-env.sh files like we do hadoop-env.sh. i.e., yarn-env.sh will gets re-read multiple times.
          2. This only works correctly because the vars in hadoop-env.sh are also exported.... which is probably the wrong thing to do as well.
          3. Now that there are more than just vars in *-env.sh, the system really does need to re-read *-env.sh to pick those up.

          Some potential solutions worth exploring:

          • What happens if we no longer export anything in *-env.sh and re-read them at every command run like we do with the other files? (This brings Hadoop in line with most other OS utilities.) One side effect: users upgrading to trunk would need to also remove the export lines as part of upgrading.
          • If we protect all the *-env.sh and move user defined functions to a new file? This has a lot of potential! One big problem is what to do about YARN_OPTS.
          Show
          aw Allen Wittenauer added a comment - - edited The basic problem is the export of HADOOP_ENV_PROCESSED. This is used to prevent hadoop-env.sh from being re-read. This protection makes sense if you have, e.g., FOO_OPT="$FOO_OPTS blah" in this file. But with the current state of the world, there's quite a few things going wrong here: 1. We don't protect the other *-env.sh files like we do hadoop-env.sh. i.e., yarn-env.sh will gets re-read multiple times. 2. This only works correctly because the vars in hadoop-env.sh are also exported.... which is probably the wrong thing to do as well. 3. Now that there are more than just vars in *-env.sh, the system really does need to re-read *-env.sh to pick those up. Some potential solutions worth exploring: What happens if we no longer export anything in *-env.sh and re-read them at every command run like we do with the other files? (This brings Hadoop in line with most other OS utilities.) One side effect: users upgrading to trunk would need to also remove the export lines as part of upgrading. If we protect all the *-env.sh and move user defined functions to a new file? This has a lot of potential! One big problem is what to do about YARN_OPTS.
          Hide
          aw Allen Wittenauer added a comment -

          This patch implements the last bullet point:

          a) Protects all of the *-env.sh files
          b) Moves user defined functions into a new hadoop-user-functions.sh script
          c) Moves the examples from hadoop-env.sh to hadoop-user-functions.sh.example
          d) Fixes the log rotation example (HADOOP-11023)

          The gotcha here is that this assumes that the *-env.sh vars are exported! This is bw compat with previous versions of Hadoop. This isn't ideal, but...

          Show
          aw Allen Wittenauer added a comment - This patch implements the last bullet point: a) Protects all of the *-env.sh files b) Moves user defined functions into a new hadoop-user-functions.sh script c) Moves the examples from hadoop-env.sh to hadoop-user-functions.sh.example d) Fixes the log rotation example ( HADOOP-11023 ) The gotcha here is that this assumes that the *-env.sh vars are exported! This is bw compat with previous versions of Hadoop. This isn't ideal, but...
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12665209/HADOOP-11022.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4579//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665209/HADOOP-11022.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4579//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Cancelling this patch since the example rotate function is incorrect.

          Show
          aw Allen Wittenauer added a comment - Cancelling this patch since the example rotate function is incorrect.
          Hide
          aw Allen Wittenauer added a comment -

          Argh, no the patch is fine. NM. not awake yet.

          Show
          aw Allen Wittenauer added a comment - Argh, no the patch is fine. NM. not awake yet.
          Hide
          aw Allen Wittenauer added a comment -

          Note that HADOOP-10950 will require more changes.

          Show
          aw Allen Wittenauer added a comment - Note that HADOOP-10950 will require more changes.
          Hide
          aw Allen Wittenauer added a comment -

          Self +1 this one, after having a discussion with another committer about the changes (esp since I'm adding another config file)

          Show
          aw Allen Wittenauer added a comment - Self +1 this one, after having a discussion with another committer about the changes (esp since I'm adding another config file)
          Hide
          aw Allen Wittenauer added a comment -

          Committed to trunk.

          Show
          aw Allen Wittenauer added a comment - Committed to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #683 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/683/)
          HADOOP-11022. User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-mapreduce-project/bin/mapred-config.sh
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #683 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/683/ ) HADOOP-11022 . User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea) hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh hadoop-common-project/hadoop-common/CHANGES.txt hadoop-mapreduce-project/bin/mapred-config.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1899 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1899/)
          HADOOP-11022. User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • hadoop-mapreduce-project/bin/mapred-config.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1899 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1899/ ) HADOOP-11022 . User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh hadoop-mapreduce-project/bin/mapred-config.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1874 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/)
          HADOOP-11022. User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-mapreduce-project/bin/mapred-config.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
          • hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh
          • hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example
          • hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh
          • hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1874 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1874/ ) HADOOP-11022 . User replaced functions get lost 2-3 levels deep (e.g., sbin) (aw) (aw: rev 7971c97ec18f4fc6504f6a87d9ad02a1f5f01eea) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-mapreduce-project/bin/mapred-config.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-env.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh hadoop-yarn-project/hadoop-yarn/bin/yarn-config.sh hadoop-common-project/hadoop-common/src/main/conf/hadoop-user-functions.sh.example hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs-config.sh hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh

            People

            • Assignee:
              aw Allen Wittenauer
              Reporter:
              aw Allen Wittenauer
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development