Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3408

yarn-daemon.sh unconditionnaly sets yarn.root.logger

    Details

    • Hadoop Flags:
      Reviewed

      Description

      yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any override from happening.
      From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
      > export YARN_ROOT_LOGGER="INFO,DRFA"
      > export YARN_JHS_LOGGER="INFO,JSA"

      and then yarn-daemon.sh will call "$YARN_HOME"/bin/yarn which does the following:
      > YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=$

      {YARN_ROOT_LOGGER:-INFO,console}

      "
      > YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=$

      {YARN_ROOT_LOGGER:-INFO,console}

      "

      This has at least 2 issues:

      • I cannot override hadoop.root.logger when using the yarn-daemon.sh script
      • I cannot have different values for hadoop.root.logger and yarn.root.logger

      I currently see two different ways to proceed forward:
      1/ Make the script yarn-daemon.sh only sets a default value for YARN_ROOT_LOGGER if this variable is not defined
      2/ Remove the quoted code from yarn-daemon.sh since yarn already does something similar
      3/ Entirely remove that chunk and let people define their logging however they want through some properties files (see log4j.properties in the conf directories for instance)

      I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if either option 1/ or 2/ would be taken.

      I don't really have any preference toward any of these solutions. What would you recommend? What is the Apache Hadoop way for this matter?

      Note: This is probably happening as well for the other daemons, and I will take a look at it once this issue is resolved.

        Activity

        Hide
        Bruno Mahé added a comment -

        IMHO option 2/ is the safest and most conservative approach.
        I will prepare a patch tomorrow.
        Please let me know if you think of a better way.

        Show
        Bruno Mahé added a comment - IMHO option 2/ is the safest and most conservative approach. I will prepare a patch tomorrow. Please let me know if you think of a better way.
        Hide
        Bruno Mahé added a comment -

        I changed my mind and went for option 1/

        Show
        Bruno Mahé added a comment - I changed my mind and went for option 1/
        Hide
        Bruno Mahé added a comment -

        Please find patch attached

        Show
        Bruno Mahé added a comment - Please find patch attached
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12504258/MAPREDUCE-3408.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests:
        org.apache.hadoop.mapreduce.v2.TestMRJobs
        org.apache.hadoop.mapred.TestMiniMRClientCluster
        org.apache.hadoop.mapred.TestClientRedirect
        org.apache.hadoop.mapreduce.v2.TestUberAM

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1322//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1322//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12504258/MAPREDUCE-3408.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapreduce.v2.TestMRJobs org.apache.hadoop.mapred.TestMiniMRClientCluster org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.v2.TestUberAM +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1322//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1322//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        Looks good. Just committed this to trunk and 0.23. Thanks Bruno.

        Show
        Mahadev konar added a comment - Looks good. Just committed this to trunk and 0.23. Thanks Bruno.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Commit #203 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/203/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #203 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/203/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-0.23-Commit #206 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/206/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #206 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/206/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1388 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1388/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1388 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1388/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1315/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1315 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1315/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Commit #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/217/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #217 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/217/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1338 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1338/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1338 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1338/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #85 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/85/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #85 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/85/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #872 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/872/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #872 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/872/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #102 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/102/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #102 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/102/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) - Merging r1205200 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205201 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #906 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/906/)
        MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev)

        mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #906 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/906/ ) MAPREDUCE-3408 . yarn-daemon.sh unconditionnaly sets yarn.root.logger (Bruno Mahe via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1205200 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh

          People

          • Assignee:
            Bruno Mahé
            Reporter:
            Bruno Mahé
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development