Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2537

MR-279: The RM writes its log to yarn-mapred-resourcemanager-<RM_Host>.out

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      mr-279, mrv2
    1. MR-2537-v1.patch
      0.5 kB
      Robert Joseph Evans
    2. MR-2537-v2.patch
      2 kB
      Robert Joseph Evans

      Activity

      Robert Joseph Evans created issue -
      Hide
      Robert Joseph Evans added a comment -

      Output should end in .log

      Show
      Robert Joseph Evans added a comment - Output should end in .log
      Hide
      Luke Lu added a comment -

      I tried to fix this pre yarn move to MR-279 branch. The problem is that we don't set -Dhadoop.root.logger for log4j.properties and it defaults to INFO,console which is redirected to .out.

      I think just fixing the scripts to define hadoop.root.logger to as the same as yarn.root.logger and hadoop.log.dir, yarn.log.dir would fix the problem.

      Show
      Luke Lu added a comment - I tried to fix this pre yarn move to MR-279 branch. The problem is that we don't set -Dhadoop.root.logger for log4j.properties and it defaults to INFO,console which is redirected to .out. I think just fixing the scripts to define hadoop.root.logger to as the same as yarn.root.logger and hadoop.log.dir, yarn.log.dir would fix the problem.
      Hide
      Robert Joseph Evans added a comment -

      Actually it is set in the shell script yarn/bin/yarn-daemon.sh

      Show
      Robert Joseph Evans added a comment - Actually it is set in the shell script yarn/bin/yarn-daemon.sh
      Robert Joseph Evans made changes -
      Field Original Value New Value
      Attachment MR-2537-v1.patch [ 12480589 ]
      Hide
      Robert Joseph Evans added a comment -

      No unit tests, because this is a very simple shell script fix.

      Show
      Robert Joseph Evans added a comment - No unit tests, because this is a very simple shell script fix.
      Robert Joseph Evans made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12480589/MR-2537-v1.patch
      against trunk revision 1127444.

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      -1 patch. The patch command could not apply the patch.

      Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/311//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480589/MR-2537-v1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/311//console This message is automatically generated.
      Hide
      Luke Lu added a comment -

      This is actually not the right fix. It basically redirect stdout to .log which is not the right thing to do. Currently in our logs dir, you'll see both .log and .out files. You'll notice that .log rolls daily but .out doesn't.

      Although yarn-daemon.sh defines YARN_ROOT_LOGGER as INFO,DRFA it's not being used to define -Dhadoop.root.logger which can then be picked up by log4j.properties.

      Show
      Luke Lu added a comment - This is actually not the right fix. It basically redirect stdout to .log which is not the right thing to do. Currently in our logs dir, you'll see both .log and .out files. You'll notice that .log rolls daily but .out doesn't. Although yarn-daemon.sh defines YARN_ROOT_LOGGER as INFO,DRFA it's not being used to define -Dhadoop.root.logger which can then be picked up by log4j.properties.
      Hide
      Robert Joseph Evans added a comment -

      OK I understand now. The original description of the issue by the person who found it was very vague.

      The issue is that yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out is where stdout of the command is placed, and that even though yarn.log.dir, yarn.log.file, and yarn.root.logger system properties are being set they are not being used by the log4j configuration. The log4j configuration defaults to CONSOLE which outputs the logs to stdout which in turn goes to yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out.

      The fix is to either update the log4j conf to use the yarn variants of these variables, or to set the hadoop equivalents of these variables instead. I assume from your comment that we need to switch them over to use the hadoop variants. Is this correct?

      Show
      Robert Joseph Evans added a comment - OK I understand now. The original description of the issue by the person who found it was very vague. The issue is that yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out is where stdout of the command is placed, and that even though yarn.log.dir, yarn.log.file, and yarn.root.logger system properties are being set they are not being used by the log4j configuration. The log4j configuration defaults to CONSOLE which outputs the logs to stdout which in turn goes to yarn-$YARN_IDENT_STRING-$command-$HOSTNAME.out. The fix is to either update the log4j conf to use the yarn variants of these variables, or to set the hadoop equivalents of these variables instead. I assume from your comment that we need to switch them over to use the hadoop variants. Is this correct?
      Hide
      Luke Lu added a comment -

      Exactly. Since the log4j.properties is from hadoop-common, adding yarn vars would not be kosher. I'd set both vars for posterity (yarn itself (non-mapreduce specific stuff) becomes a sub or top-level project).

      Show
      Luke Lu added a comment - Exactly. Since the log4j.properties is from hadoop-common, adding yarn vars would not be kosher. I'd set both vars for posterity (yarn itself (non-mapreduce specific stuff) becomes a sub or top-level project).
      Robert Joseph Evans made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Hide
      Robert Joseph Evans added a comment -

      OK added in hadoop.root.logger hadoop.log.dir and hadoop.log.file to the -D options

      Show
      Robert Joseph Evans added a comment - OK added in hadoop.root.logger hadoop.log.dir and hadoop.log.file to the -D options
      Robert Joseph Evans made changes -
      Attachment MR-2537-v2.patch [ 12480673 ]
      Robert Joseph Evans made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Luke Lu added a comment -

      +1, lgtm, assuming you have tested on at least a single node setup.

      Show
      Luke Lu added a comment - +1, lgtm, assuming you have tested on at least a single node setup.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12480673/MR-2537-v2.patch
      against trunk revision 1128394.

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      -1 patch. The patch command could not apply the patch.

      Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/314//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480673/MR-2537-v2.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/314//console This message is automatically generated.
      Hide
      Robert Joseph Evans added a comment -

      Yes. I installed it on a single node cluster. I launched both the nodemanager and the resource manager, and verified that the results were being logged to the proper files.

      Show
      Robert Joseph Evans added a comment - Yes. I installed it on a single node cluster. I launched both the nodemanager and the resource manager, and verified that the results were being logged to the proper files.
      Mahadev konar made changes -
      Fix Version/s 0.23.0 [ 12315570 ]
      Hide
      Mahadev konar added a comment -

      I just committed this. thanks robert!

      Show
      Mahadev konar added a comment - I just committed this. thanks robert!
      Mahadev konar made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Hadoop Flags [Reviewed]
      Resolution Fixed [ 1 ]
      Arun C Murthy made changes -
      Status Resolved [ 5 ] Closed [ 6 ]

        People

        • Assignee:
          Robert Joseph Evans
          Reporter:
          Robert Joseph Evans
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development