Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2644

NodeManager fails to create containers when NM_LOG_DIR is not explicitly set in the Configuration

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If the yarn configuration does not explicitly specify a value for the yarn.server.nodemanager.log.dir property, container allocation will fail on the NodeManager w/an NPE when the LocalDirAllocator goes to create the temp directory. In most of the code, we handle this by defaulting to /tmp/logs, but we cannot do this in the LocalDirAllocator context, so we need to set the default value explicitly in the Configuration.

      Marking this as major b/c it's annoying to bump into it when you're getting your first MRv2 cluster up and running.

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this to MR-279 branch. Thanks Josh!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this to MR-279 branch. Thanks Josh!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12485459/MAPREDUCE-2644.patch
          against trunk revision 1145424.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/453//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12485459/MAPREDUCE-2644.patch against trunk revision 1145424. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/453//console This message is automatically generated.
          Hide
          Josh Wills added a comment -

          Moves the default setting for NM_LOG_DIR to the yarn-default.xml file, as per Vinod's comment.

          Show
          Josh Wills added a comment - Moves the default setting for NM_LOG_DIR to the yarn-default.xml file, as per Vinod's comment.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Josh, thanks for looking.

          Like Aaron mentioned, the correct fix is to mention it in the default files, but add it in yarn-default.xml instead of mapred-site.xml.

          Also just to be sure, you can do the same null-check and set in the base NodeManager class, that will solve it completely.

          Show
          Vinod Kumar Vavilapalli added a comment - Josh, thanks for looking. Like Aaron mentioned, the correct fix is to mention it in the default files, but add it in yarn-default.xml instead of mapred-site.xml. Also just to be sure, you can do the same null-check and set in the base NodeManager class, that will solve it completely.
          Hide
          Aaron T. Myers added a comment -

          One option would be to change LocalDirAllocator to accept a default value for the config key you pass in. Another (probably better) option would be to just add a default value to the mapred-default.xml file, which gets loaded any time an MR config object is created.

          Show
          Aaron T. Myers added a comment - One option would be to change LocalDirAllocator to accept a default value for the config key you pass in. Another (probably better) option would be to just add a default value to the mapred-default.xml file, which gets loaded any time an MR config object is created.
          Hide
          Josh Wills added a comment -

          Okay Aaron-- then is the right place to fix this the LocalDirAllocator under HDFS? It's the code that is being executed hee that doesn't allow for the two-arg version of Configuration.get(...).

          Show
          Josh Wills added a comment - Okay Aaron-- then is the right place to fix this the LocalDirAllocator under HDFS? It's the code that is being executed hee that doesn't allow for the two-arg version of Configuration.get(...).
          Hide
          Aaron T. Myers added a comment -

          Hey Josh, the usual thing done in the rest of the Hadoop code when providing a default value for a config is to use the two-argument version of Configuration.get(...) whose first argument is the config key and whose second argument will be returned as the value if no value for that key is set in the conf object.

          Show
          Aaron T. Myers added a comment - Hey Josh, the usual thing done in the rest of the Hadoop code when providing a default value for a config is to use the two-argument version of Configuration.get(...) whose first argument is the config key and whose second argument will be returned as the value if no value for that key is set in the conf object.
          Hide
          Josh Wills added a comment -

          Not my prettiest impl, but it's functional. If there's a better place in the code to do this, let me know.

          Show
          Josh Wills added a comment - Not my prettiest impl, but it's functional. If there's a better place in the code to do this, let me know.

            People

            • Assignee:
              Josh Wills
              Reporter:
              Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development