Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4369

Fix streaming job failures with WindowsResourceCalculatorPlugin

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1-win
    • None
    • None

    Description

      Some streaming jobs use local mode job runs that do not start tasks trackers. In these cases, the jvm context is not setup and hence local mode execution causes the code to crash.
      Fix is to not not use ResourceCalculatorPlugin in such cases or make the local job run creating dummy jvm contexts. Choosing the first option because thats the current implicit behavior in Linux. The ProcfsBasedProcessTree (used inside the LinuxResourceCalculatorPlugin) does no real work when the process pid is not setup correctly. This is what happens when local job mode runs.

      Attachments

        1. MAPREDUCE-4369.branch-1-win.1.patch
          4 kB
          Bikas Saha
        2. MAPREDUCE-4369.branch-1-win.2.patch
          6 kB
          Bikas Saha

        Issue Links

          is part of

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-8079 Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              bikassaha Bikas Saha
              bikassaha Bikas Saha
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: