Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5641

History for failed Application Masters should be made available to the Job History Server

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.2.0
    • None
    • applicationmaster, jobhistoryserver
    • None

    Description

      Currently, the JHS has no information about jobs whose AMs have failed. This is because the History is written by the AM to the intermediate folder just before finishing, so when it fails for any reason, this information isn't copied there. However, it is not lost as its in the AM's staging directory. To make the History available in the JHS, all we need to do is have another mechanism to move the History from the staging directory to the intermediate directory. The AM also writes a "Summary" file before exiting normally, which is also unavailable when the AM fails.

      Attachments

        1. MAPREDUCE-5641.patch
          27 kB
          Robert Kanter
        2. MAPREDUCE-5641.patch
          27 kB
          Robert Kanter

        Issue Links

          depends upon

          Improvement - An improvement or enhancement to an existing feature or task. YARN-1731 ResourceManager should record killed ApplicationMasters for History

          • Major - Major loss of function.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-5418 JobHistoryServer has no information about applications if the MR-AM crashes

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              rkanter Robert Kanter
              Votes:
              1 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated: