Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6771

RMContainerAllocator sends container diagnostics event after corresponding completion event

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.7.3
    • 2.8.0, 2.7.4, 3.0.0-alpha2
    • mrv2
    • None

    Description

      Task containers can go over their resource limit, and killed by Node Manager. Then MR AM gets notified of the container status and diagnostics information through its heartbeat with RM. However, it is possible that the diagnostics information never gets into .jhist file, so when the job completes, the diagnostics information associated with the failed task attempts is empty. This makes it hard for users to root cause job failures that are often caused by memory leak.

      Attachments

        1. mapreduce6771.001.patch
          1 kB
          Haibo Chen
        2. TaUnsuccessfullyEventEmission.jpg
          1.69 MB
          Haibo Chen
        3. mapreduce6771.002.patch
          8 kB
          Haibo Chen
        4. mapreduce6771.003.patch
          8 kB
          Haibo Chen
        5. mapreduce6771.004.patch
          8 kB
          Haibo Chen
        6. mapreduce6771.branch-2.8.patch
          8 kB
          Haibo Chen

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-4955 NM container diagnostics for excess resource usage can be lost if task fails while being killed

          • Major - Major loss of function.
          • Open

          Activity

            People

              haibochen Haibo Chen
              haibochen Haibo Chen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: