Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5465

Tasks are often killed before they exit on their own

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: mr-am, mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If there is profiling enabled for mapper or reducer then hprof dumps profile.out at process exit. It is dumped after task signaled to AM that work is finished.

      AM kills container with finished work without waiting for hprof to finish dumps. If hprof is dumping larger outputs (such as with depth=4 while depth=3 works) , it could not finish dump in time before being killed making entire dump unusable because cpu and heap stats are missing.

      There needs to be better delay before container is killed if profiling is enabled.

        Attachments

        1. MAPREDUCE-5465.patch
          3 kB
          Ravi Prakash
        2. MAPREDUCE-5465-2.patch
          36 kB
          Ming Ma
        3. MAPREDUCE-5465-3.patch
          41 kB
          Ming Ma
        4. MAPREDUCE-5465-4.patch
          89 kB
          Ming Ma
        5. MAPREDUCE-5465-5.patch
          90 kB
          Ming Ma
        6. MAPREDUCE-5465-6.patch
          90 kB
          Ming Ma
        7. MAPREDUCE-5465-7.patch
          86 kB
          Ming Ma
        8. MAPREDUCE-5465-8.patch
          87 kB
          Ming Ma
        9. MAPREDUCE-5465-9.patch
          87 kB
          Ming Ma
        10. MAPREDUCE-5465-branch-2.patch
          86 kB
          Ming Ma

          Issue Links

            Activity

              People

              • Assignee:
                mingma Ming Ma
                Reporter:
                hsn Radim Kolar
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: