Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4099

ApplicationMaster may fail to remove staging directory

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.2
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      When the ApplicationMaster shuts down it's supposed to remove the staging directory, assuming properties weren't set to override this behavior. During shutdown the AM tells the ResourceManager that it has finished before it cleans up the staging directory. However upon hearing the AM has finished, the RM turns right around and kills the AM container. If the AM is too slow, the AM will be killed before the staging directory is removed.

      We're seeing the AM lose this race fairly consistently on our clusters, and the lack of staging directory cleanup quickly leads to filesystem quota issues for some users.

      1. MAPREDUCE-4099-addendum.patch
        13 kB
        Jason Lowe
      2. MAPREDUCE-4099-addendum.patch
        13 kB
        Jason Lowe
      3. MAPREDUCE-4099.patch
        35 kB
        Jason Lowe
      4. MAPREDUCE-4099.patch
        35 kB
        Jason Lowe
      5. MAPREDUCE-4099.patch
        5 kB
        Jason Lowe

        Activity

        Jason Lowe created issue -
        Jason Lowe made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4099.patch [ 12521769 ]
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Target Version/s 0.23.3 [ 12320060 ]
        Assignee Jason Lowe [ jlowe ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Jason Lowe made changes -
        Attachment MAPREDUCE-4099.patch [ 12521985 ]
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Jason Lowe made changes -
        Attachment MAPREDUCE-4099.patch [ 12522123 ]
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Robert Joseph Evans made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Fix Version/s 0.23.3 [ 12320060 ]
        Fix Version/s 2.0.0 [ 12320354 ]
        Resolution Fixed [ 1 ]
        Siddharth Seth made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Jason Lowe made changes -
        Attachment MAPREDUCE-4099-addendum.patch [ 12522240 ]
        Jason Lowe made changes -
        Status Reopened [ 4 ] Patch Available [ 10002 ]
        Jason Lowe made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Jason Lowe made changes -
        Attachment MAPREDUCE-4099-addendum.patch [ 12522250 ]
        Jason Lowe made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Siddharth Seth made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Arun C Murthy made changes -
        Fix Version/s 2.0.2-alpha [ 12322471 ]
        Fix Version/s 2.0.0-alpha [ 12320354 ]

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development