Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6984

MR AM to clean up temporary files from previous attempt in case of no recovery

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.1.0, 3.0.1, 2.10.0
    • Component/s: applicationmaster
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the MR AM restarts, the {outputDir}/_temporary/{appAttemptNumber} directory remains on HDFS, even though this directory is not used during the next attempt if the restart has been done without recovery. So if recovery is not used for the AM restart, then the deletion of this directory can be done earlier (at the start of the next attempt). The benefit is that more free HDFS space is available for the next attempt.

        Attachments

        1. MAPREDUCE-6984.010.patch
          8 kB
          Gergo Repas
        2. MAPREDUCE-6984.009.patch
          9 kB
          Gergo Repas
        3. MAPREDUCE-6984.008.patch
          24 kB
          Gergo Repas
        4. MAPREDUCE-6984.007.patch
          25 kB
          Gergo Repas
        5. MAPREDUCE-6984.006.patch
          20 kB
          Gergo Repas
        6. MAPREDUCE-6984.005.patch
          20 kB
          Gergo Repas
        7. MAPREDUCE-6984.004.patch
          19 kB
          Gergo Repas
        8. MAPREDUCE-6984.003.patch
          19 kB
          Gergo Repas
        9. MAPREDUCE-6984.001.patch
          18 kB
          Gergo Repas
        10. MAPREDUCE-6984.000.patch
          18 kB
          Gergo Repas

          Issue Links

            Activity

              People

              • Assignee:
                grepas Gergo Repas
                Reporter:
                grepas Gergo Repas
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: