Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6984

MR AM to clean up temporary files from previous attempt in case of no recovery

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: 3.1.0, 3.0.1, 2.10.0
    • Component/s: applicationmaster
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When the MR AM restarts, the {outputDir}/_temporary/{appAttemptNumber} directory remains on HDFS, even though this directory is not used during the next attempt if the restart has been done without recovery. So if recovery is not used for the AM restart, then the deletion of this directory can be done earlier (at the start of the next attempt). The benefit is that more free HDFS space is available for the next attempt.

        Attachments

        1. MAPREDUCE-6984.000.patch
          18 kB
          Gergo Repas
        2. MAPREDUCE-6984.001.patch
          18 kB
          Gergo Repas
        3. MAPREDUCE-6984.003.patch
          19 kB
          Gergo Repas
        4. MAPREDUCE-6984.004.patch
          19 kB
          Gergo Repas
        5. MAPREDUCE-6984.005.patch
          20 kB
          Gergo Repas
        6. MAPREDUCE-6984.006.patch
          20 kB
          Gergo Repas
        7. MAPREDUCE-6984.007.patch
          25 kB
          Gergo Repas
        8. MAPREDUCE-6984.008.patch
          24 kB
          Gergo Repas
        9. MAPREDUCE-6984.009.patch
          9 kB
          Gergo Repas
        10. MAPREDUCE-6984.010.patch
          8 kB
          Gergo Repas

        Issue Links

          Activity

            People

            • Assignee:
              grepas Gergo Repas
              Reporter:
              grepas Gergo Repas

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment