Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4815

Speed up FileOutputCommitter#commitJob for many output files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.3, 2.0.1-alpha, 2.4.1
    • 2.7.0
    • mrv2
    • Reviewed

    Description

      If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing.

      Attachments

        1. MAPREDUCE-4815.v10.patch
          18 kB
          Siqi Li
        2. MAPREDUCE-4815.v11.patch
          23 kB
          Siqi Li
        3. MAPREDUCE-4815.v12.patch
          27 kB
          Siqi Li
        4. MAPREDUCE-4815.v13.patch
          29 kB
          Siqi Li
        5. MAPREDUCE-4815.v14.patch
          31 kB
          Siqi Li
        6. MAPREDUCE-4815.v15.patch
          32 kB
          Siqi Li
        7. MAPREDUCE-4815.v16.patch
          32 kB
          Siqi Li
        8. MAPREDUCE-4815.v17.patch
          32 kB
          Siqi Li
        9. MAPREDUCE-4815.v3.patch
          26 kB
          Siqi Li
        10. MAPREDUCE-4815.v4.patch
          27 kB
          Siqi Li
        11. MAPREDUCE-4815.v5.patch
          27 kB
          Siqi Li
        12. MAPREDUCE-4815.v6.patch
          29 kB
          Siqi Li
        13. MAPREDUCE-4815.v7.patch
          24 kB
          Siqi Li
        14. MAPREDUCE-4815.v8.patch
          33 kB
          Siqi Li
        15. MAPREDUCE-4815.v9.patch
          14 kB
          Siqi Li

        Issue Links

          is depended upon by

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6608 Work Preserving AM Restart for MapReduce

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-6336 Enable v2 FileOutputCommitter by default

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-5485 Allow repeating job commit by extending OutputCommitter API

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-6275 Race condition in FileOutputCommitter v2 for user-specified task output subdirs

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              l201514 Siqi Li
              jlowe Jason Darrell Lowe
              Votes:
              0 Vote for this issue
              Watchers:
              46 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: