Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4815

Speed up FileOutputCommitter#commitJob for many output files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.3, 2.0.1-alpha, 2.4.1
    • 2.7.0
    • mrv2
    • Reviewed

    Description

      If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing.

      Attachments

        1. MAPREDUCE-4815.v10.patch
          18 kB
          Siqi Li
        2. MAPREDUCE-4815.v11.patch
          23 kB
          Siqi Li
        3. MAPREDUCE-4815.v12.patch
          27 kB
          Siqi Li
        4. MAPREDUCE-4815.v13.patch
          29 kB
          Siqi Li
        5. MAPREDUCE-4815.v14.patch
          31 kB
          Siqi Li
        6. MAPREDUCE-4815.v15.patch
          32 kB
          Siqi Li
        7. MAPREDUCE-4815.v16.patch
          32 kB
          Siqi Li
        8. MAPREDUCE-4815.v17.patch
          32 kB
          Siqi Li
        9. MAPREDUCE-4815.v3.patch
          26 kB
          Siqi Li
        10. MAPREDUCE-4815.v4.patch
          27 kB
          Siqi Li
        11. MAPREDUCE-4815.v5.patch
          27 kB
          Siqi Li
        12. MAPREDUCE-4815.v6.patch
          29 kB
          Siqi Li
        13. MAPREDUCE-4815.v7.patch
          24 kB
          Siqi Li
        14. MAPREDUCE-4815.v8.patch
          33 kB
          Siqi Li
        15. MAPREDUCE-4815.v9.patch
          14 kB
          Siqi Li

        Issue Links

          Activity

            People

              l201514 Siqi Li
              jlowe Jason Darrell Lowe
              Votes:
              0 Vote for this issue
              Watchers:
              47 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: