Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4815

Speed up FileOutputCommitter#commitJob for many output files

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.1-alpha, 2.4.1
    • Fix Version/s: 2.7.0
    • Component/s: mrv2
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing.

        Attachments

        1. MAPREDUCE-4815.v10.patch
          18 kB
          Siqi Li
        2. MAPREDUCE-4815.v11.patch
          23 kB
          Siqi Li
        3. MAPREDUCE-4815.v12.patch
          27 kB
          Siqi Li
        4. MAPREDUCE-4815.v13.patch
          29 kB
          Siqi Li
        5. MAPREDUCE-4815.v14.patch
          31 kB
          Siqi Li
        6. MAPREDUCE-4815.v15.patch
          32 kB
          Siqi Li
        7. MAPREDUCE-4815.v16.patch
          32 kB
          Siqi Li
        8. MAPREDUCE-4815.v17.patch
          32 kB
          Siqi Li
        9. MAPREDUCE-4815.v3.patch
          26 kB
          Siqi Li
        10. MAPREDUCE-4815.v4.patch
          27 kB
          Siqi Li
        11. MAPREDUCE-4815.v5.patch
          27 kB
          Siqi Li
        12. MAPREDUCE-4815.v6.patch
          29 kB
          Siqi Li
        13. MAPREDUCE-4815.v7.patch
          24 kB
          Siqi Li
        14. MAPREDUCE-4815.v8.patch
          33 kB
          Siqi Li
        15. MAPREDUCE-4815.v9.patch
          14 kB
          Siqi Li

          Issue Links

            Activity

              People

              • Assignee:
                l201514 Siqi Li
                Reporter:
                jlowe Jason Lowe
              • Votes:
                0 Vote for this issue
                Watchers:
                46 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: