Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 3.0.0-alpha1
    • Component/s: mrv2
    • Labels:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      mapreduce.fileoutputcommitter.algorithm.version now defaults to 2.
        
      In algorithm version 1:

        1. commitTask renames directory
        $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
        to
        $joboutput/_temporary/$appAttemptID/$taskID/

        2. recoverTask renames
        $joboutput/_temporary/$appAttemptID/$taskID/
        to
        $joboutput/_temporary/($appAttemptID + 1)/$taskID/

        3. commitJob merges every task output file in
        $joboutput/_temporary/$appAttemptID/$taskID/
        to
        $joboutput/, then it will delete $joboutput/_temporary/
        and write $joboutput/_SUCCESS

      commitJob's run time, number of RPC, is O(n) in terms of output files, which is discussed in MAPREDUCE-4815, and can take minutes.

      Algorithm version 2 changes the behavior of commitTask, recoverTask, and commitJob.

        1. commitTask renames all files in
        $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/
        to $joboutput/

        2. recoverTask is a nop strictly speaking, but for
        upgrade from version 1 to version 2 case, it checks if there
        are any files in
        $joboutput/_temporary/($appAttemptID - 1)/$taskID/
        and renames them to $joboutput/

        3. commitJob deletes $joboutput/_temporary and writes
        $joboutput/_SUCCESS

      Algorithm 2 takes advantage of task parallelism and makes commitJob itself O(1). However, the window of vulnerability for having incomplete output in $jobOutput directory is much larger. Therefore, pipeline logic for consuming job outputs should be built on checking for existence of _SUCCESS marker.
      Show
      mapreduce.fileoutputcommitter.algorithm.version now defaults to 2.    In algorithm version 1:   1. commitTask renames directory   $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/   to   $joboutput/_temporary/$appAttemptID/$taskID/   2. recoverTask renames   $joboutput/_temporary/$appAttemptID/$taskID/   to   $joboutput/_temporary/($appAttemptID + 1)/$taskID/   3. commitJob merges every task output file in   $joboutput/_temporary/$appAttemptID/$taskID/   to   $joboutput/, then it will delete $joboutput/_temporary/   and write $joboutput/_SUCCESS commitJob's run time, number of RPC, is O(n) in terms of output files, which is discussed in MAPREDUCE-4815 , and can take minutes. Algorithm version 2 changes the behavior of commitTask, recoverTask, and commitJob.   1. commitTask renames all files in   $joboutput/_temporary/$appAttemptID/_temporary/$taskAttemptID/   to $joboutput/   2. recoverTask is a nop strictly speaking, but for   upgrade from version 1 to version 2 case, it checks if there   are any files in   $joboutput/_temporary/($appAttemptID - 1)/$taskID/   and renames them to $joboutput/   3. commitJob deletes $joboutput/_temporary and writes   $joboutput/_SUCCESS Algorithm 2 takes advantage of task parallelism and makes commitJob itself O(1). However, the window of vulnerability for having incomplete output in $jobOutput directory is much larger. Therefore, pipeline logic for consuming job outputs should be built on checking for existence of _SUCCESS marker.

      Description

      This JIRA is to propose making new FileOutputCommitter behavior from MAPREDUCE-4815 enabled by default in trunk, and potentially in branch-2.

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 33s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core.
              35m 48s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12727957/MAPREDUCE-6336.v1.patch
          Optional Tests javadoc javac unit
          git revision trunk / cf6c8a1
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/testReport/
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 33s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 30s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 mapreduce tests 1m 41s Tests passed in hadoop-mapreduce-client-core.     35m 48s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727957/MAPREDUCE-6336.v1.patch Optional Tests javadoc javac unit git revision trunk / cf6c8a1 hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/testReport/ Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5444/console This message was automatically generated.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks for posting a patch Siqi Li.

          Jason Lowe I would appreciate if you share your thoughts? We believe it should be a safe change for trunk at least.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks for posting a patch Siqi Li . Jason Lowe I would appreciate if you share your thoughts? We believe it should be a safe change for trunk at least.
          Hide
          jlowe Jason Lowe added a comment -

          I'm +1 for trunk, assuming there hasn't been any reports of bad breakage from those who have enabled it so far.

          Only concern for enabling for branch-2 is it does change the semantics of jobs that fail in a catastrophic way (e.g.: AM crashes and exhausts retries or killed via YARN and could not cleanup). In the old scheme there would normally not be any files at the top-level of the output directory, only a temporary directory holding the attempts. In the new scheme it is very likely there can be partial output (i.e.: some files have been committed to final destination, others have not). Downstream jobs in workflows would need to check via other means (e.g.: _SUCCESS file, query RM/JHS for job status, etc.) to verify the output is complete.

          Show
          jlowe Jason Lowe added a comment - I'm +1 for trunk, assuming there hasn't been any reports of bad breakage from those who have enabled it so far. Only concern for enabling for branch-2 is it does change the semantics of jobs that fail in a catastrophic way (e.g.: AM crashes and exhausts retries or killed via YARN and could not cleanup). In the old scheme there would normally not be any files at the top-level of the output directory, only a temporary directory holding the attempts. In the new scheme it is very likely there can be partial output (i.e.: some files have been committed to final destination, others have not). Downstream jobs in workflows would need to check via other means (e.g.: _SUCCESS file, query RM/JHS for job status, etc.) to verify the output is complete.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          We've been running on v2 on some clusters for probably a month already, and had no issues.

          Show
          jira.shegalov Gera Shegalov added a comment - We've been running on v2 on some clusters for probably a month already, and had no issues.
          Hide
          jira.shegalov Gera Shegalov added a comment -

          Thanks, Siqi Li for contribution, and Jason Lowe for review! Committed to trunk.

          Show
          jira.shegalov Gera Shegalov added a comment - Thanks, Siqi Li for contribution, and Jason Lowe for review! Committed to trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #7910 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7910/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #7910 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7910/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #211 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/211/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #941 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/941/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2139 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2139/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #199 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/199/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #209 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/209/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/)
          MAPREDUCE-6336. Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2157 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2157/ ) MAPREDUCE-6336 . Enable v2 FileOutputCommitter by default. (Siqi Li via gera) (gera: rev 788406e90cdfd1c7b55bae63602f744b484c040b) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

            People

            • Assignee:
              l201514 Siqi Li
              Reporter:
              jira.shegalov Gera Shegalov
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development