Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6751

Add debug log message when splitting is not possible due to unsplittable compression

    Details

    • Hadoop Flags:
      Reviewed

      Description

      There should be a message logged in case of the mapreduce will only spam one mapper since the source file is compressed with an unsplitable algorithm

      1. MAPREDUCE-6751.patch
        4 kB
        Peter Vary
      2. MAPREDUCE-6751.2.patch
        4 kB
        Peter Vary

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10285 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10285/)
        MAPREDUCE-6751. Add debug log message when splitting is not possible due (rchiang: rev 6c154abd33279475315b5f7f78dc47f1b0aa7028)

        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java
        • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10285 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10285/ ) MAPREDUCE-6751 . Add debug log message when splitting is not possible due (rchiang: rev 6c154abd33279475315b5f7f78dc47f1b0aa7028) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/CombineFileInputFormat.java
        Hide
        rchiang Ray Chiang added a comment -

        Committed to trunk and branch-2.

        Thanks Peter Vary for the contribution! Thanks Haibo Chen for reviewing!

        Show
        rchiang Ray Chiang added a comment - Committed to trunk and branch-2. Thanks Peter Vary for the contribution! Thanks Haibo Chen for reviewing!
        Hide
        rchiang Ray Chiang added a comment -

        +1. Looks like a very useful message to me. Committing...

        Show
        rchiang Ray Chiang added a comment - +1. Looks like a very useful message to me. Committing...
        Hide
        haibochen Haibo Chen added a comment -

        Thanks Peter Vary for the patch. LGTM, non-binding.

        Show
        haibochen Haibo Chen added a comment - Thanks Peter Vary for the patch. LGTM, non-binding.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 38s trunk passed
        +1 compile 0m 24s trunk passed
        +1 checkstyle 0m 18s trunk passed
        +1 mvnsite 0m 28s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 0m 49s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 23s the patch passed
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 16s the patch passed
        +1 mvnsite 0m 32s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 4s the patch passed
        +1 javadoc 0m 22s the patch passed
        +1 unit 2m 20s hadoop-mapreduce-client-core in the patch passed.
        +1 asflicense 0m 16s The patch does not generate ASF License warnings.
        15m 56s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823344/MAPREDUCE-6751.2.patch
        JIRA Issue MAPREDUCE-6751
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux b275382a67e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 874577a
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6669/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6669/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 38s trunk passed +1 compile 0m 24s trunk passed +1 checkstyle 0m 18s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 49s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 23s the patch passed +1 compile 0m 23s the patch passed +1 javac 0m 23s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 32s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 4s the patch passed +1 javadoc 0m 22s the patch passed +1 unit 2m 20s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 16s The patch does not generate ASF License warnings. 15m 56s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823344/MAPREDUCE-6751.2.patch JIRA Issue MAPREDUCE-6751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b275382a67e9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 874577a Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6669/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6669/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        pvary Peter Vary added a comment -

        Update the patch, to consider the "mapreduce.input.fileinputformat.split.minsize" configuration value when deciding how much splits would be if the file would be splittable.

        Show
        pvary Peter Vary added a comment - Update the patch, to consider the "mapreduce.input.fileinputformat.split.minsize" configuration value when deciding how much splits would be if the file would be splittable.
        Hide
        pvary Peter Vary added a comment -

        There is no extra test, because it is only about logging.

        Tested the logs with hive, which uses the hadoop-mapreduce-client-core to execute mapreduce tasks.

        The result was as expected.

        Please review.

        Thanks,
        Peter

        Show
        pvary Peter Vary added a comment - There is no extra test, because it is only about logging. Tested the logs with hive, which uses the hadoop-mapreduce-client-core to execute mapreduce tasks. The result was as expected. Please review. Thanks, Peter
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 1s trunk passed
        +1 compile 0m 25s trunk passed
        +1 checkstyle 0m 19s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 0m 48s trunk passed
        +1 javadoc 0m 21s trunk passed
        +1 mvninstall 0m 22s the patch passed
        +1 compile 0m 21s the patch passed
        +1 javac 0m 21s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 25s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 0m 52s the patch passed
        +1 javadoc 0m 18s the patch passed
        +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed.
        +1 asflicense 0m 15s The patch does not generate ASF License warnings.
        15m 30s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823148/MAPREDUCE-6751.patch
        JIRA Issue MAPREDUCE-6751
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 573f6564fcf3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / e83be44
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6668/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6668/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 1s trunk passed +1 compile 0m 25s trunk passed +1 checkstyle 0m 19s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 48s trunk passed +1 javadoc 0m 21s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 21s the patch passed +1 javac 0m 21s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 52s the patch passed +1 javadoc 0m 18s the patch passed +1 unit 2m 2s hadoop-mapreduce-client-core in the patch passed. +1 asflicense 0m 15s The patch does not generate ASF License warnings. 15m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823148/MAPREDUCE-6751.patch JIRA Issue MAPREDUCE-6751 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 573f6564fcf3 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / e83be44 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6668/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6668/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        pvary Peter Vary added a comment -

        Patch with the new log messages

        Show
        pvary Peter Vary added a comment - Patch with the new log messages

          People

          • Assignee:
            pvary Peter Vary
            Reporter:
            pvary Peter Vary
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development