Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5225

SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Now, SplitSampler only samples the first maxSplitsSampled splits, caused by MAPREDUCE-1820. However, jumping around all splits is in general preferable than the first N splits.

        Issue Links

          Activity

          Hide
          zjshen Zhijie Shen added a comment -

          The patch makes SplitSampler jump by sampling step when it does sampling. Then, the behavior of SplitSampler in both mapred and mapreduce is identical.

          Show
          zjshen Zhijie Shen added a comment - The patch makes SplitSampler jump by sampling step when it does sampling. Then, the behavior of SplitSampler in both mapred and mapreduce is identical.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3602//console This message is automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          +1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 14m 36s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 javadoc 9m 32s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 22s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core.
          +1 mapreduce tests 108m 31s Tests passed in hadoop-mapreduce-client-jobclient.
              147m 31s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / f1a152c
          hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 36s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 22s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 1m 54s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 mapreduce tests 1m 34s Tests passed in hadoop-mapreduce-client-core. +1 mapreduce tests 108m 31s Tests passed in hadoop-mapreduce-client-jobclient.     147m 31s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / f1a152c hadoop-mapreduce-client-core test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/console This message was automatically generated.

            People

            • Assignee:
              zjshen Zhijie Shen
              Reporter:
              zjshen Zhijie Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development