Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3968

add support for getNumMapTasks() into mapreduce JobContext

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
      None
    • Environment:

      hadoop 0.22

      Description

      In old mapred api there was way to query number of mappers:

      job.getNumMapTasks())

      No such function exists in new mapreduce api

      1. maptasks4.txt
        7 kB
        Radim Kolar
      2. maptasks3.txt
        7 kB
        Radim Kolar
      3. maptasks2.txt
        7 kB
        Radim Kolar
      4. maptasks.txt
        7 kB
        Radim Kolar
      5. MAPREDUCE-3968.patch
        4 kB
        madhukara phatak

        Activity

        Hide
        madhukara phatak added a comment -

        added new property. Doesn't need the test case since it just gives nicer interface for the existing configuration property

        Show
        madhukara phatak added a comment - added new property. Doesn't need the test case since it just gives nicer interface for the existing configuration property
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12528063/MAPREDUCE-3968.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2395//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2395//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12528063/MAPREDUCE-3968.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2395//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2395//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Radim,

        What good will this serve really? There isn't a setNumMapTasks() in new API anymore (for good reason). The FileInputFormat now expects and acts on different params.

        The only thing it can be worthy of is to check # of splits post submission, which should be done via other means anyway, not like this. Is that what you want it for?

        Show
        Harsh J added a comment - Radim, What good will this serve really? There isn't a setNumMapTasks() in new API anymore (for good reason). The FileInputFormat now expects and acts on different params. The only thing it can be worthy of is to check # of splits post submission, which should be done via other means anyway, not like this. Is that what you want it for?
        Hide
        Radim Kolar added a comment -

        Yes, i need to know number of splits.

        Show
        Radim Kolar added a comment - Yes, i need to know number of splits.
        Hide
        Radim Kolar added a comment -

        I do not think that this has chances to ever be committed.

        Show
        Radim Kolar added a comment - I do not think that this has chances to ever be committed.
        Hide
        Harsh J added a comment -

        Radim,

        I think this can go in - my earlier questions were just to know purpose.

        Sorry I did not comment back earlier. Could you also add in some simple tests (just for regression and API presence's sake)?

        Also a rough "since" comment would be well too, in the javadocs of the new API method, and an explanation of what to tweak if the intention is to set number of mappers (i.e. point to using InputFormat#getSplits(…)).

        Show
        Harsh J added a comment - Radim, I think this can go in - my earlier questions were just to know purpose. Sorry I did not comment back earlier. Could you also add in some simple tests (just for regression and API presence's sake)? Also a rough "since" comment would be well too, in the javadocs of the new API method, and an explanation of what to tweak if the intention is to set number of mappers (i.e. point to using InputFormat#getSplits(…)).
        Hide
        Radim Kolar added a comment -

        its for compatibility with old api

        Show
        Radim Kolar added a comment - its for compatibility with old api
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12554696/maptasks.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3055//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554696/maptasks.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3055//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12554698/maptasks2.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3056//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554698/maptasks2.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3056//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12554700/maptasks3.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:

        org.apache.hadoop.mapreduce.TestJob

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3057//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3057//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554700/maptasks3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core: org.apache.hadoop.mapreduce.TestJob +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3057//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3057//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12554701/maptasks4.txt
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3058//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3058//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554701/maptasks4.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3058//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3058//console This message is automatically generated.

          People

          • Assignee:
            Unassigned
            Reporter:
            Radim Kolar
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development