Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-3182

loadgen ignores -m command line when writing random data

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.0, 2.3.0
    • Fix Version/s: None
    • Component/s: mrv2, test
    • Labels:
      None
    • Target Version/s:

      Description

      If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code.

        Activity

        Hide
        Chen He added a comment -

        I will take look at this issue.

        Show
        Chen He added a comment - I will take look at this issue.
        Hide
        Chen He added a comment -

        There two GenericLoadGenerator classes in current Hadoop source code.
        One is under org.apache.hadoop.mapreduce package. It has two documentation problems. Firstly, it does not actually parse the "-m" command line option but still show this option in the "Usage". Secondly, if user does not specify the input directory, it will create input data using RandomWriter with default setting( 10GB per map task and 10 map task per node). However, it does not show this option in the "Usage".

        The other is under org.apache.hadoop.mapred package; It is an older version of GenericLoadGenerator. It has the second documentation problem described in above paragraph.

        Show
        Chen He added a comment - There two GenericLoadGenerator classes in current Hadoop source code. One is under org.apache.hadoop.mapreduce package. It has two documentation problems. Firstly, it does not actually parse the "-m" command line option but still show this option in the "Usage". Secondly, if user does not specify the input directory, it will create input data using RandomWriter with default setting( 10GB per map task and 10 map task per node). However, it does not show this option in the "Usage". The other is under org.apache.hadoop.mapred package; It is an older version of GenericLoadGenerator. It has the second documentation problem described in above paragraph.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12640090/MAPREDUCE-3182.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4512//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4512//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12640090/MAPREDUCE-3182.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4512//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4512//console This message is automatically generated.
        Hide
        Chen He added a comment -

        Hi Jonathan Eagles, would you mind take a look of this patch. Thank you very much!

        Show
        Chen He added a comment - Hi Jonathan Eagles , would you mind take a look of this patch. Thank you very much!

          People

          • Assignee:
            Chen He
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development