Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-987

Exposing MiniDFS and MiniMR clusters as a single process command-line

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 2.0.2-alpha
    • Fix Version/s: 1.2.0, 2.0.2-alpha
    • Component/s: build, test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It's hard to test non-Java programs that rely on significant mapreduce functionality. The patch I'm proposing shortly will let you just type "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar minicluster" to start a cluster (internally, it's using Mini

      {MR,HDFS}

      Cluster) with a specified number of daemons, etc. A test that checks how some external process interacts with Hadoop might start minicluster as a subprocess, run through its thing, and then simply kill the java subprocess.

      I've been using just such a system for a couple of weeks, and I like it. It's significantly easier than developing a lot of scripts to start a pseudo-distributed cluster, and then clean up after it. I figure others might find it useful as well.

      I'm at a bit of a loss as to where to put it in 0.21. hdfs-with-mr tests have all the required libraries, so I've put it there. I could conceivably split this into "minimr" and "minihdfs", but it's specifically the fact that they're configured to talk to each other that I like about having them together. And one JVM is better than two for my test programs.

      1. MAPREDUCE-987_branch-1.0_rev2.patch
        17 kB
        Ahmed Radwan
      2. MAPREDUCE-987_rev4.patch
        17 kB
        Ahmed Radwan
      3. MAPREDUCE-987_rev3.patch
        17 kB
        Ahmed Radwan
      4. MAPREDUCE-987_rev2.patch
        12 kB
        Ahmed Radwan
      5. MAPREDUCE-987_branch-1.0.patch
        11 kB
        Ahmed Radwan
      6. MAPREDUCE-987.patch
        12 kB
        Ahmed Radwan
      7. MAPREDUCE-987.patch
        10 kB
        Philip Zeyliger
      8. HDFS-621-0.20-patch
        10 kB
        Philip Zeyliger
      9. HDFS-621.patch
        11 kB
        Philip Zeyliger

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          I guess you need a separated MAPREDUCE patch for MiniMR.

          Show
          Tsz Wo Nicholas Sze added a comment - I guess you need a separated MAPREDUCE patch for MiniMR.
          Hide
          Konstantin Boudnik added a comment -

          .bq And one JVM is better than two for my test programs.
          Is it better for the testing of the product ?

          Show
          Konstantin Boudnik added a comment - .bq And one JVM is better than two for my test programs. Is it better for the testing of the product ?
          Hide
          Philip Zeyliger added a comment -

          Uploading patch to start a single-process cluster.

          Show
          Philip Zeyliger added a comment - Uploading patch to start a single-process cluster.
          Hide
          Philip Zeyliger added a comment -

          I guess you need a separated MAPREDUCE patch for MiniMR.

          If you would, take a quick look to see how I use MiniMRCluster. Do you feel I'm abusing the fact that hdfs-hdfswithmr-test exists?

          Is it better for the testing of the product?

          It doesn't contribute anything to testing Hadoop. It does make it easier to write tests for programs that are downstream of Hadoop. One might argue that such facilities ought to be "top-level" and not shunted into a "-test" jar. I'm open to suggestions. (I could also see this being shunted to contrib, if folks have a strong preference.)

          Show
          Philip Zeyliger added a comment - I guess you need a separated MAPREDUCE patch for MiniMR. If you would, take a quick look to see how I use MiniMRCluster. Do you feel I'm abusing the fact that hdfs-hdfswithmr-test exists? Is it better for the testing of the product? It doesn't contribute anything to testing Hadoop. It does make it easier to write tests for programs that are downstream of Hadoop. One might argue that such facilities ought to be "top-level" and not shunted into a "-test" jar. I'm open to suggestions. (I could also see this being shunted to contrib, if folks have a strong preference.)
          Hide
          Owen O'Malley added a comment -

          I would rather that the local runner be made to work better than fake a mini cluster. For testing the framework, the mini cluster makes sense. But I think that for testing applications, the local runner is a much better fit.

          Show
          Owen O'Malley added a comment - I would rather that the local runner be made to work better than fake a mini cluster. For testing the framework, the mini cluster makes sense. But I think that for testing applications, the local runner is a much better fit.
          Hide
          Philip Zeyliger added a comment -

          Owen,

          I totally agree that LocalJobRunner should be maximally useful. That's great for testing jobs.

          Let's say I have a python class that knows how to interact with HDFS and MR. It knows how to look at files, start jobs, etc. I call out to hadoop binaries to interact with HDFS, and I want to capture all the details that occur when I talk to my real cluster. For this, if I were in Java, I'd spin up a Mini* cluster. Since I'm not in Java, I resort to spinning up a subprocess. I could also mock everything out, but at the end of the day, I want an integration test, and I really don't want to run it against a cluster that has to be setup externally: I'd rather the cluster be spun up and shut down by my test itself.

          I'm happy to throw this contrib/ if you feel strongly about it. I figure it'd be useful to other folks.

          – Philip

          Show
          Philip Zeyliger added a comment - Owen, I totally agree that LocalJobRunner should be maximally useful. That's great for testing jobs. Let's say I have a python class that knows how to interact with HDFS and MR. It knows how to look at files, start jobs, etc. I call out to hadoop binaries to interact with HDFS, and I want to capture all the details that occur when I talk to my real cluster. For this, if I were in Java, I'd spin up a Mini* cluster. Since I'm not in Java, I resort to spinning up a subprocess. I could also mock everything out, but at the end of the day, I want an integration test, and I really don't want to run it against a cluster that has to be setup externally: I'd rather the cluster be spun up and shut down by my test itself. I'm happy to throw this contrib/ if you feel strongly about it. I figure it'd be useful to other folks. – Philip
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > If you would, take a quick look to see how I use MiniMRCluster. Do you feel I'm abusing the fact that hdfs-hdfswithmr-test exists?

          No more mapreduce codes in hdfs, please. Having hdfs-with-mr in hdfs is a mistake. It leads to a circular dependence. Indeed, we should move hdfs-with-mr to mapreduce.

          Show
          Tsz Wo Nicholas Sze added a comment - > If you would, take a quick look to see how I use MiniMRCluster. Do you feel I'm abusing the fact that hdfs-hdfswithmr-test exists? No more mapreduce codes in hdfs, please. Having hdfs-with-mr in hdfs is a mistake. It leads to a circular dependence. Indeed, we should move hdfs-with-mr to mapreduce.
          Hide
          Philip Zeyliger added a comment -

          Attaching the 0.20 version. Conveniently, where to place it is not a problem there

          Show
          Philip Zeyliger added a comment - Attaching the 0.20 version. Conveniently, where to place it is not a problem there
          Hide
          Philip Zeyliger added a comment -

          Nicholas,

          Agreed that circular dependencies are to be avoided. I've moved this issue into MAPREDUCE, and spun up a new patch.

          Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it only depends on the FileSystem interfaces)?

          – Philip

          Show
          Philip Zeyliger added a comment - Nicholas, Agreed that circular dependencies are to be avoided. I've moved this issue into MAPREDUCE, and spun up a new patch. Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it only depends on the FileSystem interfaces)? – Philip
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Agreed that circular dependencies are to be avoided. I've moved this issue into MAPREDUCE, and spun up a new patch.
          Thanks, Philip.

          > Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it only depends on the FileSystem interfaces)?
          Theoretically, mapreduce does not depend on hdfs. However, the performance of mapreduce would be bad without hdfs. I guess there is no serious applications using mapreduce without hdfs, except for testing.

          Show
          Tsz Wo Nicholas Sze added a comment - > Agreed that circular dependencies are to be avoided. I've moved this issue into MAPREDUCE, and spun up a new patch. Thanks, Philip. > Do we anticipate a world where MR doesn't depend statically on HDFS (i.e., it only depends on the FileSystem interfaces)? Theoretically, mapreduce does not depend on hdfs. However, the performance of mapreduce would be bad without hdfs. I guess there is no serious applications using mapreduce without hdfs, except for testing.
          Hide
          Todd Lipcon added a comment -

          +1 - I code reviewed this internally before posting, and have also been using it for a couple of weeks, finding it very useful. Happy to post a copy of our internal review discussion if necessary.

          Show
          Todd Lipcon added a comment - +1 - I code reviewed this internally before posting, and have also been using it for a couple of weeks, finding it very useful. Happy to post a copy of our internal review discussion if necessary.
          Hide
          Philip Zeyliger added a comment -

          Submitting patch: our friend Hudson may have gotten confused.

          Does anyone have any objections to exposing Mini*Cluster in this way?

          Thanks,

          – Philip

          Show
          Philip Zeyliger added a comment - Submitting patch: our friend Hudson may have gotten confused. Does anyone have any objections to exposing Mini*Cluster in this way? Thanks, – Philip
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419700/MAPREDUCE-987.patch
          against trunk revision 823227.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419700/MAPREDUCE-987.patch against trunk revision 823227. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/158/console This message is automatically generated.
          Hide
          Chris Douglas added a comment -

          This seems appropriate for the test jar. Small notes:

          • This picks up -D params like the generic parser; would it make sense to also accept -conf? The other params make less sense in this context, though it may be worth considering Tool/ToolRunner
          • It'd be better if sleepForever monitored the Mini*Cluster, rather than waking up every minute for no reason. Not sure if it makes sense to include a poison pill (Path?) + configurable polling interval that might signal an orderly shutdown.
          • If this is intended for tests, should start wait for the TT/DNs to come up before returning?
          Show
          Chris Douglas added a comment - This seems appropriate for the test jar. Small notes: This picks up -D params like the generic parser; would it make sense to also accept -conf? The other params make less sense in this context, though it may be worth considering Tool/ToolRunner It'd be better if sleepForever monitored the Mini*Cluster, rather than waking up every minute for no reason. Not sure if it makes sense to include a poison pill (Path?) + configurable polling interval that might signal an orderly shutdown. If this is intended for tests, should start wait for the TT/DNs to come up before returning?
          Hide
          Allen Wittenauer added a comment -

          There is a patch for this in the Cloudera distribution but it doesn't appear to have been committed upstream. What is the status?

          Show
          Allen Wittenauer added a comment - There is a patch for this in the Cloudera distribution but it doesn't appear to have been committed upstream. What is the status?
          Hide
          Konstantin Boudnik added a comment -

          Phillip, is there any intention to address last comments from Chris, so this can be committed into trunk?

          Show
          Konstantin Boudnik added a comment - Phillip, is there any intention to address last comments from Chris, so this can be committed into trunk?
          Hide
          Ahmed Radwan added a comment -

          Here is an updated patch for trunk, various changes were done due to MR2 changes.

          Show
          Ahmed Radwan added a comment - Here is an updated patch for trunk, various changes were done due to MR2 changes.
          Hide
          Ahmed Radwan added a comment -

          Please note that the patch requires also the patches from the blocker tickets MAPREDUCE-4406 and MAPREDUCE-4407.

          Show
          Ahmed Radwan added a comment - Please note that the patch requires also the patches from the blocker tickets MAPREDUCE-4406 and MAPREDUCE-4407 .
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535470/MAPREDUCE-987.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 2071 javac compiler warnings (more than the trunk's current 2070 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535470/MAPREDUCE-987.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The applied patch generated 2071 javac compiler warnings (more than the trunk's current 2070 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2552//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Here is also the updated patch for branch-1.0.

          Show
          Ahmed Radwan added a comment - Here is also the updated patch for branch-1.0.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12535777/MAPREDUCE-987_branch-1.0.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2561//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12535777/MAPREDUCE-987_branch-1.0.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2561//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          Here is an updated version of the trunk patch based on new MAPREDUCE-4406 change for using properties to control if fixed ports are used or not.

          Show
          Ahmed Radwan added a comment - Here is an updated version of the trunk patch based on new MAPREDUCE-4406 change for using properties to control if fixed ports are used or not.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12536742/MAPREDUCE-987_rev2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2605//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12536742/MAPREDUCE-987_rev2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2605//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1. Ahmed, please add documentation for it (in the yarn-site submodule, in APT format), and we are good to go.

          Show
          Alejandro Abdelnur added a comment - +1. Ahmed, please add documentation for it (in the yarn-site submodule, in APT format), and we are good to go.
          Hide
          Ahmed Radwan added a comment -

          Thanks Tucu! Here is the updated patch with the new docs.

          Show
          Ahmed Radwan added a comment - Thanks Tucu! Here is the updated patch with the new docs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537142/MAPREDUCE-987_rev3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 2067 javac compiler warnings (more than the trunk's current 2066 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:

          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537142/MAPREDUCE-987_rev3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The applied patch generated 2067 javac compiler warnings (more than the trunk's current 2066 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2625//console This message is automatically generated.
          Hide
          Ahmed Radwan added a comment -

          The Jenkins reported test failure doesn't seem to be related to this patch.

          Show
          Ahmed Radwan added a comment - The Jenkins reported test failure doesn't seem to be related to this patch.
          Hide
          Alejandro Abdelnur added a comment -

          Ahmed, a couple of Nits in the docs:

          1

          The sentence:

          + The CLI MiniCLuster can
          + start a full Hadoop cluster including the <<<Yarn>>> <<<ResourceManager>>>,
          + <<<NodeManager(s)>>>, and <<<HDFS>>> <<<NameNode>>> and <<<DataNode(s)>>>, in
          + addition to the <<<MapReduce>>> <<<JobHistoryServer>>>.

          it is a bit too complex, why not just replace it with:

          The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters.

          2

          Instead using VERSION and a paragraph explaining what VERSION has to be replaced with, you can use $

          {project.version}

          , within *.apt.vm files you can use Maven variables.

          Show
          Alejandro Abdelnur added a comment - Ahmed, a couple of Nits in the docs: 1 The sentence: + The CLI MiniCLuster can + start a full Hadoop cluster including the <<<Yarn>>> <<<ResourceManager>>>, + <<<NodeManager(s)>>>, and <<<HDFS>>> <<<NameNode>>> and <<<DataNode(s)>>>, in + addition to the <<<MapReduce>>> <<<JobHistoryServer>>>. it is a bit too complex, why not just replace it with: The CLI MiniCluster starts both a YARN/MapReduce & HDFS clusters. 2 Instead using VERSION and a paragraph explaining what VERSION has to be replaced with, you can use $ {project.version} , within *.apt.vm files you can use Maven variables.
          Hide
          Ahmed Radwan added a comment -

          Thanks Tucu! Here is the updated patch incorporating your comments.

          Show
          Ahmed Radwan added a comment - Thanks Tucu! Here is the updated patch incorporating your comments.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537227/MAPREDUCE-987_rev4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 2067 javac compiler warnings (more than the trunk's current 2066 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site:

          org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
          org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537227/MAPREDUCE-987_rev4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The applied patch generated 2067 javac compiler warnings (more than the trunk's current 2066 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site: org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2628//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          +1

          Show
          Alejandro Abdelnur added a comment - +1
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2575 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2575/)
          MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2575 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2575/ ) MAPREDUCE-987 . Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2510 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2510/)
          MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020)

          Result = SUCCESS
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2510 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2510/ ) MAPREDUCE-987 . Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2531 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2531/)
          MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2531 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2531/ ) MAPREDUCE-987 . Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1111/)
          MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1111/ ) MAPREDUCE-987 . Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/)
          MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020)

          Result = FAILURE
          tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1143 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1143/ ) MAPREDUCE-987 . Exposing MiniDFS and MiniMR clusters as a single process command-line. (ahmed via tucu) (Revision 1364020) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364020 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/MiniHadoopClusterManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/test/MapredTestDriver.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CLIMiniCluster.apt.vm /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/index.apt.vm
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Ahmed. Committed to trunk and branch-2, I'll leave it open until we have the patch with docs for branch-1 and we commit it there.

          Show
          Alejandro Abdelnur added a comment - Thanks Ahmed. Committed to trunk and branch-2, I'll leave it open until we have the patch with docs for branch-1 and we commit it there.
          Hide
          Ahmed Radwan added a comment -

          Thanks Tucu! Here is the updated patch for branch-1.0 including the new docs.

          Show
          Ahmed Radwan added a comment - Thanks Tucu! Here is the updated patch for branch-1.0 including the new docs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12537535/MAPREDUCE-987_branch-1.0_rev2.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2646//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537535/MAPREDUCE-987_branch-1.0_rev2.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2646//console This message is automatically generated.
          Hide
          Alejandro Abdelnur added a comment -

          Thanks Philip & Ahmed. Committed to branch-1 as well.

          Show
          Alejandro Abdelnur added a comment - Thanks Philip & Ahmed. Committed to branch-1 as well.

            People

            • Assignee:
              Ahmed Radwan
              Reporter:
              Philip Zeyliger
            • Votes:
              0 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development