Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-641

WordCount unit test plus a helper class to facilitate testing Mappers and Reducers

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Not a Problem
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: examples
    • Labels:
      None

      Description

      There are to pieces to this The first is a test for WordCount, not because word count actually needed one but because it would be useful to beginners to have an example of how to unit test Mappers and Reducers.

      The second piece is AOutputCollector and it's associated unit test TestAOutputCollector. This is an abstract class that can be quickly extended by a stub OutputCollector in your unit tests to collect the output from your Mapper and Reducer tests and make it available for easy retreival when testing to see if the fourth key that was emitted was the one you expected. I think that this would be a useful tool to have in the main test folder but wasn't sure where would be best to put it. Also, since nothing else in Hadoop uses Hungarian notation you'll probably want to rename it. I didn't because I'm not confident about the naming conventions here and figured that since it and its test probably wouldn't end up living in the same folder as WordCount that you could just rename it when you moved it.

        Activity

        Hide
        Harsh J added a comment -

        Resolving this since it has gone stale. Please re-open if this is still required.

        (P.s. Hadoop uses Mockito for tests now, if its required to note.)

        Show
        Harsh J added a comment - Resolving this since it has gone stale. Please re-open if this is still required. (P.s. Hadoop uses Mockito for tests now, if its required to note.)
        Hide
        Devaraj Das added a comment -

        Cancelling the patch for this bug since Tom has a comment on the approach taken to address the issue.

        Show
        Devaraj Das added a comment - Cancelling the patch for this bug since Tom has a comment on the approach taken to address the issue.
        Hide
        Tom White added a comment -

        Sorry to come to this so late, but I would vote to use a mock object library (jMock) to write tests for mappers and reducers as you argued for on the user list (http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg02261.html). Using jMock is much more concise than the approach in this patch (see http://www.matthewekent.com/?p=15), although I admit it does have a bit of a learning curve.

        Show
        Tom White added a comment - Sorry to come to this so late, but I would vote to use a mock object library (jMock) to write tests for mappers and reducers as you argued for on the user list ( http://www.mail-archive.com/hadoop-user@lucene.apache.org/msg02261.html ). Using jMock is much more concise than the approach in this patch (see http://www.matthewekent.com/?p=15 ), although I admit it does have a bit of a learning curve.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12366628/WordCountTestExampleAndOutputCollectorTestHelper.patch
        against trunk revision r580166.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12366628/WordCountTestExampleAndOutputCollectorTestHelper.patch against trunk revision r580166. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/836/console This message is automatically generated.
        Hide
        Kate Rhodes added a comment -

        unit tests should pass in every environment now.
        added getInjectedKeyAt (...) and getInjectedValueAt(...) to OutputCollectorHelper

        Show
        Kate Rhodes added a comment - unit tests should pass in every environment now. added getInjectedKeyAt (...) and getInjectedValueAt(...) to OutputCollectorHelper
        Hide
        Kate Rhodes added a comment -

        Fixed key ordering bug in previous patch

        Show
        Kate Rhodes added a comment - Fixed key ordering bug in previous patch
        Hide
        Kate Rhodes added a comment -

        failed it's own test. will investigate fix and resubmit. It did pass before submitting...

        Show
        Kate Rhodes added a comment - failed it's own test. will investigate fix and resubmit. It did pass before submitting...
        Hide
        Kate Rhodes added a comment -

        how ironic. I'll see why it failed this but not locally and resubmit.

        Show
        Kate Rhodes added a comment - how ironic. I'll see why it failed this but not locally and resubmit.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12366564/WordCountTestExampleAndOutputCollectorTestHelper.patch
        against trunk revision r579410.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs +1. The patch does not introduce any new Findbugs warnings.

        core tests -1. The patch failed core unit tests.

        contrib tests +1. The patch passed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12366564/WordCountTestExampleAndOutputCollectorTestHelper.patch against trunk revision r579410. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/829/console This message is automatically generated.
        Hide
        Kate Rhodes added a comment -

        Same as ticket except things have been moved into
        src/test/org/apache/hadoop/examples
        and the abstract helper class has been renamed to OutputHelperCollector

        Show
        Kate Rhodes added a comment - Same as ticket except things have been moved into src/test/org/apache/hadoop/examples and the abstract helper class has been renamed to OutputHelperCollector
        Hide
        Kate Rhodes added a comment -

        A src/test/org/apache/hadoop/examples
        A src/test/org/apache/hadoop/examples/TestWordCount.java
        A src/test/org/apache/hadoop/examples/TestOutputCollectorHelper.java
        A src/test/org/apache/hadoop/examples/OutputCollectorHelper.java

        Show
        Kate Rhodes added a comment - A src/test/org/apache/hadoop/examples A src/test/org/apache/hadoop/examples/TestWordCount.java A src/test/org/apache/hadoop/examples/TestOutputCollectorHelper.java A src/test/org/apache/hadoop/examples/OutputCollectorHelper.java
        Hide
        Kate Rhodes added a comment -

        better patch coming

        Show
        Kate Rhodes added a comment - better patch coming
        Hide
        Kate Rhodes added a comment -

        the contrib tests -1 for unit tests is faulty.
        this patch WAS unit tests. 1 unit test, one new class (to facilitate testing), and one test for the new class.

        Show
        Kate Rhodes added a comment - the contrib tests -1 for unit tests is faulty. this patch WAS unit tests. 1 unit test, one new class (to facilitate testing), and one test for the new class.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12366410/WordCountTestExampleAndOutputCollectorTestHelper.patch
        against trunk revision r578348.

        @author +1. The patch does not contain any @author tags.

        javadoc +1. The javadoc tool did not generate any warning messages.

        javac +1. The applied patch does not generate any new compiler warnings.

        findbugs -1. The patch appears to introduce 4 new Findbugs warnings.

        core tests +1. The patch passed core unit tests.

        contrib tests -1. The patch failed contrib unit tests.

        Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/testReport/
        Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12366410/WordCountTestExampleAndOutputCollectorTestHelper.patch against trunk revision r578348. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs -1. The patch appears to introduce 4 new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/810/console This message is automatically generated.
        Hide
        Kate Rhodes added a comment -

        Requesting code review of pach as per suggestion at
        http://wiki.apache.org/lucene-hadoop/HowToContribute

        Please see the tickets notes about the AOutputCollector class and TestAOutputCollector needing to be renamed to something more in-line with Hadoop naming conventions and probably moved to a more appropriate directory.

        Show
        Kate Rhodes added a comment - Requesting code review of pach as per suggestion at http://wiki.apache.org/lucene-hadoop/HowToContribute Please see the tickets notes about the AOutputCollector class and TestAOutputCollector needing to be renamed to something more in-line with Hadoop naming conventions and probably moved to a more appropriate directory.
        Hide
        Kate Rhodes added a comment -

        this patch is against r 578500 and it's contents are described in the ticket

        Show
        Kate Rhodes added a comment - this patch is against r 578500 and it's contents are described in the ticket

          People

          • Assignee:
            Unassigned
            Reporter:
            Kate Rhodes
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development