Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2225

MultipleOutputs should not require the use of 'Writable'

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.23.0
    • Component/s: job submission
    • Labels:
      None
    • Environment:

      Linux

    • Hadoop Flags:
      Reviewed
    • Release Note:
      MultipleOutputs should not require the use/check of 'Writable' interfaces in key and value classes.
    • Tags:
      multipleoutputs, writable, serialization

      Description

      MultipleOutputs right now requires for Key/Value classes to utilize the Writable and WritableComparable interfaces, and fails if the associated key/value classes aren't doing so.

      With support for alternates like Avro serialization, using Writables isn't necessary and thus the MO class must not strictly check for them.

      And since comparators may be given separately, key class doesn't need to be checked for implementing a comparable (although it is good design if the key class does implement Comparable at least).

      Am not sure if this brings about an incompatible change (does Java have BIC? No idea).

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #631 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/631/ )
        Hide
        Tom White added a comment -

        I've just committed this. Thanks Harsh!

        Show
        Tom White added a comment - I've just committed this. Thanks Harsh!
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12472201/multipleoutputs.nowritables.r2.diff
        against trunk revision 1075569.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472201/multipleoutputs.nowritables.r2.diff against trunk revision 1075569. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/103//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Sure, it should avoid a raw types warning I think? Here's a patch for that update.

        Show
        Harsh J added a comment - Sure, it should avoid a raw types warning I think? Here's a patch for that update.
        Hide
        Tom White added a comment -

        This looks good to me. Should the return type of o.a.h.mapred.lib.MultipleOutputs be Class<?>, not just the raw type Class, like its new API equivalent?

        Show
        Tom White added a comment - This looks good to me. Should the return type of o.a.h.mapred.lib.MultipleOutputs be Class<?>, not just the raw type Class, like its new API equivalent?
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12467578/multipleoutputs.nowritables.r2.diff
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 6 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467578/multipleoutputs.nowritables.r2.diff against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/44//console This message is automatically generated.
        Hide
        Harsh J added a comment -

        Setting to BLOCKER as per Nigel's mail.

        Show
        Harsh J added a comment - Setting to BLOCKER as per Nigel's mail.
        Hide
        Harsh J added a comment -

        Perhaps this could go in 0.22 itself, if improvements are still accepted?

        Show
        Harsh J added a comment - Perhaps this could go in 0.22 itself, if improvements are still accepted?
        Hide
        Harsh J added a comment -

        Setting state to PA.

        'ant test-patch' passes with:

             [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 6 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) warnings.
             [exec] 
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec] 
             [exec]     +1 system test framework.  The patch passed system test framework compile.
             [exec] 
             [exec] 
             [exec] 
             [exec] 
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
             [exec] 
             [exec] 
        
        BUILD SUCCESSFUL
        Total time: 23 minutes 14 seconds
        
        Show
        Harsh J added a comment - Setting state to PA. 'ant test-patch' passes with: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD SUCCESSFUL Total time: 23 minutes 14 seconds
        Hide
        Harsh J added a comment -

        New patch, with test cases for testing JavaSerialization with MO.

        Show
        Harsh J added a comment - New patch, with test cases for testing JavaSerialization with MO.
        Hide
        Harsh J added a comment -

        Attached a patch that removes Writable checks out of MultipleOutputs (In both mapred and mapreduce packages).

        Ran the TestMultipleOutputs with successful results:
        run-test-mapred-all-withtestcaseonly:
        [mkdir] Created dir: /home/hack/projects/work/MapReduce/build/test/data
        [mkdir] Created dir: /home/hack/projects/work/MapReduce/build/test/logs
        [copy] Copying 1 file to /home/hack/projects/work/MapReduce/build/test/extraconf
        [copy] Copying 1 file to /home/hack/projects/work/MapReduce/build/test/extraconf
        [junit] WARNING: multiple versions of ant detected in path for junit
        [junit] jar:file:/usr/share/java/apache-ant/lib/ant.jar!/org/apache/tools/ant/Project.class
        [junit] and jar:file:/home/hack/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
        [junit] Running org.apache.hadoop.mapred.lib.TestMultipleOutputs
        [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 3.52 sec

        Show
        Harsh J added a comment - Attached a patch that removes Writable checks out of MultipleOutputs (In both mapred and mapreduce packages). Ran the TestMultipleOutputs with successful results: run-test-mapred-all-withtestcaseonly: [mkdir] Created dir: /home/hack/projects/work/MapReduce/build/test/data [mkdir] Created dir: /home/hack/projects/work/MapReduce/build/test/logs [copy] Copying 1 file to /home/hack/projects/work/MapReduce/build/test/extraconf [copy] Copying 1 file to /home/hack/projects/work/MapReduce/build/test/extraconf [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar: file:/usr/share/java/apache-ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar: file:/home/hack/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.mapred.lib.TestMultipleOutputs [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 3.52 sec

          People

          • Assignee:
            Harsh J
            Reporter:
            Harsh J
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 1m
              1m
              Remaining:
              Remaining Estimate - 1m
              1m
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development