Hadoop Common
  1. Hadoop Common
  2. HADOOP-3684

The data_join should allow the user to implement a customer cloning function

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Allowed user to overwrite clone function in a subclass of TaggedMapOutput class.

      Description

      Currently, the framework uses serialization/deserialization to clone the values passed to the resuce function.
      This amounts to a very heavy weight deep copy of the value objects.
      That is way too expensive. Although that may be a generic way to work for all possible value classes, thus good as a default way,
      the framework should allow the user to implemet an application specific yet efficient cloning function.

      1. H-3684.txt
        2 kB
        Runping Qi

        Activity

        Runping Qi created issue -
        Hide
        Runping Qi added a comment -

        Attach a simple patch.

        This patch allows the user to overwrite the clone(JobConf job)
        method in the subclass of TaggedMapOutputclass.

        Show
        Runping Qi added a comment - Attach a simple patch. This patch allows the user to overwrite the clone(JobConf job) method in the subclass of TaggedMapOutputclass.
        Runping Qi made changes -
        Field Original Value New Value
        Attachment H-3684.txt [ 12385153 ]
        Runping Qi made changes -
        Release Note make it possible for the user to overwrite clone function in a subclass of TaggedMapOutput class
        Fix Version/s 0.19.0 [ 12313211 ]
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12385153/H-3684.txt
        against trunk revision 673517.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        -1 patch. The patch command could not apply the patch.

        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2786/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385153/H-3684.txt against trunk revision 673517. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2786/console This message is automatically generated.
        Runping Qi made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Runping Qi made changes -
        Attachment H-3684.txt [ 12385153 ]
        Runping Qi made changes -
        Attachment H-3684.txt [ 12385161 ]
        Hide
        Runping Qi added a comment -


        regenerate the patch

        Show
        Runping Qi added a comment - regenerate the patch
        Runping Qi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12385161/H-3684.txt
        against trunk revision 673517.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no tests are needed for this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12385161/H-3684.txt against trunk revision 673517. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2789/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        +1

        Show
        Chris Douglas added a comment - +1
        Chris Douglas made changes -
        Hadoop Flags [Reviewed]
        Assignee Runping Qi [ runping ]
        Description
        Currently, the framework uses serialization/deserialization to clone the values passed to the resuce function.
        This amounts to a very heavy weight deep copy of the value objects.
        That is way too expensive. Although that may be a generic way to work for all possible value classes, thus good as a default way,
        the framework should allow the user to implemet an application specific yet efficient cloning function.
         
        Currently, the framework uses serialization/deserialization to clone the values passed to the resuce function.
        This amounts to a very heavy weight deep copy of the value objects.
        That is way too expensive. Although that may be a generic way to work for all possible value classes, thus good as a default way,
        the framework should allow the user to implemet an application specific yet efficient cloning function.
         
        Chris Douglas made changes -
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Hide
        Chris Douglas added a comment -

        I just committed this. Thanks, Runping

        Show
        Chris Douglas added a comment - I just committed this. Thanks, Runping
        Chris Douglas made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hide
        Hudson added a comment -
        Show
        Hudson added a comment - Integrated in Hadoop-trunk #581 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/ )
        Robert Chansler made changes -
        Release Note make it possible for the user to overwrite clone function in a subclass of TaggedMapOutput class Allowed user to overwrite clone function in a subclass of TaggedMapOutput class.
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        3h 22m 1 Runping Qi 03/Jul/08 01:44
        Open Open Patch Available Patch Available
        21m 27s 2 Runping Qi 03/Jul/08 01:46
        Patch Available Patch Available Resolved Resolved
        7d 20h 29m 1 Chris Douglas 10/Jul/08 22:16
        Resolved Resolved Closed Closed
        133d 1h 22m 1 Nigel Daley 20/Nov/08 23:38

          People

          • Assignee:
            Runping Qi
            Reporter:
            Runping Qi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development