Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. patch-372-4.txt
      95 kB
      Amareshwari Sriramadasu
    2. patch-372-3.txt
      94 kB
      Amareshwari Sriramadasu
    3. patch-372-2.txt
      95 kB
      Amareshwari Sriramadasu
    4. patch-372-1.txt
      80 kB
      Amareshwari Sriramadasu
    5. patch-372.txt
      57 kB
      Amareshwari Sriramadasu
    6. mapred-372.patch
      15 kB
      Jothi Padmanabhan
    7. mapred-372.patch
      64 kB
      Jothi Padmanabhan
    8. mapred-372.patch
      92 kB
      Jothi Padmanabhan

      Issue Links

        Activity

        Hide
        Amareshwari Sriramadasu added a comment -

        As per my understanding "In ChainMapper, the Mapper classes are invoked in a chained (or piped) fashion, the output of the first becomes the input of the second, and so on until the last Mapper, the output of the last Mapper will be written to the task's output." This is mainly to reduce disk IO.

        With new api interface I see an issue to achieve similar functionality.
        New api Mapper interface looks like the following:

        protected void setup(Context context);
        protected void map(KEYIN key, VALUEIN value, Context context);
        protected void cleanup(Context context);
        public void run(Context context);
        

        If we want to chain mappers, we have to chain them in run method(), since run() is the only public method. But Mapper.run() is going to run map on all (key,value) pairs. Then Chaining would mean running different map only jobs.

        One solution I could see is :
        1. Make setup(), map() and cleanup() methods public.
        2. Do chaining at map(). But User's Mapper.run() implementation is not considered.

        Thoughts?

        Show
        Amareshwari Sriramadasu added a comment - As per my understanding "In ChainMapper, the Mapper classes are invoked in a chained (or piped) fashion, the output of the first becomes the input of the second, and so on until the last Mapper, the output of the last Mapper will be written to the task's output." This is mainly to reduce disk IO. With new api interface I see an issue to achieve similar functionality. New api Mapper interface looks like the following: protected void setup(Context context); protected void map(KEYIN key, VALUEIN value, Context context); protected void cleanup(Context context); public void run(Context context); If we want to chain mappers, we have to chain them in run method(), since run() is the only public method. But Mapper.run() is going to run map on all (key,value) pairs. Then Chaining would mean running different map only jobs. One solution I could see is : 1. Make setup(), map() and cleanup() methods public. 2. Do chaining at map(). But User's Mapper.run() implementation is not considered. Thoughts?
        Hide
        Owen O'Malley added a comment -

        We really should make the context objects into interfaces.

        I agree that the new API makes this harder, because the run method means you have to allow a pull model instead of a push. The easiest way to do it would be to have a blocking queue and each stage in the pipeline is a separate thread. So the first mapper would read from the RecordReader (via the "real" context) and write outputs into a BlockingQueue. The next step would pull from that BlockingQueue and write to the next BlockingQueue and so on until the last wrote to the "real" context. Thus each thread is in the "run" method of each pipeline.

        Issues include:
        1. Needing a thread per a step.
        2. Need to clone the keys and values between steps.
        3. Need to figure out the size of the queues. Probably 1 to start with...

        Show
        Owen O'Malley added a comment - We really should make the context objects into interfaces. I agree that the new API makes this harder, because the run method means you have to allow a pull model instead of a push. The easiest way to do it would be to have a blocking queue and each stage in the pipeline is a separate thread. So the first mapper would read from the RecordReader (via the "real" context) and write outputs into a BlockingQueue. The next step would pull from that BlockingQueue and write to the next BlockingQueue and so on until the last wrote to the "real" context. Thus each thread is in the "run" method of each pipeline. Issues include: 1. Needing a thread per a step. 2. Need to clone the keys and values between steps. 3. Need to figure out the size of the queues. Probably 1 to start with...
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching patch for review.

        Patch implements ChainMapper and ChainReducer as proposed by Owen.

        The key and values are always passed by value from one element of the chain to the next, because we
        clone the keys and values between steps. This removes the option to pass by value or by reference, that was available in old api.

        Show
        Amareshwari Sriramadasu added a comment - Attaching patch for review. Patch implements ChainMapper and ChainReducer as proposed by Owen. The key and values are always passed by value from one element of the chain to the next, because we clone the keys and values between steps. This removes the option to pass by value or by reference, that was available in old api.
        Hide
        Amareshwari Sriramadasu added a comment -

        submitting for hudson

        Show
        Amareshwari Sriramadasu added a comment - submitting for hudson
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12414890/patch-372.txt
        against trunk revision 800329.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414890/patch-372.txt against trunk revision 800329. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/439/console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12414890/patch-372.txt
        against trunk revision 800684.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12414890/patch-372.txt against trunk revision 800684. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/440/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching patch with
        1. Added testcases for submission errors, single mapper/reducer in chain, one of the maps throwing an exception thus breaking the chain, one of the maps eats up all the input and doesn't output any.
        2. Made helper class org.apache.hadoop.mapreduce.lib.Chain a public class. And changed old api code to reuse the code from new api.
        3. Fixed some indentation issues
        4. Added more documentation.

        Show
        Amareshwari Sriramadasu added a comment - Attaching patch with 1. Added testcases for submission errors, single mapper/reducer in chain, one of the maps throwing an exception thus breaking the chain, one of the maps eats up all the input and doesn't output any. 2. Made helper class org.apache.hadoop.mapreduce.lib.Chain a public class. And changed old api code to reuse the code from new api. 3. Fixed some indentation issues 4. Added more documentation.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12415700/patch-372-1.txt
        against trunk revision 801517.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12415700/patch-372-1.txt against trunk revision 801517. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-vesta.apache.org/445/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Test failure TestTaskTrackerMemoryManager is not related to the patch. It is due to HADOOP-6075

        Show
        Amareshwari Sriramadasu added a comment - Test failure TestTaskTrackerMemoryManager is not related to the patch. It is due to HADOOP-6075
        Hide
        Devaraj Das added a comment -

        I just committed this. Thanks, Amareshwari!

        Show
        Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #46 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/46/ )
        Hide
        Arun C Murthy added a comment -

        I reverted this in light of MAPREDUCE-917 (hence, MAPREDUCE-901 too).

        Show
        Arun C Murthy added a comment - I reverted this in light of MAPREDUCE-917 (hence, MAPREDUCE-901 too).
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #14 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/14/)
        Reverted .

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #14 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/14/ ) Reverted .
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #75 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/75/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #75 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/75/ )
        Hide
        Jothi Padmanabhan added a comment -

        With more than a generous bout of help from Amareshwari and Devaraj, attaching a patch that modifies the original one to not rely on the getters for the context attributes. The approach here is to have a sort of pseudo copy constructor that creates a context object based on attributes from the passed in context object.

        If we do not want to proceed with this copy-constructor approach, we might just have to postpone this patch until MAPREDUCE-954 is done. Thoughts?

        Show
        Jothi Padmanabhan added a comment - With more than a generous bout of help from Amareshwari and Devaraj, attaching a patch that modifies the original one to not rely on the getters for the context attributes. The approach here is to have a sort of pseudo copy constructor that creates a context object based on attributes from the passed in context object. If we do not want to proceed with this copy-constructor approach, we might just have to postpone this patch until MAPREDUCE-954 is done. Thoughts?
        Hide
        Jothi Padmanabhan added a comment -

        Just to make it easier, here is the pseudo copy constructor that we needed to add

          /**
           * A pseudo copy constructor. Set the record writer and configuration values
           * to the passed arguments and the rest of the fields from the other
           * context object. Used by Chain Reducer
           * @param context
           * @param output
           * @param conf
           * @throws InterruptedException
           * @throws IOException
           */
          public ReduceContext(ReduceContext<KEYIN, VALUEIN, ?, ?> context,
              RecordWriter<KEYOUT,VALUEOUT> output,
              Configuration conf)
          throws InterruptedException, IOException{
            super(conf, context.taskid, output, context.getOutputCommitter(),
                context.getReporter());
            this.input = context.input;
            this.inputCounter = context.inputCounter;
            this.comparator = context.comparator;
            this.serializationFactory = context.serializationFactory;
            this.keyDeserializer = context.keyDeserializer;
            this.valueDeserializer = context.valueDeserializer;
            this.keyClass = context.keyClass;
            this.valueClass = context.valueClass;
            this.conf = conf;
            this.taskid = context.taskid;
            this.hasMore = context.hasMore;
            this.buffer = context.buffer;
          }
        
        
        Show
        Jothi Padmanabhan added a comment - Just to make it easier, here is the pseudo copy constructor that we needed to add /** * A pseudo copy constructor. Set the record writer and configuration values * to the passed arguments and the rest of the fields from the other * context object. Used by Chain Reducer * @param context * @param output * @param conf * @ throws InterruptedException * @ throws IOException */ public ReduceContext(ReduceContext<KEYIN, VALUEIN, ?, ?> context, RecordWriter<KEYOUT,VALUEOUT> output, Configuration conf) throws InterruptedException, IOException{ super (conf, context.taskid, output, context.getOutputCommitter(), context.getReporter()); this .input = context.input; this .inputCounter = context.inputCounter; this .comparator = context.comparator; this .serializationFactory = context.serializationFactory; this .keyDeserializer = context.keyDeserializer; this .valueDeserializer = context.valueDeserializer; this .keyClass = context.keyClass; this .valueClass = context.valueClass; this .conf = conf; this .taskid = context.taskid; this .hasMore = context.hasMore; this .buffer = context.buffer; }
        Hide
        Arun C Murthy added a comment -

        Jothi, the patch you attached seems to be missing files... can you please upload a more complete one?

        Show
        Arun C Murthy added a comment - Jothi, the patch you attached seems to be missing files... can you please upload a more complete one?
        Hide
        Jothi Padmanabhan added a comment -

        Sorry, forgot to do svn add before. Here is the full patch

        Show
        Jothi Padmanabhan added a comment - Sorry, forgot to do svn add before. Here is the full patch
        Hide
        Jothi Padmanabhan added a comment -

        Patch synched with trunk. The individual mappers and reducer now create and use a context that wraps the original context and overrides recordreader/writer and conf as necessary.

        Show
        Jothi Padmanabhan added a comment - Patch synched with trunk. The individual mappers and reducer now create and use a context that wraps the original context and overrides recordreader/writer and conf as necessary.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12421622/mapred-372.patch
        against trunk revision 819740.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421622/mapred-372.patch against trunk revision 819740. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/151/console This message is automatically generated.
        Hide
        Jothi Padmanabhan added a comment -

        -1 core tests. The patch failed core unit tests

        TestCopyFiles is a known issue
        Failure with TestMultipleLevelCaching looks spurious. This is unrelated to the code and the test passes on my local box.

        Show
        Jothi Padmanabhan added a comment - -1 core tests. The patch failed core unit tests TestCopyFiles is a known issue Failure with TestMultipleLevelCaching looks spurious. This is unrelated to the code and the test passes on my local box.
        Hide
        Sharad Agarwal added a comment -

        Some comments:
        Access to Chain#throwable must be thread safe.
        Poll in ChainRecordReader#readFromQueue() and ChainRecordWriter#writeToQueue() is not efficient. Can we use wait/notify mechanism instead ?

        Show
        Sharad Agarwal added a comment - Some comments: Access to Chain#throwable must be thread safe. Poll in ChainRecordReader#readFromQueue() and ChainRecordWriter#writeToQueue() is not efficient. Can we use wait/notify mechanism instead ?
        Hide
        Tom White added a comment -

        Moving to open while comments are addressed.

        Show
        Tom White added a comment - Moving to open while comments are addressed.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch with review comments incorporated.

        Show
        Amareshwari Sriramadasu added a comment - Patch with review comments incorporated.
        Hide
        Sharad Agarwal added a comment -

        Looked at the ChainBlockingQueue part of the code. Some comments:
        1. Can we avoid the casting in Chain#stopAllThreads? One way could be to override interrupt() in MapRunner and ReduceRunner. Also interruptAllThreads would be a better name IMO.
        2. I think instead of interrupting the runners and then calling interrupt on both readers and writers, it would be simpler we can directly interrupt all the blocking queues.

        Show
        Sharad Agarwal added a comment - Looked at the ChainBlockingQueue part of the code. Some comments: 1. Can we avoid the casting in Chain#stopAllThreads? One way could be to override interrupt() in MapRunner and ReduceRunner. Also interruptAllThreads would be a better name IMO. 2. I think instead of interrupting the runners and then calling interrupt on both readers and writers, it would be simpler we can directly interrupt all the blocking queues.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12426877/patch-372-2.txt
        against trunk revision 887135.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12426877/patch-372-2.txt against trunk revision 887135. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/164/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Incorporated comment #2. This removes the code to do with comment #1.
        Renamed stopAllThreads to interruptAllThreads.

        Show
        Amareshwari Sriramadasu added a comment - Incorporated comment #2. This removes the code to do with comment #1. Renamed stopAllThreads to interruptAllThreads.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427143/patch-372-3.txt
        against trunk revision 887844.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427143/patch-372-3.txt against trunk revision 887844. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/296/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        -1 contrib tests

        Is due to MAPREDUCE-1124

        Show
        Amareshwari Sriramadasu added a comment - -1 contrib tests Is due to MAPREDUCE-1124
        Hide
        Amareshwari Sriramadasu added a comment -

        Incorporated offline review comments from Sharad. The comments include:

        1. The data structures threads and blockingQueues may not be thread safe. Populate them before starting the threads.
        2. Extend TestMapReduceChain to verify the order in-which the maps are executed.

        Show
        Amareshwari Sriramadasu added a comment - Incorporated offline review comments from Sharad. The comments include: 1. The data structures threads and blockingQueues may not be thread safe. Populate them before starting the threads. 2. Extend TestMapReduceChain to verify the order in-which the maps are executed.
        Hide
        Sharad Agarwal added a comment -

        +1

        Show
        Sharad Agarwal added a comment - +1
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427693/patch-372-4.txt
        against trunk revision 889496.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427693/patch-372-4.txt against trunk revision 889496. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/188/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        test failures are not related to the patch. Re-submitting for hudson

        Show
        Amareshwari Sriramadasu added a comment - test failures are not related to the patch. Re-submitting for hudson
        Hide
        Amareshwari Sriramadasu added a comment -

        All core and contrib tests passed on my machine

        Show
        Amareshwari Sriramadasu added a comment - All core and contrib tests passed on my machine
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12427693/patch-372-4.txt
        against trunk revision 889786.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 9 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12427693/patch-372-4.txt against trunk revision 889786. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/324/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        test failures are due to MAPREDUCE-1275. All the core and contrib tests passed on my machine.

        Show
        Amareshwari Sriramadasu added a comment - test failures are due to MAPREDUCE-1275 . All the core and contrib tests passed on my machine.
        Hide
        Sharad Agarwal added a comment -

        I just committed this. Thanks Amareshwari.

        Show
        Sharad Agarwal added a comment - I just committed this. Thanks Amareshwari.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/198/)
        . Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new mapreduce api. Contributed by Amareshwari Sriramadasu.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #198 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/198/ ) . Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer to use new mapreduce api. Contributed by Amareshwari Sriramadasu.

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development