Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2584

Check for serializers early, and give out more information regarding missing serializers

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: task
    • Labels:
    • Target Version/s:
    • Release Note:
      Check for proper Key/Value serializers during submission of MR jobs.
    • Tags:
      serializers

      Description

      As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

      This issue covers the MapReduce parts of the improvements (made to IFile, MapOutputBuffer, etc. and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

      1. MAPREDUCE-2584.r9.diff
        9 kB
        Harsh J
      2. MAPREDUCE-2584.r8.diff
        8 kB
        Harsh J
      3. MAPREDUCE-2584.r7.diff
        7 kB
        Harsh J
      4. MAPREDUCE-2584.r7.diff
        7 kB
        Harsh J
      5. MAPREDUCE-2584.r6.diff
        8 kB
        Harsh J
      6. MAPREDUCE-2584.r5.diff
        8 kB
        Harsh J
      7. MAPREDUCE-2584.r4.diff
        7 kB
        Harsh J
      8. MAPREDUCE-2584.r3.diff
        7 kB
        Harsh J
      9. MAPREDUCE-2584.r2.diff
        5 kB
        Harsh J

        Issue Links

          Activity

          Harsh J created issue -
          Harsh J made changes -
          Field Original Value New Value
          Link This issue depends on HADOOP-7328 [ HADOOP-7328 ]
          Hide
          Harsh J added a comment -

          Will require the changes from HADOOP-7328. (Linked)

          Show
          Harsh J added a comment - Will require the changes from HADOOP-7328 . (Linked)
          Hide
          Harsh J added a comment -

          Please review patch at https://reviews.apache.org/r/884/

          Show
          Harsh J added a comment - Please review patch at https://reviews.apache.org/r/884/
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Harsh J added a comment -

          Sorry, that should be https://reviews.apache.org/r/885/

          Show
          Harsh J added a comment - Sorry, that should be https://reviews.apache.org/r/885/
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/
          -----------------------------------------------------------

          Review request for hadoop-mapreduce.

          Summary
          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.
          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs


          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing
          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/#review806
          -----------------------------------------------------------

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1758>

          nit: the javadoc style guide says this should be:
          @param job the job to check for

          (so it formats nicely)

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1759>

          I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1760>

          same comments apply

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1761>

          rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."?

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1762>

          missing a space in this exception method.

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java
          <https://reviews.apache.org/r/885/#comment1763>

          add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps?

          • Todd

          On 2011-06-11 22:16:24, Harsh Chouraria wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/885/

          -----------------------------------------------------------

          (Updated 2011-06-11 22:16:24)

          Review request for hadoop-mapreduce.

          Summary

          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.

          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs

          -----

          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing

          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/#review806 ----------------------------------------------------------- src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1758 > nit: the javadoc style guide says this should be: @param job the job to check for (so it formats nicely) src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1759 > I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1760 > same comments apply src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1761 > rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."? src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1762 > missing a space in this exception method. src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java < https://reviews.apache.org/r/885/#comment1763 > add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps? Todd On 2011-06-11 22:16:24, Harsh Chouraria wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-11 22:16:24) Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs ----- src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/
          -----------------------------------------------------------

          (Updated 2011-06-12 04:36:10.322990)

          Review request for hadoop-mapreduce.

          Changes
          -------

          Fixes per Todd's review.

          Summary
          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.
          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs (updated)


          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing
          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-12 04:36:10.322990) Review request for hadoop-mapreduce. Changes ------- Fixes per Todd's review. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs (updated) src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 400

          > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line400>

          >

          > nit: the javadoc style guide says this should be:

          > @param job the job to check for

          >

          > (so it formats nicely)

          Done.

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 401-403

          > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line401>

          >

          > I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines

          Agreed. It came on by default in Eclipse, and I had let them be instead of filling it up. Removed.

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 417-420

          > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line417>

          >

          > same comments apply

          Removed.

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 439-440

          > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line439>

          >

          > rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."?

          Fixed to a more direct message. Sometimes my commenting language is pretty verbose and confusing, thanks for correcting!

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 470

          > <https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line470>

          >

          > missing a space in this exception method.

          Added.

          On 2011-06-12 02:31:03, Todd Lipcon wrote:

          > src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java, line 87

          > <https://reviews.apache.org/r/885/diff/1/?file=21002#file21002line87>

          >

          > add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps?

          Added as well.

          • Harsh

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/#review806
          -----------------------------------------------------------

          On 2011-06-11 22:16:24, Harsh Chouraria wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/885/

          -----------------------------------------------------------

          (Updated 2011-06-11 22:16:24)

          Review request for hadoop-mapreduce.

          Summary

          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.

          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs

          -----

          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing

          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 400 > < https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line400 > > > nit: the javadoc style guide says this should be: > @param job the job to check for > > (so it formats nicely) Done. On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 401-403 > < https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line401 > > > I'm generally against putting empty @throws in JavaDoc - it doesn't tell you anything the method signature doesn't already say. I'd just delete these lines Agreed. It came on by default in Eclipse, and I had let them be instead of filling it up. Removed. On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 417-420 > < https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line417 > > > same comments apply Removed. On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, lines 439-440 > < https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line439 > > > rather than this "note" why not just change the above line to "Checks if the map-output key and value classes have serializers and deserializers associated with them."? Fixed to a more direct message. Sometimes my commenting language is pretty verbose and confusing, thanks for correcting! On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 470 > < https://reviews.apache.org/r/885/diff/1/?file=21001#file21001line470 > > > missing a space in this exception method. Added. On 2011-06-12 02:31:03, Todd Lipcon wrote: > src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java, line 87 > < https://reviews.apache.org/r/885/diff/1/?file=21002#file21002line87 > > > add an assert that e.getMessage().contains("Couldn't find a serializer") perhaps? Added as well. Harsh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/#review806 ----------------------------------------------------------- On 2011-06-11 22:16:24, Harsh Chouraria wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-11 22:16:24) Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs ----- src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/#review808
          -----------------------------------------------------------

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java
          <https://reviews.apache.org/r/885/#comment1770>

          ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above.

          Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above.

          • Todd

          On 2011-06-12 04:36:10, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/885/

          -----------------------------------------------------------

          (Updated 2011-06-12 04:36:10)

          Review request for hadoop-mapreduce.

          Summary

          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.

          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs

          -----

          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing

          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/#review808 ----------------------------------------------------------- src/java/org/apache/hadoop/mapreduce/JobSubmitter.java < https://reviews.apache.org/r/885/#comment1770 > ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above. Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above. Todd On 2011-06-12 04:36:10, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-12 04:36:10) Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs ----- src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/
          -----------------------------------------------------------

          (Updated 2011-06-12 14:00:13.155848)

          Review request for hadoop-mapreduce.

          Changes
          -------

          Merged exception handling with common, formattable message.

          Summary
          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.
          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs (updated)


          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2
          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing
          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-12 14:00:13.155848) Review request for hadoop-mapreduce. Changes ------- Merged exception handling with common, formattable message. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs (updated) src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-06-12 04:39:03, Todd Lipcon wrote:

          > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 471

          > <https://reviews.apache.org/r/885/diff/2/?file=21008#file21008line471>

          >

          > ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above.

          >

          > Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above.

          Fixed. Unable to run tests right now locally due to a missing protobuf class (which is fixed, but I don't seem to have the fresh jar pulled yet in my maven2 cache). Will update the test when it works (and if it does not).

          • Harsh

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/#review808
          -----------------------------------------------------------

          On 2011-06-12 14:00:13, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/885/

          -----------------------------------------------------------

          (Updated 2011-06-12 14:00:13)

          Review request for hadoop-mapreduce.

          Summary

          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.

          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs

          -----

          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing

          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-06-12 04:39:03, Todd Lipcon wrote: > src/java/org/apache/hadoop/mapreduce/JobSubmitter.java, line 471 > < https://reviews.apache.org/r/885/diff/2/?file=21008#file21008line471 > > > ah, I noticed one more thing. This catch clause is only for the exceptions you've thrown above. > > Rather than do the two nested exceptions, better to have just one layer of exceptions. You could move the generic instructions from the lower exception to a constant and then just append it to either of the two more specific methods above. Fixed. Unable to run tests right now locally due to a missing protobuf class (which is fixed, but I don't seem to have the fresh jar pulled yet in my maven2 cache). Will update the test when it works (and if it does not). Harsh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/#review808 ----------------------------------------------------------- On 2011-06-12 14:00:13, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-12 14:00:13) Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs ----- src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/#review811
          -----------------------------------------------------------

          Looks good. Can you upload latest patch to JIRA so QA-bot picks it up?

          • Todd

          On 2011-06-12 14:00:13, Harsh J wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/885/

          -----------------------------------------------------------

          (Updated 2011-06-12 14:00:13)

          Review request for hadoop-mapreduce.

          Summary

          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.

          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs

          -----

          src/java/org/apache/hadoop/mapred/MapTask.java 21599c2

          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528

          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing

          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/#review811 ----------------------------------------------------------- Looks good. Can you upload latest patch to JIRA so QA-bot picks it up? Todd On 2011-06-12 14:00:13, Harsh J wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-12 14:00:13) Review request for hadoop-mapreduce. Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs ----- src/java/org/apache/hadoop/mapred/MapTask.java 21599c2 src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Harsh J made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Harsh J added a comment -

          Review patch attached.

          Show
          Harsh J added a comment - Review patch attached.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r3.diff [ 12482324 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Release Note Check for proper Key/Value serializers during submission of MR jobs.
          Hide
          Harsh J added a comment -

          Test will fail without https://issues.apache.org/jira/browse/HADOOP-7328 as that blocks this.

          Show
          Harsh J added a comment - Test will fail without https://issues.apache.org/jira/browse/HADOOP-7328 as that blocks this.
          Hide
          Harsh J added a comment -

          From Lists, think its more appropriate here

          On Tue, Jun 14, 2011 at 6:19 PM, Sudharsan Sampath
          <sudharsan.sampath-at-ericsson-dot-com> wrote:
          > Just my thoughts...
          >
          > To me throwing a specific exception would be better. The checkSerializerSpecs attempts to see if we can initialise a new instance of the serializer/deserializer from the jvm where the job is submitted. How does it guarantee that the libs/jars from which these classes are loaded are available for the jvm that executes the job or vice versa? Even if this methods succeeds isn't there a chance then the original problem might occur due to the libs missing from the actual jvm that executes the job?

          What you describe about the JVM can surely occur and I've pushed some handling at the MapOutputBuffer level as well but in any case its better to catch it as early as possible.

          Don't think guaranteeing that the client is indeed packing things along with the job for the DC is practically possible, but at least it is an improvement to where the client follows a bit of general guidelines I think?

          Show
          Harsh J added a comment - From Lists, think its more appropriate here On Tue, Jun 14, 2011 at 6:19 PM, Sudharsan Sampath <sudharsan.sampath-at-ericsson-dot-com> wrote: > Just my thoughts... > > To me throwing a specific exception would be better. The checkSerializerSpecs attempts to see if we can initialise a new instance of the serializer/deserializer from the jvm where the job is submitted. How does it guarantee that the libs/jars from which these classes are loaded are available for the jvm that executes the job or vice versa? Even if this methods succeeds isn't there a chance then the original problem might occur due to the libs missing from the actual jvm that executes the job? What you describe about the JVM can surely occur and I've pushed some handling at the MapOutputBuffer level as well but in any case its better to catch it as early as possible. Don't think guaranteeing that the client is indeed packing things along with the job for the DC is practically possible, but at least it is an improvement to where the client follows a bit of general guidelines I think?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12482324/MAPREDUCE-2584.r3.diff
          against trunk revision 1135462.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem
          org.apache.hadoop.mapred.TestJobClient
          org.apache.hadoop.mapred.TestMapredSystemDir
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.TestMRJobClient

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482324/MAPREDUCE-2584.r3.diff against trunk revision 1135462. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestMapredSystemDir org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.TestMRJobClient -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/390//console This message is automatically generated.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/885/
          -----------------------------------------------------------

          (Updated 2011-06-16 12:24:09.519284)

          Review request for hadoop-mapreduce.

          Changes
          -------

          IOException catching ('stead of null).

          Summary
          -------

          As discussed on HADOOP-7328, MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc..

          This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case.

          This addresses bug MAPREDUCE-2584.
          http://issues.apache.org/jira/browse/MAPREDUCE-2584

          Diffs (updated)


          src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528
          src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a

          Diff: https://reviews.apache.org/r/885/diff

          Testing
          -------

          Added a test case that expects a failure if no io.serializers are present.

          Thanks,

          Harsh

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/885/ ----------------------------------------------------------- (Updated 2011-06-16 12:24:09.519284) Review request for hadoop-mapreduce. Changes ------- IOException catching ('stead of null). Summary ------- As discussed on HADOOP-7328 , MapReduce can handle serializers in a much better way in case of bad configuration, improper imports (Some odd Text class instead of the Writable Text set as key), etc.. This issue covers the MapReduce parts of the improvements (made to MapOutputBuffer and possible early-check of serializer availability pre-submit) that provide more information than just an NPE as is the current case. This addresses bug MAPREDUCE-2584 . http://issues.apache.org/jira/browse/MAPREDUCE-2584 Diffs (updated) src/java/org/apache/hadoop/mapreduce/JobSubmitter.java 751d528 src/test/mapred/org/apache/hadoop/mapreduce/TestMRJobClient.java 5fa329a Diff: https://reviews.apache.org/r/885/diff Testing ------- Added a test case that expects a failure if no io.serializers are present. Thanks, Harsh
          Hide
          Harsh J added a comment -

          Patch updated as per HADOOP-7328 changes.

          (Still blocked by HADOOP-7328 inclusion and will not compile until)

          Show
          Harsh J added a comment - Patch updated as per HADOOP-7328 changes. (Still blocked by HADOOP-7328 inclusion and will not compile until)
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r2.diff [ 12482789 ]
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r3.diff [ 12482324 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12482789/MAPREDUCE-2584.r2.diff
          against trunk revision 1136261.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The patch appears to cause tar ant target to fail.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:

          -1 contrib tests. The patch failed contrib unit tests.

          -1 system test framework. The patch failed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482789/MAPREDUCE-2584.r2.diff against trunk revision 1136261. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/399//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          (Re)attaching an older revision patch, as reviewed (diff #3) on the reviews site by Todd.

          Show
          Harsh J added a comment - (Re)attaching an older revision patch, as reviewed (diff #3) on the reviews site by Todd.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r3.diff [ 12483866 ]
          Harsh J made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483866/MAPREDUCE-2584.r3.diff
          against trunk revision 1139400.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem
          org.apache.hadoop.mapred.TestJobClient
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.TestMRJobClient

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483866/MAPREDUCE-2584.r3.diff against trunk revision 1139400. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.TestMRJobClient -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/422//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          -4 findbugs. My bad, sorry.

          Show
          Harsh J added a comment - -4 findbugs. My bad, sorry.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r4.diff [ 12483921 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12483921/MAPREDUCE-2584.r4.diff
          against trunk revision 1139400.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.cli.TestMRCLI
          org.apache.hadoop.fs.TestFileSystem
          org.apache.hadoop.mapred.TestJobClient
          org.apache.hadoop.mapred.TestMiniMRClasspath
          org.apache.hadoop.mapreduce.TestMRJobClient

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12483921/MAPREDUCE-2584.r4.diff against trunk revision 1139400. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.mapred.TestJobClient org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapreduce.TestMRJobClient -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/428//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          This can run once again after HADOOP-7328 is committed.

          Show
          Harsh J added a comment - This can run once again after HADOOP-7328 is committed.
          Hide
          Harsh J added a comment -

          Updating patch for trunk…

          Show
          Harsh J added a comment - Updating patch for trunk…
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r5.diff [ 12493044 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493044/MAPREDUCE-2584.r5.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493044/MAPREDUCE-2584.r5.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/608//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          +1

          Show
          Todd Lipcon added a comment - +1
          Hide
          Todd Lipcon added a comment -

          Actually, not committing this yet - for some reason the test does not pass on my setup. Maybe something related to Maven/Ivy not pulling the right dependencies... looking into it.

          Show
          Todd Lipcon added a comment - Actually, not committing this yet - for some reason the test does not pass on my setup. Maybe something related to Maven/Ivy not pulling the right dependencies... looking into it.
          Hide
          Harsh J added a comment -

          Hey Todd,

          I'll run the test now. But this requires the HADOOP-7328 patch, to pass (minor return behavior change, being the reason).

          Show
          Harsh J added a comment - Hey Todd, I'll run the test now. But this requires the HADOOP-7328 patch, to pass (minor return behavior change, being the reason).
          Hide
          Harsh J added a comment -

          Todd,

          Looks like some form of build issue.

          Here's what I tried to do to get this test to pass:

          hadoop $ (Applied patch)
          hadoop $ mvn install -DskipTests -P-cbuild
          (Success)
          hadoop-mapreduce-project $ ant veryclean
          (Success)
          hadoop-mapreduce-project $ ant -Dresolvers=internal -Dtestcase=TestMRJobClient test
          (Failed)
          hadoop-mapreduce-project $ grep "find a serializer for the" build/ -R
          Binary file build/test/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches
          Binary file build/test/mapred/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches

          This looks wrong. That string ought to be present in the binary MR jar as well, but it doesn't show up as part of grep result. It would mean that locally published jars aren't being used? Also, TestMRJobClient lies outside of the client test suites, in the older area… Shouldn't it be moved in?

          Show
          Harsh J added a comment - Todd, Looks like some form of build issue. Here's what I tried to do to get this test to pass: hadoop $ (Applied patch) hadoop $ mvn install -DskipTests -P-cbuild (Success) hadoop-mapreduce-project $ ant veryclean (Success) hadoop-mapreduce-project $ ant -Dresolvers=internal -Dtestcase=TestMRJobClient test (Failed) hadoop-mapreduce-project $ grep "find a serializer for the" build/ -R Binary file build/test/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches Binary file build/test/mapred/classes/org/apache/hadoop/mapreduce/TestMRJobClient.class matches This looks wrong. That string ought to be present in the binary MR jar as well, but it doesn't show up as part of grep result. It would mean that locally published jars aren't being used? Also, TestMRJobClient lies outside of the client test suites, in the older area… Shouldn't it be moved in?
          Hide
          Harsh J added a comment -

          0.20-security branch patch. Added tests and TestMapRed pass, after applied atop HADOOP-7328's 0.20-security patch.

          Show
          Harsh J added a comment - 0.20-security branch patch. Added tests and TestMapRed pass, after applied atop HADOOP-7328 's 0.20-security patch.
          Harsh J made changes -
          Attachment 0.20-security-MAPREDUCE-2584.r5.diff [ 12493922 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12493922/0.20-security-MAPREDUCE-2584.r5.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/688//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493922/0.20-security-MAPREDUCE-2584.r5.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/688//console This message is automatically generated.
          Arun C Murthy made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Fix Version/s 0.23.0 [ 12315570 ]
          Hide
          Harsh J added a comment -

          HADOOP-7328 was committed to 0.23/trunk. This one still awaits. I re-ran the tests and they seem to pass.

          I'd also like this committed to branch-0.1 (patch was attached earlier).

          Let me know if anything further is required!

          Show
          Harsh J added a comment - HADOOP-7328 was committed to 0.23/trunk. This one still awaits. I re-ran the tests and they seem to pass. I'd also like this committed to branch-0.1 (patch was attached earlier). Let me know if anything further is required!
          Hide
          Harsh J added a comment -

          (Er, 1.0)

          Show
          Harsh J added a comment - (Er, 1.0)
          Harsh J made changes -
          Fix Version/s 0.24.0 [ 12317654 ]
          Target Version/s 0.23.1, 0.24.0, 1.1.0 [ 12318883, 12317654, 12317960 ]
          Hide
          Harsh J added a comment -

          Skipping 1.x, not very important there. Lets focus on getting this into 0.23.x though – any further comments on latest patch, anyone?

          Show
          Harsh J added a comment - Skipping 1.x, not very important there. Lets focus on getting this into 0.23.x though – any further comments on latest patch, anyone?
          Harsh J made changes -
          Target Version/s 1.1.0, 0.24.0, 0.23.1 [ 12317960, 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Robert Joseph Evans made changes -
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 2.0.0, 3.0.0 [ 12320354, 12320355 ]
          Harsh J made changes -
          Attachment 0.20-security-MAPREDUCE-2584.r5.diff [ 12493922 ]
          Hide
          Harsh J added a comment -

          Rebased patch for current trunk (had to move the test file around).

          Added test fails without fix, passes with fix:

          Running org.apache.hadoop.mapreduce.TestMRJobClient
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.03 sec
          
          Results :
          
          Tests run: 1, Failures: 0, Errors: 0, Skipped: 1
          

          Does anyone have any further comments on this improvement? I believe its complete and ought to have gone in with HADOOP-7328 but has instead lied stale for months now, although HADOOP-7328 (a dependency) was put in.

          Show
          Harsh J added a comment - Rebased patch for current trunk (had to move the test file around). Added test fails without fix, passes with fix: Running org.apache.hadoop.mapreduce.TestMRJobClient Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.03 sec Results : Tests run: 1, Failures: 0, Errors: 0, Skipped: 1 Does anyone have any further comments on this improvement? I believe its complete and ought to have gone in with HADOOP-7328 but has instead lied stale for months now, although HADOOP-7328 (a dependency) was put in.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r6.diff [ 12526621 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12526621/MAPREDUCE-2584.r6.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12526621/MAPREDUCE-2584.r6.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2381//console This message is automatically generated.
          Harsh J made changes -
          Link This issue is related to HADOOP-8531 [ HADOOP-8531 ]
          Hide
          Harsh J added a comment -

          Rebased to trunk again.

          Show
          Harsh J added a comment - Rebased to trunk again.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r7.diff [ 12534111 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12534111/MAPREDUCE-2584.r7.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12534111/MAPREDUCE-2584.r7.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2536//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Rebased yet again. Part of this patch's changes on the common side were already committed, so am inclined to commit this in as well.

          But I'd like to know if there are any concerns with this new front-end check. Anyone? Is throwing an exception bad if there's a ClassNotFound? Should we just WARN?

          Show
          Harsh J added a comment - Rebased yet again. Part of this patch's changes on the common side were already committed, so am inclined to commit this in as well. But I'd like to know if there are any concerns with this new front-end check. Anyone? Is throwing an exception bad if there's a ClassNotFound? Should we just WARN?
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r7.diff [ 12548162 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12548162/MAPREDUCE-2584.r7.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestNoDefaultsJobConf

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548162/MAPREDUCE-2584.r7.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestNoDefaultsJobConf +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2914//console This message is automatically generated.
          Hide
          Tom White added a comment -

          This looks like a good check to me. Can you avoid using the @SuppressWarnings annotation?

          Show
          Tom White added a comment - This looks like a good check to me. Can you avoid using the @SuppressWarnings annotation?
          Hide
          Harsh J added a comment -

          Thanks Tom! I've removed those in this new rebased patch.

          Show
          Harsh J added a comment - Thanks Tom! I've removed those in this new rebased patch.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r8.diff [ 12555676 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555676/MAPREDUCE-2584.r8.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestNoDefaultsJobConf

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555676/MAPREDUCE-2584.r8.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestNoDefaultsJobConf +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3089//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          The failing test does look related:

          java.lang.NullPointerException
          	at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58)
          	at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:456)
          	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424)
          	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
          	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
          	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
          	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
          	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
          	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890)
          	at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82)
          

          But surprised to see it after HADOOP-8588. Perhaps its a different case, investigating.

          Show
          Harsh J added a comment - The failing test does look related: java.lang.NullPointerException at org.apache.hadoop.io.serializer.SerializationFactory.<init>(SerializationFactory.java:58) at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:456) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890) at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82) But surprised to see it after HADOOP-8588 . Perhaps its a different case, investigating.
          Hide
          Harsh J added a comment -

          Patch that addresses the TestNoJobConfDefaults test failure by fixing the SerializationFactory logic, wherein a null may be received for a serializations listing.

          Show
          Harsh J added a comment - Patch that addresses the TestNoJobConfDefaults test failure by fixing the SerializationFactory logic, wherein a null may be received for a serializations listing.
          Harsh J made changes -
          Attachment MAPREDUCE-2584.r9.diff [ 12555708 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12555708/MAPREDUCE-2584.r9.diff
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.conf.TestNoDefaultsJobConf

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12555708/MAPREDUCE-2584.r9.diff against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.conf.TestNoDefaultsJobConf +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3093//console This message is automatically generated.
          Hide
          Harsh J added a comment -

          Problem is coming from the fact that when we lookup the IO_SERIALIZATIONS_KEY, we don't pass a default classes set.

          We could add that in, but it may get harder to maintain it if we extend the list. In any case, I'll upload another patch.

          java.io.IOException: Couldn't find a serializer for the Map-Output Key class: 'class org.apache.hadoop.io.LongWritable'. If custom serialization is being used, ensure that the 'io.serializations' property is appropriately configured for the Job.
          	at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:462)
          	at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424)
          	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
          	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
          	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215)
          	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617)
          	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at javax.security.auth.Subject.doAs(Subject.java:396)
          	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437)
          	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612)
          	at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890)
          	at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82)
          
          Show
          Harsh J added a comment - Problem is coming from the fact that when we lookup the IO_SERIALIZATIONS_KEY, we don't pass a default classes set. We could add that in, but it may get harder to maintain it if we extend the list. In any case, I'll upload another patch. java.io.IOException: Couldn't find a serializer for the Map-Output Key class: 'class org.apache.hadoop.io.LongWritable'. If custom serialization is being used, ensure that the 'io.serializations' property is appropriately configured for the Job. at org.apache.hadoop.mapreduce.JobSubmitter.checkSerializerSpecs(JobSubmitter.java:462) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:424) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218) at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1215) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1215) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:617) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:612) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1437) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:612) at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:890) at org.apache.hadoop.conf.TestNoDefaultsJobConf.testNoDefaults(TestNoDefaultsJobConf.java:82)
          Harsh J made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Gavin made changes -
          Link This issue depends on HADOOP-7328 [ HADOOP-7328 ]
          Gavin made changes -
          Link This issue depends upon HADOOP-7328 [ HADOOP-7328 ]
          Harsh J made changes -
          Comment [ Appears to be a dupe of MAPREDUCE-2584. Care to look at the extras added there and update your patch here? We could close out MAPREDUCE-2584 as a dupe if you're working on this one. ]
          Harsh J made changes -
          Link This issue is duplicated by MAPREDUCE-5728 [ MAPREDUCE-5728 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3h 24m 3 Harsh J 26/Jun/11 19:24
          Patch Available Patch Available Open Open
          540d 10h 33m 3 Harsh J 03/Dec/12 08:52

            People

            • Assignee:
              Harsh J
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development