Add test for non-writable serializer

• Key: HADOOP-2997
• Type: Test
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Tom White
• Reporter: Tom White
• Fix Version/s: 0.17.0

Description

It would be useful to have a unit test that tests MapReduce works using a non-writable serializer.

Comments

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377604/hadoop-2997.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit -1. The applied patch generated 193 release audit warnings (more than the trunk's current 192 warnings).

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests -1. The patch failed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/build/test/checkstyle-errors.html
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/artifact/trunk/current/releaseAuditDiffWarnings.txt
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1945/console

This message is automatically generated.

[Devaraj Das]

Tom, could you please address the hudson warning and resubmit the patch. Thanks!

[Tom White]

I've fixed the release audit problem, the failing unit test reported by hudson was unrelated to this patch. However, this test now hangs consistently - it looks like it's related to the changes in HADOOP-2399 since the test passes when I run against revision 636622.

[Tom White]

Tracked down the problem - the change in HADOOP-2399 creates instances of keys and values to be reused, however this is not possible for Java serialization since the types don't necessarily have no-arg constructors:

java.lang.RuntimeException: java.lang.NoSuchMethodException: java.lang.Long.<init>()
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:80)
at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.<init>(ReduceTask.java:213)
at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.<init>(ReduceTask.java:306)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:388)
at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2073)
Caused by: java.lang.NoSuchMethodException: java.lang.Long.<init>()
at java.lang.Class.getConstructor0(Class.java:2678)
at java.lang.Class.getDeclaredConstructor(Class.java:1953)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:74)
... 4 more

The fix is to get the deserializer to create the key and value types, if needed. I'm testing a patch.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12377777/hadoop-2997-v3.patch
against trunk revision 619744.

@author +1. The patch does not contain any @author tags.

tests included +1. The patch appears to include 3 new or modified tests.

javadoc +1. The javadoc tool did not generate any warning messages.

javac +1. The applied patch does not generate any new javac compiler warnings.

release audit +1. The applied patch does not generate any new release audit warnings.

findbugs +1. The patch does not introduce any new Findbugs warnings.

core tests -1. The patch failed core unit tests.

contrib tests +1. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1960/console

This message is automatically generated.

[Owen O'Malley]

+1

[Devaraj Das]

I just committed this. Thanks, Tom!

[Hudson]

Integrated in Hadoop-trunk #444 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/444/)

[Hudson]

Integrated in Hadoop-trunk #445 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/445/)