[HADOOP-10686] Writables are not always configured

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Abraham Elmahrek
• Reporter: Abraham Elmahrek
• Created: 05/Jun/14 17:45
• Updated: 07/Sep/21 16:11
• Resolved: 29/Jul/14 23:16

Description

Seeing the following exception:

java.lang.Exception: java.lang.NullPointerException
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:403)
Caused by: java.lang.NullPointerException
	at org.apache.sqoop.job.io.SqoopWritable.readFields(SqoopWritable.java:59)
	at org.apache.hadoop.io.WritableComparator.compare(WritableComparator.java:129)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1248)
	at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:35)
	at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:87)
	at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:63)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1582)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1467)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:699)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:769)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:235)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)

It turns out that WritableComparator does not configure Writable objects :https://github.com/apache/hadoop-common/blob/branch-2.3.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/WritableComparator.java. This is during the sort phase for an MR job.

Issue Links

• is depended upon by: SQOOP-1349 Sqoop2: Use configurable writable to get Intermediate Data Format
• is related to: ORC-964 OrcKey / OrcValue NullPointerException during MapReduce shuffle
• relates to: TEZ-4133 key class implements writableComparable and configurable use default configuration
• links to: Review Board