[AVRO-780] union handling in ReflectDatumWriter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.1
• Component/s: java

Environment

Mac with VMWare running Linux training-vm 2.6.28-19-server #61-Ubuntu

• Hadoop Flags: Reviewed
• Assignee: Doug Cutting
• Reporter: ey-chih chow
• Created: 11/Mar/11 00:41
• Updated: 06/May/11 19:14
• Resolved: 30/Mar/11 23:37

Description

Our avdl schema definition for the record DeviceRow has a field:

union

{array<DynamicColumn4Games>, null}

Games__;

When we migrated our MR jobs from 1.4.0 to 1.5.0, we got following messages:

===================================================================================================
11/03/10 11:31:02 INFO mapred.TaskInProgress: Error from attempt_20110310113041953_0001_m_000000_0: java.lang.NullPointerException: in com.ngmoco.hbase.DeviceRow in union null of union in field Games__ of com.ngmoco.hbase.DeviceRow
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:104)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:57)
at org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:131)
at org.apache.avro.mapred.AvroSerialization$AvroWrapperSerializer.serialize(AvroSerialization.java:114)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:900)
at org.apache.hadoop.mapred.MapTask$OldOutputCollector.collect(MapTask.java:466)
at org.apache.avro.mapred.HadoopMapper$MapCollector.collect(HadoopMapper.java:69)
at com.ngmoco.ngpipes.sourcing.NgActivityGatheringMapper.map(NgActivityGatheringMapper.java:91)
at com.ngmoco.ngpipes.sourcing.NgActivityGatheringMapper.map(NgActivityGatheringMapper.java:1)
at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:80)
at org.apache.avro.mapred.HadoopMapper.map(HadoopMapper.java:34)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.NullPointerException: in union null of union in field Games__ of com.ngmoco.hbase.DeviceRow
at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:92)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:86)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:102)
... 14 more
===================================================================================================

After we change definition of the field Games__ to:

union

{null, array<DynamicColumn4Games>}

Games__;

The system stop complaining.