[PIG-5056] Fix AvroStorage writing enums

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0
• Fix Version/s: 0.17.0
• Labels: avro
• Hadoop Flags: Reviewed
• Assignee: Ádám Szita
• Reporter: Ádám Szita
• Created: 09/Nov/16 12:58
• Updated: 21/Jun/17 09:15
• Resolved: 10/Nov/16 23:15

Description

Issue is observable with latest (1.8.1) Avro since it has an extra check for enum types that the current 1.7.5 does not care about (see here: https://github.com/apache/avro/blob/release-1.8.1/lang/java/avro/src/main/java/org/apache/avro/generic/GenericDatumWriter.java#L163)

This results in TestAvroStorage#testLoadRecordsWithEnums failing: Pig reads an Avro file with a schema containing (string,int,enum) this is then represented in Pig as (chararray,int,chararray) and then Pig writes this back to an Avro file with given schema (string,int,enum).

java.lang.Exception: java.io.IOException: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Not an enum: GOOD
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.io.IOException: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Not an enum: GOOD
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:83)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:144)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:97)
	at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:655)
	at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
	at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapOnly$Map.collect(PigMapOnly.java:48)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:282)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:275)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:65)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:784)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: org.apache.avro.AvroTypeException: Not an enum: GOOD
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
	at org.apache.pig.impl.util.avro.AvroRecordWriter.write(AvroRecordWriter.java:115)
	at org.apache.pig.impl.util.avro.AvroRecordWriter.write(AvroRecordWriter.java:51)
	at org.apache.pig.builtin.AvroStorage.putNext(AvroStorage.java:520)
	at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.StoreFuncDecorator.putNext(StoreFuncDecorator.java:75)
	... 18 more
Caused by: org.apache.avro.AvroTypeException: Not an enum: GOOD
	at org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
	at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153)
	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
	... 22 more

Issue Links

• relates to: AVRO-997 Generic API should require GenericEnumSymbol when writing Avro Enums