[PIG-5134] Fix TestAvroStorage unit test in Spark mode

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Hadoop Flags: Reviewed
• Assignee: Nándor Kollár
• Reporter: liyunzhang
• Created: 22/Feb/17 08:23
• Updated: 21/Jun/17 09:18
• Resolved: 17/May/17 20:45

Description

It seems that test fails, because Avro GenericData#Record doesn't implement Serializable interface:

2017-02-23 09:14:41,887 ERROR [main] spark.JobGraphBuilder (JobGraphBuilder.java:sparkOperToRDD(183)) - throw exception in sparkOperToRDD: 
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0 in stage 9.0 (TID 9) had a not serializable result: org.apache.avro.generic.GenericData$Record
Serialization stack:
	- object not serializable (class: org.apache.avro.generic.GenericData$Record, value: {"key": "stuff in closet", "value1": {"thing": "hat", "count": 7}, "value2": {"thing": "coat", "count": 2}})
	- field (class: org.apache.pig.impl.util.avro.AvroTupleWrapper, name: avroObject, type: interface org.apache.avro.generic.IndexedRecord)
	- object (class org.apache.pig.impl.util.avro.AvroTupleWrapper, org.apache.pig.impl.util.avro.AvroTupleWrapper@3d3a58c1)
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

The failing tests is a new test introduced with merging trunk to spark branch, that's why we didn't see this error before.

Issue Links

• is related to: PIG-5233 Use Kryo serialization instead of Java serialization for Spark