[PIG-3661] Piggybank AvroStorage fails if used in more than one load or store statement

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.1
• Fix Version/s: 0.12.1, 0.13.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 09/Jan/14 18:03
• Updated: 15/Apr/14 20:44
• Resolved: 24/Jan/14 22:24

Description

To reproduce:
A =load '/tmp/data' as (a1:int, a2:int, a3:int);
B = load '/tmp/data1' as (b1:chararray, b2:chararray, b3:chararray);
store A into '/tmp/out/a' using org.apache.pig.piggybank.storage.avro.AvroStorage();
store B into '/tmp/out2/b' using org.apache.pig.piggybank.storage.avro.AvroStorage();

It either fails in the map job if schema is incompatible, or B gets schema of A and B merged leading to incorrect results.

Reason is schema is stored and accessed from a property of UDFContext without using a context signature.

UDFContext context = UDFContext.getUDFContext();
Properties property = context.getUDFProperties(ResourceSchema.class);
String prevSchemaStr = property.getProperty(AVRO_OUTPUT_SCHEMA_PROPERTY);

Issue Links

• incorporates: PIG-3717 AvroStorage is not filtering hidden files on directory recursion