[PIG-2195] AvroStorage fails to STORE when LOADing via PigStorage

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Release Note: AvroStorage support for using a schema from an Avro schema file.
• Assignee: William W. Graham Jr
• Reporter: William W. Graham Jr
• Created: 28/Jul/11 22:28
• Updated: 26/Apr/12 20:32
• Resolved: 15/Aug/11 23:25

Description

Reading data via PigStorage and writing it via AvroStorage fails with an exception like this

java.lang.ClassCastException: org.apache.pig.data.BinSedesTuple cannot be cast to org.apache.avro.generic.IndexedRecord

The Pig script in this section of the documentation shows an example like this that fails:

http://linkedin.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data#AvroStorage-PigsupportforAvrodata-A.Howtostoredataindifferentways.

A workaround currently exists to produce avro from TSVs like this:

avro = LOAD 'inputPath/' AS (foo);
STORE avro INTO 'outputPath/' USING oap.piggybank.storage.avro.AvroStorage(
  '{"data":"data_file.avro",
    "same":"data_file.avro", "field0":"def:bar"}');

This is redundant though and data and same seem to indicate the same thing. This approach also requires an existing avro data file to exist. This patch will make the following alternate constructor syntax's work as well.

1. Read schema from an existing data file:

     '{"data":"data_file.avro", "field0":"def:bar"}');
   

2. Read schema from an existing schema file:

     '{"schema_file":"data_file.avsc", "field0":"def:bar"}');