[PIG-2909] Add a new option for ignoring corrupted files to AvroStorage load func

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11
• Component/s: piggybank
• Patch Info: Patch Available
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 06/Sep/12 05:35
• Updated: 22/Feb/13 04:53
• Resolved: 13/Sep/12 21:17

Description

Currently, AvroStorage load fails with AvroRuntimeException when encountering corrupted input files. For example,

ERROR 2997: Unable to recreate exception from backed error: java.io.IOException: org.apache.avro.AvroRuntimeException: java.io.IOException: Invalid sync!
	at org.apache.pig.piggybank.storage.avro.AvroStorage.getNext(AvroStorage.java:283)

But it is not always desirable to fail the Pig job for bad files. It is sometimes more useful to skip them and continue.

Issue Links

• is related to: PIG-2614 AvroStorage crashes on LOADING a single bad error