[PIG-2824] Pushing checking number of fields into LoadFunc

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.9.0, 0.10.0
• Assignee: Unassigned
• Reporter: Jie Li
• Created: 17/Jul/12 23:10
• Updated: 12/Jan/13 12:24

Description

As described in PIG-1188, if users define a schema (w or w/o types), we need to check the number of fields after loading data, so if there are less fields we need to pad null fields, and if there are more fields we need to throw them away.

For schema with types, Pig used to insert a Foreach after the loader for type casting which also checks #fields. For schema without types there was no such Foreach, thus PIG-1188 inserted one just for checking #fields. Unfortunately, Foreach is too expensive for such checking, and ideally we can push it into the loader.

Issue Links

• is related to: PIG-2661 Pig uses an extra job for loading data in Pigmix L9