[PIG-2541] Automatic record provenance (source tagging) for PigStorage

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.1
• Fix Version/s: 0.10.0, 0.11
• Component/s: impl
• Hadoop Flags: Reviewed

Release Note

We add a new option -tagsource to PigStorage. With this flag, we can get the INPUT_FILE_NAME as the first column of the output data. eg:

a = load '1.txt' using PigStorage('\t', '-tagsource');

• Assignee: Prashant Kommireddi
• Reporter: Richard Ding
• Created: 17/Feb/12 02:24
• Updated: 26/Apr/12 20:32
• Resolved: 01/Mar/12 03:03

Description

There are a lot of interests in knowing where the data comes from when loading from a directory (or a set of directories). One can do it manually (see https://cwiki.apache.org/confluence/display/PIG/FAQ). But it will be more convenient for users if we implement this in the PigStorage with a command line option (e.g., pig.source.tagging=true/false) to turn it on/off. By default it will be off.