[PIG-4239] "pig.output.lazy" not works in spark mode

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: liyunzhang
• Reporter: liyunzhang
• Created: 16/Oct/14 06:32
• Updated: 21/Jun/17 09:18
• Resolved: 03/Dec/14 10:00

Description

If "pig.output.lazy" is "true", empty part files from the output will be omitted.
steps to reproduce
1. set "pig.output.lazy" as “true" in $PIG_HOME/conf/pig.properties.
2. run following lazy.pig script in spark mode:
cat lazy.pig
a = load '/user/pig/lazy' using PigStorage();
b = filter a by $0 == 'hey';
c = store b into '/tmp/lazy.out';
lazy.pig and lazy are attached
3. empty file "/tmp/lazy.out/part-rxxxx” is still generated, the empty file is expected not to be generated when "pig.output.lazy" is "true"

Issue Links

• blocks: PIG-4243 Fix "TestStore" for Spark engine
• relates to: PIG-4243 Fix "TestStore" for Spark engine