[PIG-4541] Skewed full outer join does not return records if any relation is empty. Outer join does not return any record if left relation is empty

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0
• Component/s: build

Environment

HDP 2.2.4

• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Dipankar
• Due: 08/May/15
• Created: 08/May/15 20:00
• Updated: 07/Jun/15 03:48
• Resolved: 26/May/15 22:54

Description

Test1:
Perform full join on two relation with left relation being blank and right containing records
empty_relation = FILTER a_relation by (join_column=='eliminate everything');
Test_output = JOIN empty_relation by (join_column) FULL , non_empty_relation by (join_column);
Result : Zero records returned.

Test2:
Perform full join on two relation with left relation being blank and right containing records using skewed
Test_output = JOIN empty_relation by (join_column) FULL , non_empty_relation by (join_column) using ‘skewed’;

Result : Zero records returned.

Test3:
Perform full join on two relation with left relation being blank and right containing records using parallel
Test_output = JOIN empty_relation by (join_column) FULL , non_empty_relation by (join_column) PARALLEL 10;

Result : Zero records returned.

Test4:
Perform full join on two relation with left relation being non empty and right not containing records using parallel
Test_output = JOIN , non_empty_relation by (join_column) FULL , empty_relation by (join_column) PARALLEL 10;

Result : valid records returned.

Observation:
1) If the either relation is blank , skewed full outer join does not return anything
2) If the non empty relation is kept on left, everything works except skewed
3) FULL OUTER will only work if the left relation is not empty
4) Skewed will only work if both relation is non empty.

Issue Links

• is broken by: PIG-4377 Skewed outer join produce wrong result if a key is oversampled