[PIG-4737] Check and fix clone implementation for all classes extending PhysicalOperator

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 16/Nov/15 22:58
• Updated: 08/Jun/16 20:48
• Resolved: 12/Jan/16 23:23

Description

PhysicalOperator.clone() eventually calls Object.clone() which only does a shallow copy (javadoc wrongly says deep copy) and this causes issues with UnionOptimizer in Tez. Most of the clone is already fixed due to issues found earlier, but recently ran into an issue with POStream where after clone same reference was retained to binaryOutputQueue and binaryInputQueue and caused the script to hang.

Mostly cloned operators in Union go to different tez vertex plans and the issue would not have occurred, but in the particular case due to replicated join and with the combination of multi-query and union optimization, both the cloned plans of union ended up in the same vertex(one that loads C). That single vertex will handle both the replicated joins and streaming in two sub-plans of split and store the final result in g.

A = LOAD 'a';
B = LOAD 'b';
C = LOAD 'c';
D = JOIN C by $0, A by $0 using 'replicated';
E = JOIN C by $0, B by $0 using 'replicated';
F = UNION D, E;
G = STREAM F through ....
STORE G into 'g';

It is good to go through all classes extending PhysicalOperator and check if it deep clones objects that are not primitive types.

Issue Links

• links to: RB