[PIG-2587] Compute LogicalPlan signature and store in job conf

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11
• Assignee: William W. Graham Jr
• Reporter: William W. Graham Jr
• Created: 14/Mar/12 05:37
• Updated: 10/Sep/13 17:12
• Resolved: 24/Apr/12 06:54

Description

We'd like to be able to uniquely identify a re-executed script (possibly with different inputs/outputs) by creating a signature of the LogicalPlan. Here's the proposal:

1. Add a new method LogicalPlan.getSignature() that returns a hash of its LogicalPlanPrinter output.
2. In PigServer.execute() set the signature on the job conf after the LP is compiled, but before it's executed.

(1) would allow an impl of PigProgressNotificationListener.setScriptPlan() to save the LP signature with the script metadata. Upon subsequent runs (2) would allow an impl of PigReducerEstimator (see PIG-2574) to retrieve the current LP signature and fetch the historical data for the script. It could then use the previous run data to better estimate the number of reducers.

Issue Links

• breaks: PIG-3455 Pig 0.11.1 OutOfMemory error