[PIG-4911] Provide option to disable DAG recovery

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 26/May/16 15:18
• Updated: 21/Jun/17 09:15
• Resolved: 23/Jun/16 18:28

Description

Tez 0.7 has lot of issues with DAG recovery with auto parallelism causing hung dags in many cases as it was not writing auto parallelism decisions to recovery history. Rewrite was done in Tez 0.8 to handle that.
Code was added to Tez to automatically disable recovery if there was auto parallelism so that it would benefit both Pig and Tez. It works fine and the second AM attempt fails with DAG cannot be recovered error when it sees there are vertices with auto parallelism. But problem is it is hard to see what the actual problem is for the users and is hard to debug as well as the whole UI state is rewritten with the partial recovery information.
Doing the disabling of recovery in Pig itself by setting tez.dag.recovery.enabled=false will make it not go for the second attempt at all which will eventually fail. It also makes it easy to debug the original failure.