[PIG-3751] Generating Splits in Tez should be configurable to AM or client

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: tez
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 07/Feb/14 01:14
• Updated: 13/Mar/15 21:51

Description

1) TEZ-752 allows setting list of URIs to get delegation tokens. Set that to make Tez get delegation tokens and calculate input splits on AM
2) Try using Tez Grouping of input splits instead of pig.maxCombinedSplitSize grouping.

Generating splits in AM is supposed to give performance boost. For those case where InputFormat or OutputFormat get delegation tokens and it is not possible to do that, then have a option to generate input splits on client.