[PIG-4443] Write inputsplits in Tez to disk if the size is huge and option to compress pig input splits

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0
• Hadoop Flags: Reviewed
• Assignee: Rohini Palaniswamy
• Reporter: Rohini Palaniswamy
• Created: 03/Mar/15 22:50
• Updated: 25/Jun/15 10:02
• Resolved: 06/Mar/15 21:24

Description

Pig sets the input split information in user payload and when running against a table with 10s of 1000s of partitions, DAG submission fails with
java.io.IOException: Requested data length 305844060 is longer than maximum
configured RPC length 67108864

Issue Links

• is broken by: TEZ-2192 Relocalization does not check for source
• is related to: HIVE-9845 HCatSplit repeats information making input split data size huge
• is related to: TEZ-2144 Compressing MRInput Split Distributor payload
• links to: Review Board