[HIVE-9845] HCatSplit repeats information making input split data size huge

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: HCatalog
• Assignee: Mithun Radhakrishnan
• Reporter: Rohini Palaniswamy
• Created: 03/Mar/15 22:55
• Updated: 22/Jul/15 21:29
• Resolved: 06/May/15 21:05

Description

Pig on Tez jobs with larger tables hit PIG-4443. Running on HDFS data which has even triple the number of splits(100K+ splits and tasks) does not hit that issue.

HCatBaseInputFormat.java:
 //Call getSplit on the InputFormat, create an
      //HCatSplit for each underlying split
      //NumSplits is 0 for our purposes
      org.apache.hadoop.mapred.InputSplit[] baseSplits = 
        inputFormat.getSplits(jobConf, 0);

      for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
        splits.add(new HCatSplit(
            partitionInfo,
            split,allCols));
      }

Each hcatSplit duplicates partition schema and table schema.

Issue Links

• is related to: HIVE-11344 HIVE-9845 makes HCatSplit.write modify the split so that PartInfo objects are unusable after it
• relates to: PIG-4443 Write inputsplits in Tez to disk if the size is huge and option to compress pig input splits