[HIVE-17257] Hive should merge empty files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 06/Aug/17 19:36
• Updated: 23/May/18 00:00
• Resolved: 18/Jan/18 21:31

Description

Currently if merging file option is turned on and the dest dir contains large number of empty files, Hive will not trigger merge task:

  private long getMergeSize(FileSystem inpFs, Path dirPath, long avgSize) {
    AverageSize averageSize = getAverageSize(inpFs, dirPath);
    if (averageSize.getTotalSize() <= 0) {
      return -1;
    }

    if (averageSize.getNumFiles() <= 1) {
      return -1;
    }

    if (averageSize.getTotalSize()/averageSize.getNumFiles() < avgSize) {
      return averageSize.getTotalSize();
    }
    return -1;
  }

This logic doesn't seem right as the it seems better to combine these empty files into one.