[HIVE-22054] Avoid recursive listing to check if a directory is empty

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0, 1.2.0, 2.1.0, 2.3.5, 3.1.1
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Metastore
• Assignee: Prabhas Kumar Samanta
• Reporter: Prabhas Kumar Samanta
• Created: 26/Jul/19 10:38
• Updated: 07/Dec/22 02:50
• Resolved: 05/Aug/19 00:36

Description

During drop partition on a managed table, first we delete the directory corresponding to the partition. After that we recursively delete the parent directory as well if parent directory becomes empty. To do this emptiness check, we call Warehouse::getContentSummary(), which in turn recursively check all files and subdirectories. This is a costly operation when a directory has a lot of files or subdirectories. This overhead is even more prominent for cloud based file systems like s3. And for emptiness check, this is unnecessary too.

This is recursive listing was introduced as part of HIVE-5220. Code snippet for reference :

// Warehouse.java
public boolean isEmpty(Path path) throws IOException, MetaException {
  ContentSummary contents = getFs(path).getContentSummary(path);
  if (contents != null && contents.getFileCount() == 0 && contents.getDirectoryCount() == 1) {
    return true;
  }
  return false;
}

// HiveMetaStore.java
private void deleteParentRecursive(Path parent, int depth, boolean mustPurge, boolean needRecycle)
  throws IOException, MetaException {
  if (depth > 0 && parent != null && wh.isWritable(parent)) {
    if (wh.isDir(parent) && wh.isEmpty(parent)) {
      wh.deleteDir(parent, true, mustPurge, needRecycle);
    }
    deleteParentRecursive(parent.getParent(), depth - 1, mustPurge, needRecycle);
  }
}

// Note: FileSystem::getContentSummary() performs a recursive listing.

Issue Links

• is caused by: HIVE-5220 Add option for removing intermediate directory for partition, which is empty
• is related to: HADOOP-16468 S3AFileSystem.getContentSummary() to use listFiles(recursive)