Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-22054

Avoid recursive listing to check if a directory is empty

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0, 1.2.0, 2.1.0, 3.1.1, 2.3.5
    • Fix Version/s: 4.0.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      During drop partition on a managed table, first we delete the directory corresponding to the partition. After that we recursively delete the parent directory as well if parent directory becomes empty. To do this emptiness check, we call Warehouse::getContentSummary(), which in turn recursively check all files and subdirectories. This is a costly operation when a directory has a lot of files or subdirectories. This overhead is even more prominent for cloud based file systems like s3. And for emptiness check, this is unnecessary too.

      This is recursive listing was introduced as part of HIVE-5220. Code snippet for reference :

      // Warehouse.java
      public boolean isEmpty(Path path) throws IOException, MetaException {
        ContentSummary contents = getFs(path).getContentSummary(path);
        if (contents != null && contents.getFileCount() == 0 && contents.getDirectoryCount() == 1) {
          return true;
        }
        return false;
      }
      
      // HiveMetaStore.java
      private void deleteParentRecursive(Path parent, int depth, boolean mustPurge, boolean needRecycle)
        throws IOException, MetaException {
        if (depth > 0 && parent != null && wh.isWritable(parent)) {
          if (wh.isDir(parent) && wh.isEmpty(parent)) {
            wh.deleteDir(parent, true, mustPurge, needRecycle);
          }
          deleteParentRecursive(parent.getParent(), depth - 1, mustPurge, needRecycle);
        }
      }
      
      // Note: FileSystem::getContentSummary() performs a recursive listing.

        Attachments

        1. HIVE-22054.patch
          2 kB
          Prabhas Kumar Samanta
        2. HIVE-22054.2.patch
          2 kB
          Prabhas Kumar Samanta

          Issue Links

            Activity

              People

              • Assignee:
                prabhas Prabhas Kumar Samanta
                Reporter:
                prabhas Prabhas Kumar Samanta
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: