Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5325

Inconsistent updates to HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ on Catalog server

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:
      None
    • Epic Color:
      ghx-label-2

      Description

      In the following snippet from HdfsTable.java

       private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
            HdfsPartition partition) throws Exception {
          Preconditions.checkNotNull(storageDescriptor);
          Preconditions.checkNotNull(partition);
          Path partDirPath = new Path(storageDescriptor.getLocation());
          FileSystem fs = partDirPath.getFileSystem(CONF);
          if (!fs.exists(partDirPath)) return;
      
          numHdfsFiles_ -= partition.getNumFileDescriptors();
          totalHdfsBytes_ -= partition.getSize();
          Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
          refreshFileMetadata(partition);
        }
      

      If refreshFileMetadata throws an exception before updating numHdfsFiles_ and totalHdfsBytes_, the table could get into an inconsistent state and could possibly hit a preconditions check in the next block metadata update.

      Fix: Update totalHdfsBytes_ and numHdfsFiles_ in an atomic fashion.

      Workaround: Run "invalidate table" to clear the full table state.

      This bug also affects older versions around method updatePartitionFds

      This issue can manifest itself as eventually hitting the Preconditions check, leading to this error message:

      TableLoadingException: Failed to load metadata for table: t CAUSED BY: IllegalStateException: null 
      

        Attachments

          Activity

            People

            • Assignee:
              bharathv Bharath Vissapragada
              Reporter:
              bharathv Bharath Vissapragada
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: