Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5325

Inconsistent updates to HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ on Catalog server

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Catalog
    • Labels:
      None
    • Epic Color:
      ghx-label-2

      Description

      In the following snippet from HdfsTable.java

       private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
            HdfsPartition partition) throws Exception {
          Preconditions.checkNotNull(storageDescriptor);
          Preconditions.checkNotNull(partition);
          Path partDirPath = new Path(storageDescriptor.getLocation());
          FileSystem fs = partDirPath.getFileSystem(CONF);
          if (!fs.exists(partDirPath)) return;
      
          numHdfsFiles_ -= partition.getNumFileDescriptors();
          totalHdfsBytes_ -= partition.getSize();
          Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
          refreshFileMetadata(partition);
        }
      

      If refreshFileMetadata throws an exception before updating numHdfsFiles_ and totalHdfsBytes_, the table could get into an inconsistent state and could possibly hit a preconditions check in the next block metadata update.

      Fix: Update totalHdfsBytes_ and numHdfsFiles_ in an atomic fashion.

      Workaround: Run "invalidate table" to clear the full table state.

      This bug also affects older versions around method updatePartitionFds

      This issue can manifest itself as eventually hitting the Preconditions check, leading to this error message:

      TableLoadingException: Failed to load metadata for table: t CAUSED BY: IllegalStateException: null 
      

        Attachments

          Activity

            People

            • Assignee:
              bharathv bharath v
              Reporter:
              bharathv bharath v
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: