[IMPALA-5325] Inconsistent updates to HdfsTable#totalHdfsBytes_ and HdfsTable#numHdfsFiles_ on Catalog server

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0, Impala 2.9.0
• Fix Version/s: Impala 2.8.0
• Component/s: Catalog
• Epic Color: ghx-label-2
• Assignee: Bharath Vissapragada
• Reporter: Bharath Vissapragada
• Created: 15/May/17 17:32
• Updated: 22/Feb/18 00:25
• Resolved: 26/May/17 15:21

Description

In the following snippet from HdfsTable.java

 private void loadPartitionFileMetadata(StorageDescriptor storageDescriptor,
      HdfsPartition partition) throws Exception {
    Preconditions.checkNotNull(storageDescriptor);
    Preconditions.checkNotNull(partition);
    Path partDirPath = new Path(storageDescriptor.getLocation());
    FileSystem fs = partDirPath.getFileSystem(CONF);
    if (!fs.exists(partDirPath)) return;

    numHdfsFiles_ -= partition.getNumFileDescriptors();
    totalHdfsBytes_ -= partition.getSize();
    Preconditions.checkState(numHdfsFiles_ >= 0 && totalHdfsBytes_ >= 0);
    refreshFileMetadata(partition);
  }

If refreshFileMetadata throws an exception before updating numHdfsFiles_ and totalHdfsBytes_, the table could get into an inconsistent state and could possibly hit a preconditions check in the next block metadata update.

Fix: Update totalHdfsBytes_ and numHdfsFiles_ in an atomic fashion.

Workaround: Run "invalidate table" to clear the full table state.

This bug also affects older versions around method updatePartitionFds

This issue can manifest itself as eventually hitting the Preconditions check, leading to this error message:

TableLoadingException: Failed to load metadata for table: t CAUSED BY: IllegalStateException: null