[IMPALA-12856] IllegalStateException in processing RELOAD events due to malformed HMS Partition objects

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Impala 4.5.0
• Component/s: Catalog
• Labels: catalog-2024
• Epic Link: event-processor-completeness
• Epic Color: ghx-label-11
• Assignee: Sai Hemanth Gantasala
• Reporter: Quanlong Huang
• Created: 29/Feb/24 08:19
• Updated: 28/Aug/24 16:52
• Resolved: 28/Aug/24 16:52

Description

When processing RELOAD events on partitions, catalogd fetch the Partition objects from HMS. The returned Partition objects could be malformed which causes an IllegalStateException and stops the event-processor. This was observed when a partition is added and dropped in a loop.

E0229 15:19:27.945312 12668 MetastoreEventsProcessor.java:990] Unexpected exception received while processing event
Java exception follows:
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:496)
        at org.apache.impala.catalog.HdfsTable.getTypeCompatiblePartValues(HdfsTable.java:2598)
        at org.apache.impala.catalog.HdfsTable.reloadPartitionsFromNames(HdfsTable.java:2856)
        at org.apache.impala.service.CatalogOpExecutor.reloadPartitionsFromNamesIfExists(CatalogOpExecutor.java:4805)
        at org.apache.impala.service.CatalogOpExecutor.reloadPartitionsIfExist(CatalogOpExecutor.java:4742)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.reloadPartitions(MetastoreEvents.java:1050)
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processPartitionReload(MetastoreEvents.java:2941)
        at org.apache.impala.catalog.events.MetastoreEvents$ReloadEvent.processTableEvent(MetastoreEvents.java:2906)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreTableEvent.process(MetastoreEvents.java:1248)
        at org.apache.impala.catalog.events.MetastoreEvents$MetastoreEvent.processIfEnabled(MetastoreEvents.java:672)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:1164)
        at org.apache.impala.catalog.events.MetastoreEventsProcessor.processEvents(MetastoreEventsProcessor.java:972)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
E0229 15:19:27.963455 12668 MetastoreEventsProcessor.java:1251] Event id: 8697728
Event Type: RELOAD
Event time: 1709191166
Database name: default
Table name: part_tbl
Event message: H4s

The failed check is asserting the number of partition columns cached in catalogd matches the number of partition values from the HMS object:

  public List<LiteralExpr> getTypeCompatiblePartValues(List<String> values) {
    List<LiteralExpr> result = new ArrayList<>();
    List<Column> partitionColumns = getClusteringColumns();
    Preconditions.checkState(partitionColumns.size() == values.size()); // This failed

After adding some debug logs, I found the Partition obejct got from HMS had an empty values list:

I0229 16:04:04.679625 25867 HdfsTable.java:2829] HMS Partition: Partition(values:[], dbName:default, tableName:part_tbl, createTime:1709193844, lastAccessTime:0, sd:StorageDescriptor(cols:[FieldSchema(name:i, type:int, comment:null)], location:hdf
s://localhost:20500/test-warehouse/part_tbl/p=1, inputFormat:org.apache.hadoop.mapred.TextInputFormat, outputFormat:org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat, compressed:false, numBuckets:0, serdeInfo:SerDeInfo(name:null, serializ
ationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{}), bucketCols:[], sortCols:[], parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[], skewedColValueLocationMaps:{}), storedAsSubDirectories:false), par
ameters:{}, catName:hive, writeId:0)
I0229 16:04:04.680133 25867 MetastoreEventsProcessor.java:1189] Time elapsed in processing event batch: 17.145ms
E0229 16:04:04.680475 25867 MetastoreEventsProcessor.java:990] Unexpected exception received while processing event
Java exception follows:
java.lang.IllegalStateException: partitionColumns.size(): 1, values.size(): 0. partCols: Column{name_=p, type_=INT, comment_=null, stats=ColumnStats{avgSize_=4.0, avgSerializedSize_=4.0, maxSize_=4, numDistinct_=-1, numNulls_=-1, numTrues=-1, numFalses=-1, lowValue=-1, highValue=-1}, position_=0}, partValues: 

EventProcessor should recover in this case.

On the other hand, returning a Partition object with empty partition value list seems a bug of HMS. CC hemanth619, VenuReddy

Issue Links

• is caused by: HIVE-28145 getPartitionsByNames API returns partition objects with empty values in many fields when it is executed concurrently with dropPartition API