[IMPALA-12099] Catalogd NPE when refreshing a partitioned table

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Epic Color: ghx-label-8
• Assignee: Wenzhe Zhou
• Reporter: Wenzhe Zhou
• Created: 25/Apr/23 19:23
• Updated: 27/Apr/23 18:27
• Resolved: 27/Apr/23 18:13

Description

The following NPE occurs in the catalogd when it tries to refresh certain partitioned tables

I0331 13:33:04.345813 256 TAcceptQueueServer.cpp:355] New connection to server CatalogService from client <Host: 127.0.0.6 Port: 50285>
E0331 13:33:04.372188 31256 JniCatalog.java:424] Error in getting PartialCatalogObject of TABLE:ml_foodforecast.presence_history_archive_ahdp_5584. Time spent: 15ms.
I0331 13:33:04.372326 31256 jni-util.cc:286] java.lang.NullPointerException
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
at org.apache.impala.compat.MetastoreShim.getPartitionsForRefreshingFileMetadata(MetastoreShim.java:602)
at org.apache.impala.util.AcidUtils.getPartitionsForRefreshingFileMetadata(AcidUtils.java:778)
at org.apache.impala.catalog.CatalogServiceCatalog.getOrLoadTable(CatalogServiceCatalog.java:2252)
at org.apache.impala.catalog.CatalogServiceCatalog.doGetPartialCatalogObject(CatalogServiceCatalog.java:3528)
at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3454)
at org.apache.impala.catalog.CatalogServiceCatalog.getPartialCatalogObject(CatalogServiceCatalog.java:3421)
at org.apache.impala.service.JniCatalog.getPartialCatalogObject(JniCatalog.java:419)
I0331 13:33:04.372351 31256 status.cc:129] NullPointerException: null
    @ 0xdbeae3
    @ 0x166ade9
    @ 0xd88ec3
    @ 0xd6e013
    @ 0xd44e32
    @ 0xcae4e3
    @ 0xd19ab5
    @ 0x11d1b69
    @ 0x11c51b8
    @ 0x1748f74
    @ 0x174a3ba
    @ 0x1fec2d0
    @ 0x7fad37596ea4
    @ 0x7fad33fd7b0c
E0331 13:33:04.372615 31256 catalog-server.cc:239] NullPointerException: null

According to [1], Impala assumes that if a table is partitioned then all CompactionInfoStruct entries should not have null partitionname or id. While id is required, partitionname is defined as optional by Hive [2].

This assumption is false, however. As according to [3], it is possible for a compaction queue entry for a partitioned table to have null partitionname.

[1]https://github.com/apache/impala/blob/master/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java#L692-L694
[2]https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift#L1303-L1323
[3]https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java#L345

Function checkLatestCompaction() has similar code. It should also handle the null case.