[IMPALA-10283] IllegalStateException in applying incremental partition updates

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Epic Link: fine-grained metadata in catalogd
• Epic Color: ghx-label-14
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 26/Oct/20 07:57
• Updated: 25/Nov/20 12:15
• Resolved: 25/Nov/20 12:15

Description

When incremental metatdata updates are enabled (by default), catalogd sends incremental partition updates based on the last sent table snapshot. Coordinators will apply these partition updates on their existing table snapshots.

Each partition update is aka a partition instance. Partition instances are identified by partition ids. Each partition instance is a snapshot of the metadata of a partition. When applying incremental partition updates, there is a Precondition check assuming that new partition updates should not be duplicated with existing partition ids: https://github.com/apache/impala/blob/3ba8d637cdf38a68e25e573afa8d1d05047df2f6/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L515

The motivation of this check is to detect whether catalogd is sending duplicated partition updates. However, it could be hitted when the coordinator has a newer version of the table than the last sent table snapshot in catalogd. This happens when two coordinators both execute DMLs on the same table, and the DMLs finish within a catalog topic update time window. Note that coordinator will receive a table snapshot from catalogd as a response of the DML request. So one of the coordinator will have a table version that is lower than the latest version in catalogd but larger than the last sent table version in catalogd. When applying incremental partition updates on this coordinator, the Precondition check will be hitted. We should remove this check to accept this case.

To reproduce the issue, create a partitioned table and warm up its metadata cache by running any query on it (e.g. describe)

create table multi_inserts_tbl (id int) partitioned by (p int);
desc multi_inserts_tbl;

Run two inserts using two different coordinators in one command:

bin/impala-shell.sh -q "insert into multi_inserts_tbl partition (p) values (0, 0)"; bin/impala-shell.sh -i localhost:21051 -q "insert into multi_inserts_tbl partition (p) values (1, 1)"

We may find the following IllegalStateException in logs of the first coordinator:

I1026 14:54:06.127398 11497 ImpaladCatalog.java:224] Adding: TABLE:default.multi_inserts_tbl version: 1495 size: 1464 
I1026 14:54:06.127557 11497 ImpaladCatalog.java:224] Adding: CATALOG_SERVICE_ID version: 1495 size: 60
I1026 14:54:06.127887 11497 ImpaladCatalog.java:249] Adding 2 partition(s): HDFS_PARTITION:default.multi_inserts_tbl:(p=0,p=1), version=1495, size=(avg=597, min=597, max=597, sum=1194)
E1026 14:54:06.134311 11497 ImpaladCatalog.java:256] Error adding catalog object: null 
Java exception follows:
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:492)
        at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:515)
        at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:325)
        at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:254)
        at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:114)
        at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:378)
        at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:178)

This makes the first coordinator fail to update the table to the latest version:

$ bin/impala-shell.sh -q "show partitions multi_inserts_tbl"
Starting Impala Shell with no authentication using Python 2.7.16
Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
Opened TCP connection to localhost:21050
Connected to localhost:21050
Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build 3ba8d637cdf38a68e25e573afa8d1d05047df2f6)
Query: show partitions multi_inserts_tbl
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
| p     | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location                                                    |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
| 0     | -1    | 1      | 2B   | NOT CACHED   | NOT CACHED        | TEXT   | false             | hdfs://localhost:20500/test-warehouse/multi_inserts_tbl/p=0 |
| Total | -1    | 1      | 2B   | 0B           |                   |        |                   |                                                             |
+-------+-------+--------+------+--------------+-------------------+--------+-------------------+-------------------------------------------------------------+
Fetched 2 row(s) in 0.01s 

Partition p=1 is missing.

Issue Links

• is caused by: IMPALA-3127 Decouple partitions from tables