[IMPALA-13009] Potential leak of partition deletions in the catalog topic

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, Impala 4.1.2, Impala 4.3.0
• Fix Version/s: Impala 4.4.0
• Component/s: Catalog
• Epic Color: ghx-label-12
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 17/Apr/24 12:04
• Updated: 24/May/24 09:30
• Resolved: 07/May/24 05:43

Description

Catalogd might not send partition deletions to the catalog topic in the following scenario:

• Some partitions of a table are dropped.
• The HdfsTable object is removed sequentially before catalogd collects the dropped partitions.

In such case, catalogd loses track of the dropped partitions so their updates keep existing in the catalog topic, until the partition names are reused again.

Note that the HdfsTable object can be removed by commands like DropTable or INVALIDATE.

The leaked partitions will be detected when a coordinator restarts. An IllegalStateException complaining stale partitions will be reported, causing the table not being added to the catalog cache of coordinator.

E0417 16:41:22.317298 20746 ImpaladCatalog.java:264] Error adding catalog object: Received stale partition in a statestore update: THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, type:TColumnType(types:[TTypeNode(type:SCALAR, scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, stats:TTableStats(num_rows:-1), is_marked_cached:false, hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, partition_name:p=106, hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, blockSize:0))
Java exception follows:
java.lang.IllegalStateException: Received stale partition in a statestore update: THdfsPartition(partitionKeyExprs:[TExpr(nodes:[TExprNode(node_type:INT_LITERAL, type:TColumnType(types:[TTypeNode(type:SCALAR, scalar_type:TScalarType(type:INT))]), num_children:0, is_constant:true, int_literal:TIntLiteral(value:106), is_codegen_disabled:false)])], location:THdfsPartitionLocation(prefix_index:0, suffix:p=106), id:138, file_desc:[THdfsFileDesc(file_desc_data:18 00 00 00 00 00 00 00 00 00 0E 00 1C 00 18 00 10 00 00 00 08 00 04 00 0E 00 00 00 18 00 00 00 8B 0E 2D EB 8E 01 00 00 04 00 00 00 00 00 00 00 0C 00 00 00 01 00 00 00 4C 00 00 00 36 00 00 00 34 34 34 37 62 35 66 34 62 30 65 64 66 64 65 31 2D 32 33 33 61 64 62 38 35 30 30 30 30 30 30 30 30 5F 36 36 34 31 30 39 33 37 33 5F 64 61 74 61 2E 30 2E 74 78 74 00 00 0C 00 14 00 00 00 0C 00...)], access_level:READ_WRITE, stats:TTableStats(num_rows:-1), is_marked_cached:false, hms_parameters:{transient_lastDdlTime=1713342582, totalSize=4, numFilesErasureCoded=0, numFiles=1}, num_blocks:1, total_file_size_bytes:4, has_incremental_stats:false, write_id:0, db_name:default, tbl_name:my_part, partition_name:p=106, hdfs_storage_descriptor:THdfsStorageDescriptor(lineDelim:10, fieldDelim:1, collectionDelim:1, mapKeyDelim:1, escapeChar:0, quoteChar:1, fileFormat:TEXT, blockSize:0))
        at com.google.common.base.Preconditions.checkState(Preconditions.java:512)
        at org.apache.impala.catalog.ImpaladCatalog.addTable(ImpaladCatalog.java:523)
        at org.apache.impala.catalog.ImpaladCatalog.addCatalogObject(ImpaladCatalog.java:334)
        at org.apache.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:262)
        at org.apache.impala.service.FeCatalogManager$CatalogdImpl.updateCatalogCache(FeCatalogManager.java:120)
        at org.apache.impala.service.Frontend.updateCatalogCache(Frontend.java:565)
        at org.apache.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:196) 

Workaround
Add back the stale partitions to the table (if the table is also dropped, add back the table first) using the same partition names. Then drop them again. This gives catalogd chances to send deletions for them.

Issue Links

• is caused by: IMPALA-3127 Decouple partitions from tables