[IMPALA-11050] Skip file metadata reloading in AlterPartition event from event processor in catalogd

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Catalog
• Epic Link: event-processor-perf
• Epic Color: ghx-label-4
• Assignee: Sourabh Goyal
• Reporter: Sourabh Goyal
• Created: 07/Dec/21 13:37
• Updated: 25/Aug/22 02:23
• Resolved: 03/Jul/22 09:04

Description

HdfsPartition in catalogD is a collection of files and each file is represented by a FileDescriptor. A fd contains:

1.  RelativePath of this file
2. Length of the file
3. Compression info like GZIP etc
4. Modification time of the file
5. Blocks info that belong to this file. Each block has info like offset, length, diskIds

When the event processor processes an AlterPartitionEvent, currently it reloads the partition again along with file metadata reloading. Reloading of file metadata is a relatively expensive operation as it involves listing files in the underlying filesystem. From the Impala shell, an alter partition is triggered via ALTER TABLE PARTITION <partition_spec> <operation>.  Here operation can be: 

1. Update stats
2. Drop stats
3. Set file format
4. Set row format
5. Set table properties
6. Unset table properties
7. Set serde properties
8. Unset serde properties
9. Set cached <hdfs-pool-name>
10. Unset cached <hdfs-pool-name>
11. Set location

 

For transactional tables:

For transactional tables, if the incremental refresh is enabled, event processor reloades file metadata at the CommitTxn event. Since there is no way to know whether the commit txn event was due to alter_partition or some other event, file metadata reloading can not be skipped. 

For external tables: 

From the operations above, any operation that affects the underlying storage descriptor of a partition should trigger the file metadata reloading. Operations 3,4,7,8,11 are such operations. 

 

How to detect change in file descriptor of a partition:

HMS partition object received in alter_partition event contains metastore.api.StorageDescriptor object. This object has fields like: 

• List<FieldSchema> cols
• String location 
• String inputFormat
• String outputFormat
• Boolean compressed
• Boolean numBuckets
• SerdeInfo serdeInfo
• LIst<String> bucketCols
• List<Order> sortCols
• Map<String, String> params

 

Fetch HMS partition object from alterPartition event and compare its storage descriptor properties with the similar properties of already cached partition object

Unknowns: 

1. If a partition is cached in HDFS, should we always reload its filemetadata (irrespective of any of the operations mentioned above) to get most up to date block locations? 

 

cc - vihangk1 stigahuang hsnusonic 

Issue Links

• is related to: IMPALA-11534 Skip reloading file metadata for some ALTER_TABLE events