[IMPALA-3127] Decouple partitions from tables

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 4.0.0
• Component/s: Catalog
• Labels: catalog-server performance
• Epic Link: fine-grained metadata in catalogd
• Target Version: Product Backlog
• Assignee: Quanlong Huang
• Reporter: Dimitris Tsirogiannis
• Created: 03/Mar/16 01:12
• Updated: 07/May/24 13:00
• Resolved: 12/Aug/20 09:20

Description

Currently, partitions are tightly integrated into the HdfsTable objects, making incremental metadata updates difficult to perform. Furthermore, the catalog transmits entire table metadata even when only few partitions change, introducing significant latencies, wasting network bandwidth and CPU cycles while updating table metadata at the receiving impalads. As a first step, we should decouple partitions from tables and add them as a separate level in the hierarchy of catalog entities (server-db-table-partition). Subsequently, the catalog should transmit only entities that have changed after DDL/DML statements.

Issue Links

• blocks: IMPALA-4485 Shift metadata locking level from Table to Partition wide
• blocks: IMPALA-1628 Improve synchronization protocol in the catalog
• blocks: IMPALA-3234 Catalog should send incremental metadata changes to Impalads
• causes: IMPALA-10283 IllegalStateException in applying incremental partition updates
• causes: IMPALA-12831 HdfsTable.toMinimalTCatalogObject() should hold table read lock to generate incremental updates
• causes: IMPALA-13009 Potential leak of partition deletions in the catalog topic