[IMPALA-1702] "invalidate metadata" can cause duplicate TableIds

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.1
• Fix Version/s: Impala 2.8.0
• Component/s: Catalog
• Labels: catalog-server crash
• Target Version: Product Backlog
• Assignee: Huaisi Xu
• Reporter: Skye Wanderman-Milne
• Created: 27/Jan/15 07:14
• Updated: 02/Mar/17 18:20
• Resolved: 26/Oct/16 19:40

Description

CatalogServiceCatalog.reset() sets the table ID counter to 0, which is protected by the catalog lock. However, TableLoadingMgr.load() calls catalog_.getNextTableId() without taking the lock. This means that if reset() and multiples load()s are running concurrently, two tables created by load() can have the same ID (I'm not sure exactly what happens to these loaded tables since they'll get cleared from the catalog soon, but it appears they still show up in queries).

This means that if "invalidate metadata" is called concurrently with queries that touch at least two tables (e.g. inserts), there is a chance that operators in the BE will see the wrong table descriptor.

Issue Links

• is related to: IMPALA-4019 Crash when HdfsTableSink::table_desc_->partition_descriptors() is missing the default partition