[IMPALA-9211] CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
• Fix Version/s: Impala 3.4.0
• Component/s: Catalog
• Labels: concurrency
• Epic Color: ghx-label-13
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 03/Dec/19 01:30
• Updated: 19/Mar/20 07:31
• Resolved: 06/Dec/19 00:02

Description

The execution flow for CreateTable is

hold metastoreDdlLock_
|  create table in HMS
|  if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
|--free metastoreDdlLock_

The execution flow of CatalogServiceCatalog.addTable() is

get db from dbCache
init an IncompleteTable for the table
hold writeLock of versionLock_
|   incr catalog version and assign it to the table
|   add table to the db
|-- free writeLock of versionLock_

The execution flow for INVALIDATE METADATA is (major part in reset()):

hold writeLock of versionLock_
|   create a new dbCache
|   for each db init a new item in new dbCache
|     get table names in db from HMS
|     for each table in the db
|       init a new table object and put it into the dbCache
|   replace original dbCache with the new one
|-- free writeLock of versionLock_

Buggy scenario:

Thread-1(CreateTable):
 - create table in HMS
 - get db from dbCache
 - waiting for writeLock of versionLock_
Thread-2(Invalidate Metadata):
 - holds the writeLock of versionLock_
 - replacing dbCache with a new one.
 - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900
 - add it into the new db object
Thread-1(CreateTable):
 - get the writeLock of versionLock_
 - Incr catalog version to 1000 and assign it to the table.
 - Add the table to the stale db object.
 - Then wait in waitForSyncDdlVersion() for the update to be sent.
Thread-3(catalog-update-gathering):
 - get current catalog verion (1000) as toVersion.
 - Collecting updates in version range (800, 1000].
 - Sent the created table with catalogVersion=900.
Thread-1(CreateTable):
 - Find an older version (900) is sent.
 - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900.
Thread-3(catalog-update-gathering):
 - Collecting updates in version range (1000, 1234].
 - Do nothing for the table since its version not in range.

The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described in IMPALA-9135.

Issue Links

• is required by: IMPALA-9135 DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
• links to: Code Review