Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
Impala 3.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
-
ghx-label-13
Description
The execution flow for CreateTable is
hold metastoreDdlLock_
| create table in HMS
| if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
|--free metastoreDdlLock_
The execution flow of CatalogServiceCatalog.addTable() is
get db from dbCache
init an IncompleteTable for the table
hold writeLock of versionLock_
| incr catalog version and assign it to the table
| add table to the db
|-- free writeLock of versionLock_
The execution flow for INVALIDATE METADATA is (major part in reset()):
hold writeLock of versionLock_ | create a new dbCache | for each db init a new item in new dbCache | get table names in db from HMS | for each table in the db | init a new table object and put it into the dbCache | replace original dbCache with the new one |-- free writeLock of versionLock_
Buggy scenario:
Thread-1(CreateTable): - create table in HMS - get db from dbCache - waiting for writeLock of versionLock_ Thread-2(Invalidate Metadata): - holds the writeLock of versionLock_ - replacing dbCache with a new one. - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900 - add it into the new db object Thread-1(CreateTable): - get the writeLock of versionLock_ - Incr catalog version to 1000 and assign it to the table. - Add the table to the stale db object. - Then wait in waitForSyncDdlVersion() for the update to be sent. Thread-3(catalog-update-gathering): - get current catalog verion (1000) as toVersion. - Collecting updates in version range (800, 1000]. - Sent the created table with catalogVersion=900. Thread-1(CreateTable): - Find an older version (900) is sent. - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900. Thread-3(catalog-update-gathering): - Collecting updates in version range (1000, 1234]. - Do nothing for the table since its version not in range.
The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described inĀ IMPALA-9135.
Attachments
Issue Links
- is required by
-
IMPALA-9135 DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA
- Open
- links to