Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9211

CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 3.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
    • Fix Version/s: Impala 3.4.0
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-13

      Description

      The execution flow for CreateTable is

      hold metastoreDdlLock_
      |  create table in HMS
      |  if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
      |--free metastoreDdlLock_

      The execution flow of CatalogServiceCatalog.addTable() is

      get db from dbCache
      init an IncompleteTable for the table
      hold writeLock of versionLock_
      |   incr catalog version and assign it to the table
      |   add table to the db
      |-- free writeLock of versionLock_
      

      The execution flow for INVALIDATE METADATA is (major part in reset()):

      hold writeLock of versionLock_
      |   create a new dbCache
      |   for each db init a new item in new dbCache
      |     get table names in db from HMS
      |     for each table in the db
      |       init a new table object and put it into the dbCache
      |   replace original dbCache with the new one
      |-- free writeLock of versionLock_

      Buggy scenario:

      Thread-1(CreateTable):
       - create table in HMS
       - get db from dbCache
       - waiting for writeLock of versionLock_
      Thread-2(Invalidate Metadata):
       - holds the writeLock of versionLock_
       - replacing dbCache with a new one.
       - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900
       - add it into the new db object
      Thread-1(CreateTable):
       - get the writeLock of versionLock_
       - Incr catalog version to 1000 and assign it to the table.
       - Add the table to the stale db object.
       - Then wait in waitForSyncDdlVersion() for the update to be sent.
      Thread-3(catalog-update-gathering):
       - get current catalog verion (1000) as toVersion.
       - Collecting updates in version range (800, 1000].
       - Sent the created table with catalogVersion=900.
      Thread-1(CreateTable):
       - Find an older version (900) is sent.
       - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900.
      Thread-3(catalog-update-gathering):
       - Collecting updates in version range (1000, 1234].
       - Do nothing for the table since its version not in range.
      

      The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described inĀ IMPALA-9135.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stigahuang Quanlong Huang
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: