Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9211

CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 3.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
    • Impala 3.4.0
    • Catalog
    • ghx-label-13

    Description

      The execution flow for CreateTable is

      hold metastoreDdlLock_
      |  create table in HMS
      |  if (success) add Table to catalog cache by calling CatalogServiceCatalog.addTable()
      |--free metastoreDdlLock_

      The execution flow of CatalogServiceCatalog.addTable() is

      get db from dbCache
      init an IncompleteTable for the table
      hold writeLock of versionLock_
      |   incr catalog version and assign it to the table
      |   add table to the db
      |-- free writeLock of versionLock_
      

      The execution flow for INVALIDATE METADATA is (major part in reset()):

      hold writeLock of versionLock_
      |   create a new dbCache
      |   for each db init a new item in new dbCache
      |     get table names in db from HMS
      |     for each table in the db
      |       init a new table object and put it into the dbCache
      |   replace original dbCache with the new one
      |-- free writeLock of versionLock_

      Buggy scenario:

      Thread-1(CreateTable):
       - create table in HMS
       - get db from dbCache
       - waiting for writeLock of versionLock_
      Thread-2(Invalidate Metadata):
       - holds the writeLock of versionLock_
       - replacing dbCache with a new one.
       - find the new table in HMS, create an IncompleteTable for it using catalogVersion=900
       - add it into the new db object
      Thread-1(CreateTable):
       - get the writeLock of versionLock_
       - Incr catalog version to 1000 and assign it to the table.
       - Add the table to the stale db object.
       - Then wait in waitForSyncDdlVersion() for the update to be sent.
      Thread-3(catalog-update-gathering):
       - get current catalog verion (1000) as toVersion.
       - Collecting updates in version range (800, 1000].
       - Sent the created table with catalogVersion=900.
      Thread-1(CreateTable):
       - Find an older version (900) is sent.
       - Keep waiting a verion > 1000 for this table to be sent. Let's say no other modifications on the table happen. It remains catalogVersion=900.
      Thread-3(catalog-update-gathering):
       - Collecting updates in version range (1000, 1234].
       - Do nothing for the table since its version not in range.
      

      The created Table is added into a stale Db object. Reset() op already creates a new Db object. Thread-1 will finally run out of waiting attemps and throw an exception as described inĀ IMPALA-9135.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            stigahuang Quanlong Huang
            stigahuang Quanlong Huang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment