Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9214

REFRESH with sync_ddl may fail with concurrent INVALIDATE METADATA

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-9

      Description

      The call trace for executing a REFRESH statement in Catalogd is

      JniCatalog#resetMetadata
        CatalogOpExecutor#execResetMetadata
          CatalogServiceCatalog#reloadTable
          CatalogServiceCatalog#waitForSyncDdlVersion
      

      In CatalogServiceCatalog#reloadTable(), the Tbl object may be stale if there's a concurrent reset, i.e. INVALIDATE METADATA, running. Then CatalogServiceCatalog#reloadTable will return the thrift object of a stale Table. It can't be found in the catalog cache and the topicUpdateLog_, so waitForSyncDdlVersion will finally hang or run out of attempts.

      Here is an example. Let's say table1 is an unpartitioned table and is loaded. Two queries, "Refresh table1" and "Invalidate metadata" are running concurrently.

      Thread-1 (Refresh):

      Thread-2 (Invalidate Metadata):

      • Holds the version lock and replace the whole catalog cache with a new one. Makes all existing catalog objects stale. Now the catalog version of table1 is 90.
      • Release the version lock.

      Thread-1 (Refresh):

      • Gets the version lock and table lock
      • Get a new catalog version, let's say 100. Then release version lock.
      • Load the metadata into the stale Table object. Bump its catalog version from 50 to 100.
      • Return the thrift object of the updated stale object from reloadTable
      • Goes into waitForSyncDdlVersion. Wait for an update of table1 is sent and the sent version >= 100.

      However, table1 in the catalog cache is with version 90. Unless there's another update on this table, Thread-1 will hang or run out of attempts for waiting the expected update.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stigahuang Quanlong Huang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: