Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3568

Possible Impala hangs and leak current catalog cache when state store sends full update

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: None
    • Component/s: Catalog
    • Labels:

      Description

      In Frontend.java:

        public TUpdateCatalogCacheResponse updateCatalogCache(
            TUpdateCatalogCacheRequest req) throws CatalogException {
          ImpaladCatalog catalog = impaladCatalog_;
      
          // If this is not a delta, this update should replace the current
          // Catalog contents so create a new catalog and populate it.
          if (!req.is_delta) catalog = new ImpaladCatalog();
      
          TUpdateCatalogCacheResponse response = catalog.updateCatalog(req);
      
          if (!req.is_delta) {
            // This was not a delta update. Now that the catalog has been updated,
            // replace the references to impaladCatalog_/authzChecker_ ensure
            // clients continue don't see the catalog disappear.
            impaladCatalog_ = catalog;
            authzChecker_.set(new AuthorizationChecker(authzConfig_,
                impaladCatalog_.getAuthPolicy()));
          }
          return response;
        }
      

      We create a new catalog instance whenever is_delta=false, which means statestore sends full update. This can happen for various reason, i.e. frontend oom processing update, catalogd restarts, etc.

      After creating a new catalog instance, previous instance shared with Analyzer will become "un-updatable". In some cases, i.e. a referenced table is incomplete, Analyzer will try to load that table with

      boolean requestTblLoadAndWait(Set<TableName> requestedTbls, long timeoutMs)
      

      but frontend.java sees a different instance and it loads the complete version into this instance and tells Analyzer it is ready. But Analyzer cannot find it because it has a different catalog instance so it request load that table again.

      The result is a hang.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                henryr Henry Robinson
                Reporter:
                HuaisiXu Huaisi Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: