Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
Impala 2.5.0
-
None
Description
In Frontend.java:
public TUpdateCatalogCacheResponse updateCatalogCache( TUpdateCatalogCacheRequest req) throws CatalogException { ImpaladCatalog catalog = impaladCatalog_; // If this is not a delta, this update should replace the current // Catalog contents so create a new catalog and populate it. if (!req.is_delta) catalog = new ImpaladCatalog(); TUpdateCatalogCacheResponse response = catalog.updateCatalog(req); if (!req.is_delta) { // This was not a delta update. Now that the catalog has been updated, // replace the references to impaladCatalog_/authzChecker_ ensure // clients continue don't see the catalog disappear. impaladCatalog_ = catalog; authzChecker_.set(new AuthorizationChecker(authzConfig_, impaladCatalog_.getAuthPolicy())); } return response; }
We create a new catalog instance whenever is_delta=false, which means statestore sends full update. This can happen for various reason, i.e. frontend oom processing update, catalogd restarts, etc.
After creating a new catalog instance, previous instance shared with Analyzer will become "un-updatable". In some cases, i.e. a referenced table is incomplete, Analyzer will try to load that table with
boolean requestTblLoadAndWait(Set<TableName> requestedTbls, long timeoutMs)
but frontend.java sees a different instance and it loads the complete version into this instance and tells Analyzer it is ready. But Analyzer cannot find it because it has a different catalog instance so it request load that table again.
The result is a hang.
Attachments
Issue Links
- is related to
-
IMPALA-3494 Thrift buffer overflows when serialize more than 3355443200 bytes in impala
- Resolved
-
IMPALA-3605 Frontend OOM during catalog update may end up infinite catalog update loop
- Resolved