[IMPALA-4020] Catalog update can fail due to database creation/deletion in Hive.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.6.2, Impala 2.8.0
• Component/s: Catalog
• Labels: catalog-server
• Target Version: Impala 2.6.2, Impala 2.5.4, Impala 2.8.0
• Assignee: Bharath Vissapragada
• Reporter: Anuj Phadke
• Created: 25/Aug/16 05:43
• Updated: 08/Sep/16 16:23
• Resolved: 07/Sep/16 17:42

Description

Create a hive script that creates and drops databases:
for a in

{1..100}; do echo "create database blah$a;" >> hive_script.sql; echo "drop database blah$a;" >> hive_script.sql; done

run this continuously:

beeline -u jdbc:hive2://nightly57-unsecure-1.gce.cloudera.com:10000/default -u hive -p hive -f hive_script.sql

In another session invalidate metadata:

for a in {1..100}

; do impala-shell -q "invalidate metadata"; done
Eventually you will see the following in the catalog server logs:
E0817 01:10:53.680232 19081 CatalogServiceCatalog.java:583] NoSuchObjectException(message:blah55)
E0817 01:10:53.703266 19081 catalog-server.cc:76] CatalogException: Error initializing Catalog. Catalog may be empty.

This is a corner case and this is why it happens.
Impala reset() function in catalog invokes "invalidate metadata" and hive creates and drops the tables at the following times as shown in the comment
MetaStoreClient msClient = metaStoreClientPool_.getClient(); // create database test; done in hive
try {
for (String dbName: msClient.getHiveClient().getAllDatabases())

{ List<org.apache.hadoop.hive.metastore.api.Function> javaFns = // drop database test called in hive Lists.newArrayList(); for (String javaFn: msClient.getHiveClient().getFunctions(dbName, "*")) { // This call fails and throws an exception because this database does not exist in HMS now javaFns.add(msClient.getHiveClient().getFunction(dbName, javaFn)); }

org.apache.hadoop.hive.metastore.api.Database msDb =
msClient.getHiveClient().getDatabase(dbName);
Db db = new Db(dbName, this, msDb);
// Restore UDFs that aren't persisted.
Db oldDb = oldDbCache.get(db.getName().toLowerCase());

Also recreated it by deliberately putting a breakpoints and parallely deleting the databases through hive.