The global invalidate metadata takes a write lock on the versionLock_. However, the locking protocol for ddls release the versionLock_ as soon as the table level lock is acquired. This allows for a concurrent invalidate metadata to run while the DDL operation is in progress. This can lead to weird race conditions. One such example is below can lead to functions disappearing from the catalog until a invalidate metadata is issued again.
Following sequence of events can reproduce this race condition:
When the create function statement is executing in CatalogOpExecutor we apply the alterDatabase in HMS to persist the new db parameters here: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java#L1409
Note the we have released the versionLock_ by line 1409. Meanwhile a concurrent invalidate metadata fetches the db params from HMS here https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java#L1326 which will override the parameters of the newly created Db object. Hence effectively we are removing the function from the parameters since the operation 1 to alterDatabase is not yet committed in HMS.
All subsequent commands of show functions, drop function will show inconsistent results. I was able to reproduce this race condition by added a sleep statement just before the alterDatabase call in the createFunction method.
Note: Above code links are based of commit hash 7dd13f72784514a59f82c9a7a5e2250503dbfaf0