[HIVE-13132] Hive should lazily load and cache metastore (permanent) functions

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.13.1
• Assignee: Anthony Hsu
• Reporter: Anthony Hsu
• Created: 24/Feb/16 01:37
• Updated: 01/Oct/19 22:07
• Resolved: 05/Mar/16 06:20

Description

In Hive 0.13.1, we have noticed that as the number of databases increases, the start-up time of the Hive interactive shell increases. This is because during start-up, all databases are iterated over to fetch the permanent functions to display in the SHOW FUNCTIONS output.

FunctionRegistry.java

  private static Set<String> getFunctionNames(boolean searchMetastore) {
    Set<String> functionNames = mFunctions.keySet();
    if (searchMetastore) {
      functionNames = new HashSet<String>(functionNames);
      try {
        Hive db = getHive();
        List<String> dbNames = db.getAllDatabases();

        for (String dbName : dbNames) {
          List<String> funcNames = db.getFunctions(dbName, "*");
          for (String funcName : funcNames) {
            functionNames.add(FunctionUtils.qualifyFunctionName(funcName, dbName));
          }
        }
      } catch (Exception e) {
        LOG.error(e);
        // Continue on, we can still return the functions we've gotten to this point.
      }
    }
    return functionNames;
  }

Instead of eagerly loading all metastore functions, we should only load them the first time SHOW FUNCTIONS is invoked. We should also cache the results.

Note that this issue may have been fixed by HIVE-2573, though I haven't verified this.

Issue Links

• is related to: HIVE-2573 Create per-session function registry
• relates to: HIVE-10319 Hive CLI startup takes a long time with a large number of databases