Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13132

Hive should lazily load and cache metastore (permanent) functions

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.13.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In Hive 0.13.1, we have noticed that as the number of databases increases, the start-up time of the Hive interactive shell increases. This is because during start-up, all databases are iterated over to fetch the permanent functions to display in the SHOW FUNCTIONS output.

      FunctionRegistry.java
        private static Set<String> getFunctionNames(boolean searchMetastore) {
          Set<String> functionNames = mFunctions.keySet();
          if (searchMetastore) {
            functionNames = new HashSet<String>(functionNames);
            try {
              Hive db = getHive();
              List<String> dbNames = db.getAllDatabases();
      
              for (String dbName : dbNames) {
                List<String> funcNames = db.getFunctions(dbName, "*");
                for (String funcName : funcNames) {
                  functionNames.add(FunctionUtils.qualifyFunctionName(funcName, dbName));
                }
              }
            } catch (Exception e) {
              LOG.error(e);
              // Continue on, we can still return the functions we've gotten to this point.
            }
          }
          return functionNames;
        }
      

      Instead of eagerly loading all metastore functions, we should only load them the first time SHOW FUNCTIONS is invoked. We should also cache the results.

      Note that this issue may have been fixed by HIVE-2573, though I haven't verified this.

        Attachments

        1. HIVE-13132.1.patch
          3 kB
          Anthony Hsu

          Issue Links

            Activity

              People

              • Assignee:
                erwaman Anthony Hsu
                Reporter:
                erwaman Anthony Hsu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: