Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13132

Hive should lazily load and cache metastore (permanent) functions

Log workAgile BoardRank to TopRank to BottomVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 0.13.1
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      In Hive 0.13.1, we have noticed that as the number of databases increases, the start-up time of the Hive interactive shell increases. This is because during start-up, all databases are iterated over to fetch the permanent functions to display in the SHOW FUNCTIONS output.

      FunctionRegistry.java
        private static Set<String> getFunctionNames(boolean searchMetastore) {
          Set<String> functionNames = mFunctions.keySet();
          if (searchMetastore) {
            functionNames = new HashSet<String>(functionNames);
            try {
              Hive db = getHive();
              List<String> dbNames = db.getAllDatabases();
      
              for (String dbName : dbNames) {
                List<String> funcNames = db.getFunctions(dbName, "*");
                for (String funcName : funcNames) {
                  functionNames.add(FunctionUtils.qualifyFunctionName(funcName, dbName));
                }
              }
            } catch (Exception e) {
              LOG.error(e);
              // Continue on, we can still return the functions we've gotten to this point.
            }
          }
          return functionNames;
        }
      

      Instead of eagerly loading all metastore functions, we should only load them the first time SHOW FUNCTIONS is invoked. We should also cache the results.

      Note that this issue may have been fixed by HIVE-2573, though I haven't verified this.

        Attachments

        1. HIVE-13132.1.patch
          3 kB
          Anthony Hsu

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment