Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4126

Adding HiveMetaStore caching when impersonation is enabled.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, HiveMetastore caching is used only when impersonation is disabled, such that all the hivemetastore call goes through NonCloseableHiveClientWithCaching [1]. However, if impersonation is enabled, caching is not used for HiveMetastore access.

      This could significantly increase the planning time when hive storage plugin is enabled, or when running a query against INFORMATION_SCHEMA. Depending on the # of databases/tables in Hive storage plugin, the planning time or INFORMATION_SCHEMA query could become unacceptable. This becomes even worse if the hive metastore is running on a different node from drillbit, making the access of hivemetastore even slower.

      We are seeing that it could takes 30~60 seconds for planning time, or execution time for INFORMATION_SCHEMA query. The long planning or execution time for INFORMATION_SCHEMA query prevents Drill from acting "interactively" for such queries.

      We should enable caching when impersonation is used. As long as the authorizer verifies the user has the access to databases/tables, we should get the data from caching. By doing that, we should see reduced number of api call to HiveMetaStore.

      [1] https://github.com/apache/drill/blob/master/contrib/storage-hive/core/src/main/java/org/apache/drill/exec/store/hive/DrillHiveMetaStoreClient.java#L299

        Attachments

          Activity

            People

            • Assignee:
              jni Jinfeng Ni
              Reporter:
              jni Jinfeng Ni
              Reviewer:
              Dechang Gu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: