[SPARK-35321] Spark 3.x can't talk to HMS 1.2.x and lower due to get_all_functions Thrift API missing

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.2, 3.1.1, 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 05/May/21 21:20
• Updated: 20/Oct/21 08:15
• Resolved: 12/Jun/21 02:33

Description

https://issues.apache.org/jira/browse/HIVE-10319 introduced a new API get_all_functions which is only supported in Hive 1.3.0/2.0.0 and up. This is called when creating a new Hive object:

  private Hive(HiveConf c, boolean doRegisterAllFns) throws HiveException {
    conf = c;
    if (doRegisterAllFns) {
      registerAllFunctionsOnce();
    }
  }

registerAllFunctionsOnce will reload all the permanent functions by calling the get_all_functions API from the megastore. In Spark, we always pass doRegisterAllFns as true, and this will cause failure:

Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions'
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231)
	... 96 more
Caused by: org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions'
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845)
	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833)

It looks like Spark doesn't really need to call registerAllFunctionsOnce since it loads the Hive permanent function directly from HMS API. The Hive FunctionRegistry is only used for loading Hive built-in functions.

Issue Links

• causes: SPARK-37069 HiveClientImpl throws NoSuchMethodError: org.apache.hadoop.hive.ql.metadata.Hive.getWithoutRegisterFns
• is related to: HIVE-21563 Improve Table#getEmptyTable performance by disable registerAllFunctionsOnce
• links to: [Github] Pull Request #32446 (sunchao)
• links to: [Github] Pull Request #32446 (sunchao)
• links to: [Github] Pull Request #32471 (sunchao)
• links to: [Github] Pull Request #32472 (sunchao)
• links to: [Github] Pull Request #32887 (sunchao)
• links to: [Github] Pull Request #32887 (sunchao)