[FLINK-27999] NoSuchMethodError when using Hive 3 dialect

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.1, 1.16.0
• Component/s: Connectors / Hive
• Labels: pull-request-available
• Assignee: luoyuxia
• Reporter: luoyuxia
• Created: 10/Jun/22 10:08
• Updated: 20/Jun/22 12:34
• Resolved: 14/Jun/22 08:32

Description

VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3, causing NoSuchMethodErrors when the Hive dialect is used.

When use 'mvn test -PHive3.1.1'  Hive connector, it will throw the following exception "Caused by: MetaException(message:Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")".

From the error message, we can know the reason is the configuration "datanucleus.schema.autoCreateTables" is not true. But when create HiveCatalog, we do really set the configuration "datanucleus.schema.autoCreateTables" to true in the hive-site.xml.

After some debuging, I found the reason is that some test modify a static variable, and then boil the other test.

When running HiveCatalogFactoryTest, it will call  HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will set  the static variable "hiveSiteURL" to null.

Then, if we run "HiveCatalogHiveMetadataTest", it will call HiveTestUtils#createHiveConf() to create HiveConf. The following code will create a empty HiveConf;

 

HiveConf hiveConf = new HiveConf(); 

But in the initialize function of HiveConf, it first will apply all default variables, so "datanucleus.schema.autoCreateTables" will be set to false. And the "hiveSiteURL" is null, so skip add the resource. Then, it will check ""hive.metastore.schema.verification" is true or false. If it's true, it will set "datanucleus.schema.autoCreateTables" to false. This have a higher priority and thus overwrite the value we configure in hive-site.xml.

To fix it, we only need to reset the static variable "hiveSiteURL" with our hive-site.xml.

 

 

 

 

Issue Links

• relates to: FLINK-28042 Create an extension for resetting HiveConf
• links to: GitHub Pull Request #19931