Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-27999

NoSuchMethodError when using Hive 3 dialect

    XMLWordPrintableJSON

Details

    Description

      VirtualColumn.VIRTUAL_COLUMN_NAMES uses different types in Hive2 / Hive3, causing NoSuchMethodErrors when the Hive dialect is used.

      When use 'mvn test -PHive3.1.1'  Hive connector, it will throw the following exception "Caused by: MetaException(message:Required table missing : "DBS" in Catalog "" Schema "". DataNucleus requires this table to perform its persistence operations. Either your MetaData is incorrect, or you need to enable "datanucleus.schema.autoCreateTables")".

      From the error message, we can know the reason is the configuration "datanucleus.schema.autoCreateTables" is not true. But when create HiveCatalog, we do really set the configuration "datanucleus.schema.autoCreateTables" to true in the hive-site.xml.

      After some debuging, I found the reason is that some test modify a static variable, and then boil the other test.

      When running HiveCatalogFactoryTest, it will call  HiveCatalog#createHiveConf(@Nullable String hiveConfDir, @Nullable String hadoopConfDir) to create a HiveConf for HiveCatalog. In this method, it will set  the static variable "hiveSiteURL" to null.

      Then, if we run "HiveCatalogHiveMetadataTest", it will call HiveTestUtils#createHiveConf() to create HiveConf. The following code will create a empty HiveConf;

       

      HiveConf hiveConf = new HiveConf(); 

      But in the initialize function of HiveConf, it first will apply all default variables, so "datanucleus.schema.autoCreateTables" will be set to false. And the "hiveSiteURL" is null, so skip add the resource. Then, it will check ""hive.metastore.schema.verification" is true or false. If it's true, it will set "datanucleus.schema.autoCreateTables" to false. This have a higher priority and thus overwrite the value we configure in hive-site.xml.

      To fix it, we only need to reset the static variable "hiveSiteURL" with our hive-site.xml.

       

       

       

       

      Attachments

        Issue Links

          Activity

            People

              luoyuxia luoyuxia
              luoyuxia luoyuxia
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: