Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5218

datanucleus does not work with MS SQLServer in Hive metastore

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 0.12.0
    • 0.13.0
    • Metastore
    • None

    Description

      HIVE-3632 upgraded datanucleus version to 3.2.x, however, this version of datanucleus doesn't work with SQLServer as the metastore. The problem is that datanucleus tries to use fully qualified object name to find a table in the database but couldn't find it.

      If I downgrade the version to HIVE-2084, SQLServer works fine.

      It could be a bug in datanucleus.

      This is the detailed exception I'm getting when using datanucleus 3.2.x with SQL Server:

      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTa
      sk. MetaException(message:javax.jdo.JDOException: Exception thrown calling table
      .exists() for a2ee36af45e9f46c19e995bfd2d9b5fd1hivemetastore..SEQUENCE_TABLE
              at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusExc
      eption(NucleusJDOHelper.java:596)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
      rsistenceManager.java:732)
      …
              at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
      tore.java:111)
              at $Proxy0.createTable(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_core(HiveMetaStore.java:1071)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_with_environment_context(HiveMetaStore.java:1104)
      …
              at $Proxy11.create_table_with_environment_context(Unknown Source)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
      
      NestedThrowablesStackTrace:
      com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object name
      d 'SEQUENCE_TABLE' in the database.
              at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError
      (SQLServerException.java:197)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServ
      erStatement.java:1493)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQ
      LServerStatement.java:775)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute
      (SQLServerStatement.java:676)
              at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4615)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLSe
      rverConnection.java:1400)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLSer
      verStatement.java:179)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLS
      erverStatement.java:154)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStat
      ement.java:649)
              at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
              at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(A
      bstractTable.java:760)
              at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementLi
      st(AbstractTable.java:711)
              at org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.
      java:425)
              at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.
      java:488)
              at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryE
      xists(TableGenerator.java:242)
              at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obt
      ainGenerationBlock(AbstractRDBMSGenerator.java:86)
              at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerati
      onBlock(AbstractGenerator.java:197)
              at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractG
      enerator.java:105)
              at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGene
      rator(RDBMSStoreManager.java:2019)
              at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractS
      toreManager.java:1385)
              at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl
      .java:3727)
              at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.jav
      a:2574)
              at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOS
      tateManager.java:526)
              at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(O
      bjectProviderFactoryImpl.java:202)
              at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNe
      w(ExecutionContextImpl.java:1326)
              at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionC
      ontextImpl.java:2123)
              at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionConte
      xtImpl.java:1972)
              at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextIm
      pl.java:1820)
              at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionC
      ontextThreadedImpl.java:217)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
      rsistenceManager.java:727)
              at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersi
      stenceManager.java:752)
              at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.
      java:646)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
      tore.java:111)
              at $Proxy0.createTable(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_core(HiveMetaStore.java:1071)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_with_environment_context(HiveMetaStore.java:1104)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHM
      SHandler.java:103)
              at $Proxy11.create_table_with_environment_context(Unknown Source)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
              at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
              at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetI
      pAddressProcessor.java:48)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadP
      oolServer.java:206)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
      java:1110)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
      .java:603)
              at java.lang.Thread.run(Thread.java:722)
      

      Attachments

        1. 0001-HIVE-5218-datanucleus-does-not-work-with-SQLServer-i.patch
          2 kB
          Sergey Soldatov
        2. HIVE-5218.patch
          3 kB
          shanyu zhao
        3. HIVE-5218-v2.patch
          0.5 kB
          shanyu zhao
        4. HIVE-5218.2.patch
          0.5 kB
          Brock Noland
        5. HIVE-5218-trunk.patch
          0.8 kB
          shanyu zhao
        6. HIVE-5218-trunk.patch
          0.8 kB
          Thejas Nair

        Issue Links

          Activity

            People

              shanyu shanyu zhao
              shanyu shanyu zhao
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: