Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-5218

datanucleus does not work with MS SQLServer in Hive metastore

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.12.0
    • Fix Version/s: 0.13.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      HIVE-3632 upgraded datanucleus version to 3.2.x, however, this version of datanucleus doesn't work with SQLServer as the metastore. The problem is that datanucleus tries to use fully qualified object name to find a table in the database but couldn't find it.

      If I downgrade the version to HIVE-2084, SQLServer works fine.

      It could be a bug in datanucleus.

      This is the detailed exception I'm getting when using datanucleus 3.2.x with SQL Server:

      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTa
      sk. MetaException(message:javax.jdo.JDOException: Exception thrown calling table
      .exists() for a2ee36af45e9f46c19e995bfd2d9b5fd1hivemetastore..SEQUENCE_TABLE
              at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusExc
      eption(NucleusJDOHelper.java:596)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
      rsistenceManager.java:732)
      …
              at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
      tore.java:111)
              at $Proxy0.createTable(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_core(HiveMetaStore.java:1071)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_with_environment_context(HiveMetaStore.java:1104)
      …
              at $Proxy11.create_table_with_environment_context(Unknown Source)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
      
      NestedThrowablesStackTrace:
      com.microsoft.sqlserver.jdbc.SQLServerException: There is already an object name
      d 'SEQUENCE_TABLE' in the database.
              at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError
      (SQLServerException.java:197)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServ
      erStatement.java:1493)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQ
      LServerStatement.java:775)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement$StmtExecCmd.doExecute
      (SQLServerStatement.java:676)
              at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4615)
              at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLSe
      rverConnection.java:1400)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLSer
      verStatement.java:179)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLS
      erverStatement.java:154)
              at com.microsoft.sqlserver.jdbc.SQLServerStatement.execute(SQLServerStat
      ement.java:649)
              at com.jolbox.bonecp.StatementHandle.execute(StatementHandle.java:300)
              at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatement(A
      bstractTable.java:760)
              at org.datanucleus.store.rdbms.table.AbstractTable.executeDdlStatementLi
      st(AbstractTable.java:711)
              at org.datanucleus.store.rdbms.table.AbstractTable.create(AbstractTable.
      java:425)
              at org.datanucleus.store.rdbms.table.AbstractTable.exists(AbstractTable.
      java:488)
              at org.datanucleus.store.rdbms.valuegenerator.TableGenerator.repositoryE
      xists(TableGenerator.java:242)
              at org.datanucleus.store.rdbms.valuegenerator.AbstractRDBMSGenerator.obt
      ainGenerationBlock(AbstractRDBMSGenerator.java:86)
              at org.datanucleus.store.valuegenerator.AbstractGenerator.obtainGenerati
      onBlock(AbstractGenerator.java:197)
              at org.datanucleus.store.valuegenerator.AbstractGenerator.next(AbstractG
      enerator.java:105)
              at org.datanucleus.store.rdbms.RDBMSStoreManager.getStrategyValueForGene
      rator(RDBMSStoreManager.java:2019)
              at org.datanucleus.store.AbstractStoreManager.getStrategyValue(AbstractS
      toreManager.java:1385)
              at org.datanucleus.ExecutionContextImpl.newObjectId(ExecutionContextImpl
      .java:3727)
              at org.datanucleus.state.JDOStateManager.setIdentity(JDOStateManager.jav
      a:2574)
              at org.datanucleus.state.JDOStateManager.initialiseForPersistentNew(JDOS
      tateManager.java:526)
              at org.datanucleus.state.ObjectProviderFactoryImpl.newForPersistentNew(O
      bjectProviderFactoryImpl.java:202)
              at org.datanucleus.ExecutionContextImpl.newObjectProviderForPersistentNe
      w(ExecutionContextImpl.java:1326)
              at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionC
      ontextImpl.java:2123)
              at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionConte
      xtImpl.java:1972)
              at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextIm
      pl.java:1820)
              at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionC
      ontextThreadedImpl.java:217)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPe
      rsistenceManager.java:727)
              at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersi
      stenceManager.java:752)
              at org.apache.hadoop.hive.metastore.ObjectStore.createTable(ObjectStore.
      java:646)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawS
      tore.java:111)
              at $Proxy0.createTable(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_core(HiveMetaStore.java:1071)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_tabl
      e_with_environment_context(HiveMetaStore.java:1104)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHM
      SHandler.java:103)
              at $Proxy11.create_table_with_environment_context(Unknown Source)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6417)
              at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$cr
      eate_table_with_environment_context.getResult(ThriftHiveMetastore.java:6401)
              at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
              at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
              at org.apache.hadoop.hive.metastore.TSetIpAddressProcessor.process(TSetI
      pAddressProcessor.java:48)
              at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadP
      oolServer.java:206)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
      java:1110)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
      .java:603)
              at java.lang.Thread.run(Thread.java:722)
      

        Attachments

        1. HIVE-5218-v2.patch
          0.5 kB
          shanyu zhao
        2. HIVE-5218-trunk.patch
          0.8 kB
          shanyu zhao
        3. HIVE-5218-trunk.patch
          0.8 kB
          Thejas M Nair
        4. HIVE-5218.patch
          3 kB
          shanyu zhao
        5. HIVE-5218.2.patch
          0.5 kB
          Brock Noland
        6. 0001-HIVE-5218-datanucleus-does-not-work-with-SQLServer-i.patch
          2 kB
          Sergey Soldatov

          Issue Links

            Activity

              People

              • Assignee:
                shanyu shanyu zhao
                Reporter:
                shanyu shanyu zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: