Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-9543

MetaException(message:Metastore contains multiple versions)

Log workAgile BoardRank to TopRank to BottomBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.13.1
    • None
    • Metastore
    • None

    Description

      When i run bin/hive command, i got the following exception:

      Logging initialized using configuration in jar:file:/home/hadoop/apache-hive-0.13.1-bin/lib/hive-common-0.13.1.jar!/hive-log4j.properties
      Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
          at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
          at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
          at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
          at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
          at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
          ... 7 more
      Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
          at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
          ... 12 more
      Caused by: MetaException(message:Metastore contains multiple versions)
          at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:6368)
          at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:6330)
          at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:6289)
          at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:6277)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:606)
          at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
          at com.sun.proxy.$Proxy9.verifySchema(Unknown Source)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:476)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
          at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
          at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
          ... 17 more
      

      And i have found two record in metastore table VERSION. after reading source code, i found following code maybe cause the problem:
      In the org.apache.hadoop.hive.metastore.ObjectStore.java:6289:

          String schemaVer = getMetaStoreSchemaVersion();
          if (schemaVer == null) {
            // metastore has no schema version information
            if (strictValidation) {
                  throw new MetaException("Version information not found in metastore. ");
                } else {
                  LOG.warn("Version information not found in metastore. "
                      + HiveConf.ConfVars.METASTORE_SCHEMA_VERIFICATION.toString() +
                      " is not enabled so recording the schema version " +
                      MetaStoreSchemaInfo.getHiveSchemaVersion());
                  setMetaStoreSchemaVersion(MetaStoreSchemaInfo.getHiveSchemaVersion(),
                      "Set by MetaStore");
              }
          }
      

      If there is exception in the getMetaStoreSchemaVersion(), Hive will insert a duplication record
      And i also found my another exception maybe the same cause as above.the exception is :

       Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
              at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
              at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
              at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
              at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
              at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
              ... 7 more
      Caused by: java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
              ... 12 more
      Caused by: javax.jdo.JDODataStoreException: Insert of object "org.apache.hadoop.hive.metastore.model.MRoleMap@2fd02622" using statement "INSERT INTO `ROLE_MAP` (`ROLE_GRANT_ID`,`GRANTOR`,`A
      DD_TIME`,`ROLE_ID`,`GRANTOR_TYPE`,`GRANT_OPTION`,`PRINCIPAL_NAME`,`PRINCIPAL_TYPE`) VALUES (?,?,?,?,?,?,?,?)" failed : Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
      NestedThrowables:
      com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
              at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:732)
              at org.datanucleus.api.jdo.JDOPersistenceManager.makePersistent(JDOPersistenceManager.java:752)
              at org.apache.hadoop.hive.metastore.ObjectStore.grantRole(ObjectStore.java:3058)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
              at com.sun.proxy.$Proxy9.grantRole(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addAdminUsers(HiveMetaStore.java:611)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:399)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:55)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:60)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
              ... 17 more
      Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'hadoop-1-admin-ROLE' for key 'USERROLEMAPINDEX'
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
              at com.mysql.jdbc.Util.handleNewInstance(Util.java:406)
              at com.mysql.jdbc.Util.getInstance(Util.java:381)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
              at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3515)
              at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3447)
              at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1951)
              at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2101)
              at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2554)
              at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1761)
              at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2046)
              at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1964)
              at com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:1949)
              at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:205)
              at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeUpdate(ParamLoggingPreparedStatement.java:399)
              at org.datanucleus.store.rdbms.SQLController.executeStatementUpdate(SQLController.java:439)
              at org.datanucleus.store.rdbms.request.InsertRequest.execute(InsertRequest.java:410)
              at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertTable(RDBMSPersistenceHandler.java:167)
              at org.datanucleus.store.rdbms.RDBMSPersistenceHandler.insertObject(RDBMSPersistenceHandler.java:143)
              at org.datanucleus.state.JDOStateManager.internalMakePersistent(JDOStateManager.java:3784)
              at org.datanucleus.state.JDOStateManager.makePersistent(JDOStateManager.java:3760)
              at org.datanucleus.ExecutionContextImpl.persistObjectInternal(ExecutionContextImpl.java:2219)
              at org.datanucleus.ExecutionContextImpl.persistObjectWork(ExecutionContextImpl.java:2065)
              at org.datanucleus.ExecutionContextImpl.persistObject(ExecutionContextImpl.java:1913)
              at org.datanucleus.ExecutionContextThreadedImpl.persistObject(ExecutionContextThreadedImpl.java:217)
              at org.datanucleus.api.jdo.JDOPersistenceManager.jdoMakePersistent(JDOPersistenceManager.java:727)
              ... 32 more
      

      The following code maybe cause the exception, in In the org.apache.hadoop.hive.metastore.ObjectStore.java:3018:

            MRole nameCheck = this.getMRole(roleName);
            if (nameCheck != null) {
              throw new InvalidObjectException("Role " + roleName + " already exists.");
            }
            int now = (int)(System.currentTimeMillis()/1000);
            MRole mRole = new MRole(roleName, now,
                ownerName);
            pm.makePersistent(mRole);
            commited = commitTransaction();
            success = true;
      

      If there is exception in the getMRole(roleName) function, Hive will try to insert a duplicate record into table ROLE_MAP, this will cause exception above.

      So can we just throw the exception instead of ignoring the exception in the getMRole(ObjectStore.java:3292) and in the getMetaStoreSchemaVersion(ObjectStore.java:6326)

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            guo_tian_qi guo_tian_qi Assign to me
            lijy83 Junyong Li

            Dates

              Created:
              Updated:

              Slack

                Issue deployment