Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Catalog
    • Labels:
    • Epic Color:
      ghx-label-4

      Description

      Seen in an integration Jenkins job for 2.9. 'Metastore service failed to start within 300.0 seconds.'

      from hive-metastore.out:

      17/06/02 05:58:40 WARN conf.HiveConf: HiveConf of name hive.access.conf.url does not exist
      javax.jdo.JDOException: Exception thrown when executing query
              at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
              at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:230)
              at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:7346)
              at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:7325)
              at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7280)
              at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7264)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
              at com.sun.proxy.$Proxy0.verifySchema(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:661)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:713)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:508)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6313)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6308)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6558)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6485)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      
      NestedThrowablesStackTrace:
      org.postgresql.util.PSQLException: ERROR: column A0.SCHEMA_VERSION_V2 does not exist
        Position: 101
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2102)
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1835)
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:500)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:388)
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:273)
              at com.jolbox.bonecp.PreparedStatementHandle.executeQuery(PreparedStatementHandle.java:174)
              at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeQuery(ParamLoggingPreparedStatement.java:381)
              at org.datanucleus.store.rdbms.SQLController.executeStatementQuery(SQLController.java:504)
              at org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:651)
              at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
              at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
              at org.datanucleus.store.query.Query.execute(Query.java:1654)
              at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:221)
              at org.apache.hadoop.hive.metastore.ObjectStore.getMSchemaVersion(ObjectStore.java:7346)
              at org.apache.hadoop.hive.metastore.ObjectStore.getMetaStoreSchemaVersion(ObjectStore.java:7325)
              at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7280)
              at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7264)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:103)
              at com.sun.proxy.$Proxy0.verifySchema(Unknown Source)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:661)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:713)
              at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:508)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
              at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6313)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6308)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6558)
              at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6485)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      

        Activity

        Hide
        lv Lars Volker added a comment -

        IMPALA-5426: Update Hive schema script to 1.1.0

        A recent update to Hive changed its schema, which is causing
        the metastore not to come up when run against the latest version
        of Hive as we've hard coded the Hive schema script version in
        bin/create-test-configuration.sh to an old version.

        This patch updates the version to the latest. The schema script
        is included in Hive in the toolchain and the new version will
        already be present.

        By itself, this patch does not actually change the Hive schema
        by default as the version we usually build against doesn't have
        the change. A following patch will update impala-config.sh to pull
        in the latest version of Hive.

        In the long run, we should switch to using Hive's schema tool,
        which can do this for us automatically (IMPALA-5430).

        Testing:

        • Ran an exhaustive private Jenkins build that passed.

        Change-Id: I9ea3269c1f95f76d8c02b76a5dea3ca3aa324b70
        Reviewed-on: http://gerrit.cloudera.org:8080/7072
        Reviewed-by: Lars Volker <lv@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        lv Lars Volker added a comment - IMPALA-5426 : Update Hive schema script to 1.1.0 A recent update to Hive changed its schema, which is causing the metastore not to come up when run against the latest version of Hive as we've hard coded the Hive schema script version in bin/create-test-configuration.sh to an old version. This patch updates the version to the latest. The schema script is included in Hive in the toolchain and the new version will already be present. By itself, this patch does not actually change the Hive schema by default as the version we usually build against doesn't have the change. A following patch will update impala-config.sh to pull in the latest version of Hive. In the long run, we should switch to using Hive's schema tool, which can do this for us automatically ( IMPALA-5430 ). Testing: Ran an exhaustive private Jenkins build that passed. Change-Id: I9ea3269c1f95f76d8c02b76a5dea3ca3aa324b70 Reviewed-on: http://gerrit.cloudera.org:8080/7072 Reviewed-by: Lars Volker <lv@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            twmarshall Thomas Tauber-Marshall
            Reporter:
            twmarshall Thomas Tauber-Marshall
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development