Hive
  1. Hive
  2. HIVE-3764

Support metastore version consistency check

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0, 0.10.0, 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      Today there's no version/compatibility information stored in hive metastore. Also the datanucleus configuration property to automatically create missing tables is enabled by default. If you happen to start an older or newer hive or don't run the correct upgrade scripts during migration, the metastore would end up corrupted. The autoCreate schema is not always sufficient to upgrade metastore when migrating to newer release. It's not supported with all databases. Besides the migration often involves altering existing table, changing or moving data etc.

      Hence it's very useful to have some consistency check to make sure that hive is using correct metastore and for production systems the schema is not automatically by running hive.

      1. HIVE-3764-trunk.3.patch
        163 kB
        Prasad Mujumdar
      2. HIVE-3764-12.3.patch
        40 kB
        Prasad Mujumdar
      3. HIVE-3764.2.patch
        40 kB
        Prasad Mujumdar
      4. HIVE-3764.1.patch
        39 kB
        Prasad Mujumdar

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Hide
          Thejas M Nair added a comment -

          Patch committed to 0.12 branch.

          Show
          Thejas M Nair added a comment - Patch committed to 0.12 branch.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2350 (See https://builds.apache.org/job/Hive-trunk-h0.21/2350/)
          HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288)

          • /hive/trunk/build-common.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2350 (See https://builds.apache.org/job/Hive-trunk-h0.21/2350/ ) HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288 ) /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2 #449 (See https://builds.apache.org/job/Hive-trunk-hadoop2/449/)
          HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288)

          • /hive/trunk/build-common.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #449 (See https://builds.apache.org/job/Hive-trunk-hadoop2/449/ ) HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288 ) /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #110 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/110/)
          HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288)

          • /hive/trunk/build-common.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #110 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/110/ ) HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288 ) /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #177 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/177/)
          HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288)

          • /hive/trunk/build-common.xml
          • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
          • /hive/trunk/conf/hive-default.xml.template
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
          • /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java
          • /hive/trunk/metastore/src/model/package.jdo
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java
          • /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hive-trunk-hadoop1-ptest #177 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/177/ ) HIVE-3764 : Support metastore version consistency check (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1525288 ) /hive/trunk/build-common.xml /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java /hive/trunk/conf/hive-default.xml.template /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/hive-schema-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade-0.12.0-to-0.13.0.derby.sql /hive/trunk/metastore/scripts/upgrade/derby/upgrade.order.derby /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/hive-schema-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade-0.12.0-to-0.13.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/mysql/upgrade.order.mysql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/hive-schema-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade-0.12.0-to-0.13.0.oracle.sql /hive/trunk/metastore/scripts/upgrade/oracle/upgrade.order.oracle /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/hive-schema-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade-0.12.0-to-0.13.0.postgres.sql /hive/trunk/metastore/scripts/upgrade/postgres/upgrade.order.postgres /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java /hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java /hive/trunk/metastore/src/model/package.jdo /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Prasad!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Prasad!
          Hide
          Prasad Mujumdar added a comment -

          Rebased patch for trunk

          Show
          Prasad Mujumdar added a comment - Rebased patch for trunk
          Hide
          Prasad Mujumdar added a comment -

          Rebased patch for 0.12

          Show
          Prasad Mujumdar added a comment - Rebased patch for 0.12
          Hide
          Prasad Mujumdar added a comment -

          Thanks Ashutosh!
          I will rebase the patch on trunk and upload patches trunk and 0.12 later today.

          Show
          Prasad Mujumdar added a comment - Thanks Ashutosh! I will rebase the patch on trunk and upload patches trunk and 0.12 later today.
          Hide
          Ashutosh Chauhan added a comment -

          +1 Thanks, Prasad for quick turnaround on this one.

          Show
          Ashutosh Chauhan added a comment - +1 Thanks, Prasad for quick turnaround on this one.
          Hide
          Prasad Mujumdar added a comment -

          Updated patch per review comments.

          Show
          Prasad Mujumdar added a comment - Updated patch per review comments.
          Hide
          Ashutosh Chauhan added a comment -

          Left some comments on RB.

          Show
          Ashutosh Chauhan added a comment - Left some comments on RB.
          Hide
          Brock Noland added a comment -

          It was my fault! I removed it.

          Show
          Brock Noland added a comment - It was my fault! I removed it.
          Hide
          Prasad Mujumdar added a comment -

          Brock Noland yes, HIVE-3764.1.patch is the latest. The .4.patch left in there was the one that you added (to refresh the correct patch for test run) hence I couldn't remove that. sorry about the confusion.

          Show
          Prasad Mujumdar added a comment - Brock Noland yes, HIVE-3764 .1.patch is the latest. The .4.patch left in there was the one that you added (to refresh the correct patch for test run) hence I couldn't remove that. sorry about the confusion.
          Hide
          Brock Noland added a comment -

          FYI it looks like you tried to delete "HIVE-3764.4.patch" but it's still there? Anyway based on the date it looks like "HIVE-3764.1.patch" is the current patch.

          Show
          Brock Noland added a comment - FYI it looks like you tried to delete " HIVE-3764 .4.patch" but it's still there? Anyway based on the date it looks like " HIVE-3764 .1.patch" is the current patch.
          Hide
          Prasad Mujumdar added a comment -

          The schema tool part is addressed via HIVE-5301.

          Show
          Prasad Mujumdar added a comment - The schema tool part is addressed via HIVE-5301 .
          Hide
          Prasad Mujumdar added a comment -

          New review request for the updated patch at https://reviews.apache.org/r/14169/

          Show
          Prasad Mujumdar added a comment - New review request for the updated patch at https://reviews.apache.org/r/14169/
          Hide
          Prasad Mujumdar added a comment -

          Patch specific to version consistency check

          Show
          Prasad Mujumdar added a comment - Patch specific to version consistency check
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan sure, I can split the patch into two separate tickets. I modified to add another option and more test cases etc. will attach the updated patches directly to the new tickets.

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan sure, I can split the patch into two separate tickets. I modified to add another option and more test cases etc. will attach the updated patches directly to the new tickets.
          Hide
          Ashutosh Chauhan added a comment -

          Sure. I think it makes sense to include in 0.12. Currently, I am having bit of hard time in untangling changes related to new schemaTool and version consistency checks. Prasad Mujumdar If its not too much of work for you, may I request you to split this in two jiras: one (3764) for version consistency check and a new one for schemaTool. If its too troublesome, let me know I will try to look this one itself.

          Show
          Ashutosh Chauhan added a comment - Sure. I think it makes sense to include in 0.12. Currently, I am having bit of hard time in untangling changes related to new schemaTool and version consistency checks. Prasad Mujumdar If its not too much of work for you, may I request you to split this in two jiras: one (3764) for version consistency check and a new one for schemaTool. If its too troublesome, let me know I will try to look this one itself.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12603014/HIVE-3764.4.patch

          ERROR: -1 due to 1 failed/errored test(s), 3116 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/733/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/733/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603014/HIVE-3764.4.patch ERROR: -1 due to 1 failed/errored test(s), 3116 tests executed Failed tests: org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testNegativeCliDriver_mapreduce_stack_trace_hadoop20 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/733/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/733/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
          Hide
          Thejas M Nair added a comment -

          I think this is a very useful feature. Ashutosh Chauhan Can you also please take a look as you have done a lot of work on metastore.

          Show
          Thejas M Nair added a comment - I think this is a very useful feature. Ashutosh Chauhan Can you also please take a look as you have done a lot of work on metastore.
          Hide
          Brock Noland added a comment -

          Since the 0.13-additional-patch is only schema for 0.13, we want HIVE-3764.4.patch to run as part of the tests. The problem is that 0.13-additional-patch was uploaded later so that will be tested. Therefore I am reuploading your original patch HIVE-3764.4.patch.

          Show
          Brock Noland added a comment - Since the 0.13-additional-patch is only schema for 0.13, we want HIVE-3764 .4.patch to run as part of the tests. The problem is that 0.13-additional-patch was uploaded later so that will be tested. Therefore I am reuploading your original patch HIVE-3764 .4.patch.
          Hide
          Brock Noland added a comment -

          Thejas M Nair, I'd really like to get this into 0.12. I just had two people email me directly yesterday and both were using the incorrect metastore version. I've also seen this issue dozens of times in the past couple years. Additionally the patch looks quite good.

          Show
          Brock Noland added a comment - Thejas M Nair , I'd really like to get this into 0.12. I just had two people email me directly yesterday and both were using the incorrect metastore version. I've also seen this issue dozens of times in the past couple years. Additionally the patch looks quite good.
          Hide
          Prasad Mujumdar added a comment -

          Additional files for trunk (0.13)

          Show
          Prasad Mujumdar added a comment - Additional files for trunk (0.13)
          Hide
          Prasad Mujumdar added a comment -
          Show
          Prasad Mujumdar added a comment - Review request on https://reviews.apache.org/r/14120/
          Hide
          Prasad Mujumdar added a comment -

          Patch for 0.12 branch

          Show
          Prasad Mujumdar added a comment - Patch for 0.12 branch
          Hide
          Shreepadma Venugopalan added a comment -

          Irrespective of which option we choose to generate version numbers, we should not execute the insert/update version number statement in the schema creation/upgrade script until all other statements in the schema creation/upgrade script have completed without errors. Thanks.

          Show
          Shreepadma Venugopalan added a comment - Irrespective of which option we choose to generate version numbers, we should not execute the insert/update version number statement in the schema creation/upgrade script until all other statements in the schema creation/upgrade script have completed without errors. Thanks.
          Hide
          Shreepadma Venugopalan added a comment -

          I think adding the consistency check is a good idea too. I've not looked into all the details of the code, but I noticed that the metastore version number is the hive release version. While this makes the version numbers easily readable, we would need to provide scripts and perform a metastore upgrade on every Hive release even if there are no other patches in the release that require a metastore schema upgrade. The other option would be to use version numbers from a monotonically increasing sequence instead and bump up the version number only if there are changes in a release that require a metastore upgrade. Wondering if you have considered the later option. Thanks.

          Show
          Shreepadma Venugopalan added a comment - I think adding the consistency check is a good idea too. I've not looked into all the details of the code, but I noticed that the metastore version number is the hive release version. While this makes the version numbers easily readable, we would need to provide scripts and perform a metastore upgrade on every Hive release even if there are no other patches in the release that require a metastore schema upgrade. The other option would be to use version numbers from a monotonically increasing sequence instead and bump up the version number only if there are changes in a release that require a metastore upgrade. Wondering if you have considered the later option. Thanks.
          Hide
          Prasad Mujumdar added a comment -

          @Asuthosh, Thanks for the comments.
          HiveServer2 patch already has a mechanism to extract the version. I didn't want to duplicate that again here, so its using a hardcoded version for time being. I will change it based on what that code.

          Show
          Prasad Mujumdar added a comment - @Asuthosh, Thanks for the comments. HiveServer2 patch already has a mechanism to extract the version. I didn't want to duplicate that again here, so its using a hardcoded version for time being. I will change it based on what that code.
          Hide
          Ashutosh Chauhan added a comment -

          I like the idea of this consistency check and having autoCreate turned off by default.
          I havent looked at the patch in much detail, but hard-coding version number in the code is not the great idea. Standard mechanism for this is to determine current version at compile time via saveVersion scripts and then use that. HIVE-2926 is trying to add that in. Either we should finish that one up or take the bits out from it of saveVersion and include in this patch.

          Show
          Ashutosh Chauhan added a comment - I like the idea of this consistency check and having autoCreate turned off by default. I havent looked at the patch in much detail, but hard-coding version number in the code is not the great idea. Standard mechanism for this is to determine current version at compile time via saveVersion scripts and then use that. HIVE-2926 is trying to add that in. Either we should finish that one up or take the bits out from it of saveVersion and include in this patch.
          Hide
          Prasad Mujumdar added a comment -

          Code review request on https://reviews.apache.org/r/8314/

          Show
          Prasad Mujumdar added a comment - Code review request on https://reviews.apache.org/r/8314/
          Hide
          Prasad Mujumdar added a comment -

          The metastore version is set the in code and a new table is created in metastore to store it on the disk. The schema creation and upgrade scripts add the version string to the metastore.
          The version is verified while opening the metastore. A new property hive.metastore.schema.verification (disabled by default) is added to enable this verification. Setting this property also disables autoCreateSchema and enables fixedDataStore for datanucleus.

          Show
          Prasad Mujumdar added a comment - The metastore version is set the in code and a new table is created in metastore to store it on the disk. The schema creation and upgrade scripts add the version string to the metastore. The version is verified while opening the metastore. A new property hive.metastore.schema.verification (disabled by default) is added to enable this verification. Setting this property also disables autoCreateSchema and enables fixedDataStore for datanucleus.

            People

            • Assignee:
              Prasad Mujumdar
              Reporter:
              Prasad Mujumdar
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development