Hive
  1. Hive
  2. HIVE-5301

Add a schema tool for offline metastore schema upgrade

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.12.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      HIVE-3764 is addressing metastore version consistency.

      Besides it would be helpful to add a tool that can leverage this version information to figure out the required set of upgrade scripts, and execute those against the configured metastore. Now that Hive includes Beeline client, it can be used to execute the scripts.

      1. HIVE-5301.3.patch.12
        48 kB
        Thejas M Nair
      2. HIVE-5301.3.patch
        48 kB
        Prasad Mujumdar
      3. HIVE-5301.3.patch
        48 kB
        Prasad Mujumdar
      4. HIVE-5301-with-HIVE-3764.0.patch
        75 kB
        Prasad Mujumdar
      5. HIVE-5301.1.patch
        36 kB
        Prasad Mujumdar

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.

          Show
          Ashutosh Chauhan added a comment - This issue has been fixed and released as part of 0.12 release. If you find further issues, please create a new jira and link it to this one.
          Hide
          Hudson added a comment -

          ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See https://builds.apache.org/job/Hive-trunk-hadoop2/457/)
          HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122)

          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
          • /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java
          • /hive/trunk/bin/ext/schemaTool.sh
          • /hive/trunk/bin/schematool
          • /hive/trunk/build.xml
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          Show
          Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See https://builds.apache.org/job/Hive-trunk-hadoop2/457/ ) HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122 ) /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java /hive/trunk/bin/ext/schemaTool.sh /hive/trunk/bin/schematool /hive/trunk/build.xml /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See https://builds.apache.org/job/Hive-trunk-h0.21/2358/)
          HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122)

          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
          • /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java
          • /hive/trunk/bin/ext/schemaTool.sh
          • /hive/trunk/bin/schematool
          • /hive/trunk/build.xml
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See https://builds.apache.org/job/Hive-trunk-h0.21/2358/ ) HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122 ) /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java /hive/trunk/bin/ext/schemaTool.sh /hive/trunk/bin/schematool /hive/trunk/build.xml /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql
          Hide
          Thejas M Nair added a comment -

          patch committed to 0.12 branch

          Show
          Thejas M Nair added a comment - patch committed to 0.12 branch
          Hide
          Thejas M Nair added a comment -

          HIVE-5301.3.patch.12 - minor rebase against 0.12 branch.

          Show
          Thejas M Nair added a comment - HIVE-5301 .3.patch.12 - minor rebase against 0.12 branch.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/)
          HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122)

          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
          • /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java
          • /hive/trunk/bin/ext/schemaTool.sh
          • /hive/trunk/bin/schematool
          • /hive/trunk/build.xml
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/ ) HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122 ) /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java /hive/trunk/bin/ext/schemaTool.sh /hive/trunk/bin/schematool /hive/trunk/build.xml /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/)
          HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122)

          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
          • /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
          • /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java
          • /hive/trunk/bin/ext/schemaTool.sh
          • /hive/trunk/bin/schematool
          • /hive/trunk/build.xml
          • /hive/trunk/metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql
          • /hive/trunk/metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql
          • /hive/trunk/metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql
          • /hive/trunk/metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql
          Show
          Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/ ) HIVE-5301 : Add a schema tool for offline metastore schema upgrade (Prasad Mujumdar via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526122 ) /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLine.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/Commands.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java /hive/trunk/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java /hive/trunk/beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java /hive/trunk/bin/ext/schemaTool.sh /hive/trunk/bin/schematool /hive/trunk/build.xml /hive/trunk/metastore/scripts/upgrade/derby/014- HIVE-3764 .derby.sql /hive/trunk/metastore/scripts/upgrade/mysql/014- HIVE-3764 .mysql.sql /hive/trunk/metastore/scripts/upgrade/oracle/014- HIVE-3764 .oracle.sql /hive/trunk/metastore/scripts/upgrade/postgres/014- HIVE-3764 .postgres.sql
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk. Thanks, Prasad!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Prasad!
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12604890/HIVE-5301.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 3161 tests executed
          Failed tests:

          org.apache.hive.hcatalog.listener.TestNotificationListener.testAMQListener
          

          Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/879/testReport
          Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/879/console

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests failed with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12604890/HIVE-5301.3.patch ERROR: -1 due to 1 failed/errored test(s), 3161 tests executed Failed tests: org.apache.hive.hcatalog.listener.TestNotificationListener.testAMQListener Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/879/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/879/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 1 tests failed This message is automatically generated.
          Hide
          Prasad Mujumdar added a comment -

          Re-attaching the .3.patch for pre-commit testing

          Show
          Prasad Mujumdar added a comment - Re-attaching the .3.patch for pre-commit testing
          Hide
          Prasad Mujumdar added a comment -

          Ok, the next patch I submit will have hcatalog build changes so you can ask me to remove it

          Ashutosh Chauhan, no worries. Really appreciate your help!

          Show
          Prasad Mujumdar added a comment - Ok, the next patch I submit will have hcatalog build changes so you can ask me to remove it Ashutosh Chauhan , no worries. Really appreciate your help!
          Hide
          Ashutosh Chauhan added a comment -

          yeah.. i know..

          Show
          Ashutosh Chauhan added a comment - yeah.. i know..
          Hide
          Brock Noland added a comment -

          Ashutosh Chauhan you've been reviewing too many patches

          Show
          Brock Noland added a comment - Ashutosh Chauhan you've been reviewing too many patches
          Hide
          Ashutosh Chauhan added a comment -

          Sorry for confusion +1 on current patch.

          Show
          Ashutosh Chauhan added a comment - Sorry for confusion +1 on current patch.
          Hide
          Ashutosh Chauhan added a comment -

          Sorry I misread copying schemaTool is fine. Somehow, I saw hcatalog related changes on RB. When, now I look at patch file they arent there.. phew. I think we are good on current patch as it is. Just reupload to have Hive QA run on it.

          Show
          Ashutosh Chauhan added a comment - Sorry I misread copying schemaTool is fine. Somehow, I saw hcatalog related changes on RB. When, now I look at patch file they arent there.. phew. I think we are good on current patch as it is. Just reupload to have Hive QA run on it.
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan Thanks for the review.
          The build.xml change is just to copy the 'schematool' wrapper script into distribution. But anycase, I will remove that and file a followup Jira. It would be fairly trivial fix, but still useful for 0.12.0 IMO.
          will update the updated patch shortly. Thanks!

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan Thanks for the review. The build.xml change is just to copy the 'schematool' wrapper script into distribution. But anycase, I will remove that and file a followup Jira. It would be fairly trivial fix, but still useful for 0.12.0 IMO. will update the updated patch shortly. Thanks!
          Hide
          Ashutosh Chauhan added a comment -

          Can you upload the patch which reverts build.xml changes, that will also trigger Hive QA run.

          Show
          Ashutosh Chauhan added a comment - Can you upload the patch which reverts build.xml changes, that will also trigger Hive QA run.
          Hide
          Ashutosh Chauhan added a comment -

          +1 modulo build.xml changes. Prasad, I think build.xml changes are not required for this work. On cursory look, they seems useful but lets do that in a seprate jira. There is lot of churn in build code currently and also a clean backport for 0.12 branch is also desirable.

          Show
          Ashutosh Chauhan added a comment - +1 modulo build.xml changes. Prasad, I think build.xml changes are not required for this work. On cursory look, they seems useful but lets do that in a seprate jira. There is lot of churn in build code currently and also a clean backport for 0.12 branch is also desirable.
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan RB is updated.

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan RB is updated.
          Hide
          Prasad Mujumdar added a comment -

          I am still not able to access RB, may be some maintenance downtime. will update as soon as I can. apologies for the delay.

          Show
          Prasad Mujumdar added a comment - I am still not able to access RB, may be some maintenance downtime. will update as soon as I can. apologies for the delay.
          Hide
          Ashutosh Chauhan added a comment -

          Can you also update RB ?

          Show
          Ashutosh Chauhan added a comment - Can you also update RB ?
          Hide
          Ashutosh Chauhan added a comment -

          Cool. Thanks for testing!

          Show
          Ashutosh Chauhan added a comment - Cool. Thanks for testing!
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan The 0.7 to 0.8 MySQL upgrade requires some changes to the patch. I am currently testing the fix. will update the patch.

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan The 0.7 to 0.8 MySQL upgrade requires some changes to the patch. I am currently testing the fix. will update the patch.
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan Agreed.
          I have tested extensively with Derby and MySQL with 0.10. Did test the upgrade of empty schema (generated using this tool) as well. But the 0.7 to 0.8 upgrade is more complex due to data move which is not tested by that. I will setup 0.7 with test data and verify the upgrade options. will attach the output of the tests.

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan Agreed. I have tested extensively with Derby and MySQL with 0.10. Did test the upgrade of empty schema (generated using this tool) as well. But the 0.7 to 0.8 upgrade is more complex due to data move which is not tested by that. I will setup 0.7 with test data and verify the upgrade options. will attach the output of the tests.
          Hide
          Ashutosh Chauhan added a comment -

          +1 Prasad Mujumdar At minimum we should test this on MySQL before committing, upgrading schema from 0.7 to 0.12 If you test this on any other db before or after, please leave it in comment here, so we know with which ones testing has been done.

          Show
          Ashutosh Chauhan added a comment - +1 Prasad Mujumdar At minimum we should test this on MySQL before committing, upgrading schema from 0.7 to 0.12 If you test this on any other db before or after, please leave it in comment here, so we know with which ones testing has been done.
          Hide
          Prasad Mujumdar added a comment -

          Ashutosh Chauhan looks like I didn't post the RB link to the ticket. sorry about that.
          The review request is on https://reviews.apache.org/r/14170/

          Show
          Prasad Mujumdar added a comment - Ashutosh Chauhan looks like I didn't post the RB link to the ticket. sorry about that. The review request is on https://reviews.apache.org/r/14170/
          Hide
          Ashutosh Chauhan added a comment -

          Prasad Mujumdar Can you create RB or phabricator link for this?

          Show
          Ashutosh Chauhan added a comment - Prasad Mujumdar Can you create RB or phabricator link for this?
          Hide
          Prasad Mujumdar added a comment -

          Combined HIVE-3764 + HIVE-5301 patch testing

          Show
          Prasad Mujumdar added a comment - Combined HIVE-3764 + HIVE-5301 patch testing
          Hide
          Prasad Mujumdar added a comment -

          Patch attached, requires HIVE-3764 patch

          Show
          Prasad Mujumdar added a comment - Patch attached, requires HIVE-3764 patch

            People

            • Assignee:
              Prasad Mujumdar
              Reporter:
              Prasad Mujumdar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development