Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: Metastore
    • Labels:

      Description

      For some unknown reason, we see customer's HMS can not start because there are multiple entries in their HMS VERSION table. Schematool should provide a way to validate the HMS db and provide warning and fix options for this kind of issues.

      1. HIVE-15074.1.patch
        4 kB
        Chaoyu Tang
      2. HIVE-15074.patch
        3 kB
        Chaoyu Tang

        Activity

        Hide
        ctang.ma Chaoyu Tang added a comment -

        Aihua Xu, Yongzhi Chen, and Naveen Gangam, could you help to review the patch, thanks

        Show
        ctang.ma Chaoyu Tang added a comment - Aihua Xu , Yongzhi Chen , and Naveen Gangam , could you help to review the patch, thanks
        Hide
        ngangam Naveen Gangam added a comment -

        Functionally, it looks good to me.

        Just for semantics to make it consistent with other verify* calls, do we want to make it return a boolean to indicate success/failure instead of throwing an exception? maybe overload verifySchemaVersion() that catches the exception and returns a boolean? I will defer to Aihua Xu for the final call on this.

        Show
        ngangam Naveen Gangam added a comment - Functionally, it looks good to me. Just for semantics to make it consistent with other verify* calls, do we want to make it return a boolean to indicate success/failure instead of throwing an exception? maybe overload verifySchemaVersion() that catches the exception and returns a boolean? I will defer to Aihua Xu for the final call on this.
        Hide
        hiveqa Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12840735/HIVE-15074.patch

        ERROR: -1 due to build exiting with an error

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/console
        Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2312/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Tests exited with: NonZeroExitCodeException
        Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N'
        2016-11-29 01:44:23.550
        + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
        + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
        + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
        + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
        + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
        + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
        + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
        + export 'MAVEN_OPTS=-Xmx1g '
        + MAVEN_OPTS='-Xmx1g '
        + cd /data/hiveptest/working/
        + tee /data/hiveptest/logs/PreCommit-HIVE-Build-2312/source-prep.txt
        + [[ false == \t\r\u\e ]]
        + mkdir -p maven ivy
        + [[ git = \s\v\n ]]
        + [[ git = \g\i\t ]]
        + [[ -z master ]]
        + [[ -d apache-github-source-source ]]
        + [[ ! -d apache-github-source-source/.git ]]
        + [[ ! -d apache-github-source-source ]]
        + date '+%Y-%m-%d %T.%3N'
        2016-11-29 01:44:23.553
        + cd apache-github-source-source
        + git fetch origin
        + git reset --hard HEAD
        HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios (Sushanth Sowmyan reviewed by Vaibhav Gumashta)
        + git clean -f -d
        Removing ql/src/test/queries/clientpositive/specialChar.q
        Removing ql/src/test/results/clientpositive/specialChar.q.out
        + git checkout master
        Already on 'master'
        Your branch is up-to-date with 'origin/master'.
        + git reset --hard origin/master
        HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios (Sushanth Sowmyan reviewed by Vaibhav Gumashta)
        + git merge --ff-only origin/master
        Already up-to-date.
        + date '+%Y-%m-%d %T.%3N'
        2016-11-29 01:44:24.497
        + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
        + patchFilePath=/data/hiveptest/working/scratch/build.patch
        + [[ -f /data/hiveptest/working/scratch/build.patch ]]
        + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
        + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch
        error: a/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java: No such file or directory
        error: a/itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java: No such file or directory
        The patch does not appear to apply with p0, p1, or p2
        + exit 1
        '
        

        This message is automatically generated.

        ATTACHMENT ID: 12840735 - PreCommit-HIVE-Build

        Show
        hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12840735/HIVE-15074.patch ERROR: -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2312/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2312/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2016-11-29 01:44:23.550 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-2312/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2016-11-29 01:44:23.553 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios (Sushanth Sowmyan reviewed by Vaibhav Gumashta) + git clean -f -d Removing ql/src/test/queries/clientpositive/specialChar.q Removing ql/src/test/results/clientpositive/specialChar.q.out + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 63bdfa6 HIVE-15284: Add junit test to test replication scenarios (Sushanth Sowmyan reviewed by Vaibhav Gumashta) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2016-11-29 01:44:24.497 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java: No such file or directory error: a/itests/hive-unit/src/test/java/org/apache/hive/beeline/TestSchemaTool.java: No such file or directory The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12840735 - PreCommit-HIVE-Build
        Hide
        aihuaxu Aihua Xu added a comment -

        That's right. We are trying to return boolean rather than throwing exceptions since we will continue the following functions to detect other issues.

        Show
        aihuaxu Aihua Xu added a comment - That's right. We are trying to return boolean rather than throwing exceptions since we will continue the following functions to detect other issues.
        Hide
        ctang.ma Chaoyu Tang added a comment -

        Should not we stop the validations if there are some schema version issues (incompatible, duplicated invalid entries, no version etc)? Otherwise, the following schema validations, especially those based on the version like validateSchemaTables, will become meaningless since the based schema is already not right. I can put the validateSchemaVersions as the first validation point. Any thought?

        Show
        ctang.ma Chaoyu Tang added a comment - Should not we stop the validations if there are some schema version issues (incompatible, duplicated invalid entries, no version etc)? Otherwise, the following schema validations, especially those based on the version like validateSchemaTables, will become meaningless since the based schema is already not right. I can put the validateSchemaVersions as the first validation point. Any thought?
        Hide
        ctang.ma Chaoyu Tang added a comment -

        Provide a new patch based on Naveen Gangam's, and Aihua Xu's comments, though not totally convinced that we should still continue the validations if the version has already be detected not right
        Naveen Gangam & Aihua Xu, could you review the patch? Thanks

        Show
        ctang.ma Chaoyu Tang added a comment - Provide a new patch based on Naveen Gangam 's, and Aihua Xu 's comments, though not totally convinced that we should still continue the validations if the version has already be detected not right Naveen Gangam & Aihua Xu , could you review the patch? Thanks
        Hide
        hiveqa Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch

        SUCCESS: +1 due to 1 test(s) being added or modified.

        ERROR: -1 due to 6 failed/errored test(s), 10747 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=133)
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/console
        Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2329/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 6 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build

        Show
        hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 6 failed/errored test(s), 10747 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=133) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2329/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2329/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed This message is automatically generated. ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build
        Hide
        ctang.ma Chaoyu Tang added a comment -

        The failed tests are not related to this patch.

        Show
        ctang.ma Chaoyu Tang added a comment - The failed tests are not related to this patch.
        Hide
        hiveqa Hive QA added a comment -

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch

        SUCCESS: +1 due to 1 test(s) being added or modified.

        ERROR: -1 due to 8 failed/errored test(s), 10747 tests executed
        Failed tests:

        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59)
        org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38)
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=133)
        org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=133)
        org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=91)
        

        Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/testReport
        Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/console
        Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2330/

        Messages:

        Executing org.apache.hive.ptest.execution.TestCheckPhase
        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        Tests exited with: TestsFailedException: 8 tests failed
        

        This message is automatically generated.

        ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build

        Show
        hiveqa Hive QA added a comment - Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12840920/HIVE-15074.1.patch SUCCESS: +1 due to 1 test(s) being added or modified. ERROR: -1 due to 8 failed/errored test(s), 10747 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample2] (batchId=5) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample4] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample6] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample7] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample9] (batchId=38) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a] (batchId=133) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[transform_ppr2] (batchId=133) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2] (batchId=91) Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2330/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2330/ Messages: Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed This message is automatically generated. ATTACHMENT ID: 12840920 - PreCommit-HIVE-Build
        Hide
        aihuaxu Aihua Xu added a comment -

        The patch looks good. + 1.

        Show
        aihuaxu Aihua Xu added a comment - The patch looks good. + 1.
        Hide
        ctang.ma Chaoyu Tang added a comment -

        Committed to 2.2.0. Thanks Aihua Xu & Naveen Gangam for review.

        Show
        ctang.ma Chaoyu Tang added a comment - Committed to 2.2.0. Thanks Aihua Xu & Naveen Gangam for review.
        Hide
        leftylev Lefty Leverenz added a comment -

        Does this need to be documented in the wiki?

        Show
        leftylev Lefty Leverenz added a comment - Does this need to be documented in the wiki? The Hive Schema Tool
        Hide
        aihuaxu Aihua Xu added a comment -

        Yes. It should get documented in SchemaTool section. I will try to doc it after the child tasks are done and doc it along with HIVE-15206. Thanks Lefty Leverenz for reminding.

        Show
        aihuaxu Aihua Xu added a comment - Yes. It should get documented in SchemaTool section. I will try to doc it after the child tasks are done and doc it along with HIVE-15206 . Thanks Lefty Leverenz for reminding.
        Hide
        leftylev Lefty Leverenz added a comment -

        Okay, I added a TODOC2.2 label. Thanks.

        Show
        leftylev Lefty Leverenz added a comment - Okay, I added a TODOC2.2 label. Thanks.

          People

          • Assignee:
            ctang.ma Chaoyu Tang
            Reporter:
            ychena Yongzhi Chen
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development