Hive
  1. Hive
  2. HIVE-3014

Fix metastore test failures caused by HIVE-2757

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Issue Links

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757
        (Zhenxiao Luo via Carl Steinbach)

        Summary: HIVE-3014: Fix metastore test failures caused by HIVE-2757

        Test Plan: EMPTY

        Reviewers: JIRA, cwsteinbach

        Reviewed By: cwsteinbach

        Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3014 : Fix metastore test failures caused by HIVE-2757 Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1434 (See https://builds.apache.org/job/Hive-trunk-h0.21/1434/)
        HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757
        (Zhenxiao Luo via Carl Steinbach)

        Summary: HIVE-3014: Fix metastore test failures caused by HIVE-2757

        Test Plan: EMPTY

        Reviewers: JIRA, cwsteinbach

        Reviewed By: cwsteinbach

        Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004)

        Result = SUCCESS
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1434 (See https://builds.apache.org/job/Hive-trunk-h0.21/1434/ ) HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3014 : Fix metastore test failures caused by HIVE-2757 Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004) Result = SUCCESS cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Hide
        Phabricator added a comment -

        zhenxiao has closed the revision "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".

        Closed by cws.

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        COMMIT
        https://reviews.facebook.net/rHIVE1339004

        To: JIRA, cwsteinbach, zhenxiao

        Show
        Phabricator added a comment - zhenxiao has closed the revision " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". Closed by cws. REVISION DETAIL https://reviews.facebook.net/D3213 COMMIT https://reviews.facebook.net/rHIVE1339004 To: JIRA, cwsteinbach, zhenxiao
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Zhenxiao!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
        Hide
        Phabricator added a comment -

        cwsteinbach has accepted the revision "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        BRANCH
        HIVE-3014

        To: JIRA, cwsteinbach, zhenxiao

        Show
        Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". REVISION DETAIL https://reviews.facebook.net/D3213 BRANCH HIVE-3014 To: JIRA, cwsteinbach, zhenxiao
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".
        Reviewers: JIRA

        HIVE-3014: Fix metastore test failures caused by HIVE-2757

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/7275/

        To: JIRA, zhenxiao

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". Reviewers: JIRA HIVE-3014 : Fix metastore test failures caused by HIVE-2757 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D3213 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/7275/ To: JIRA, zhenxiao
        Hide
        Zhenxiao Luo added a comment -

        When TestMarkingPartitionSet running its first query:

        driver.run("drop database if exists tmpdb cascade");

        An exception happens, the stack trace is:

        [junit] org.apache.hadoop.hive.ql.metadata.Hive.get:135
        [junit] org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.<init>:225
        [junit] org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.<init>:172
        [junit] org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get:184
        [junit] org.apache.hadoop.hive.ql.Driver.compile:420
        [junit] org.apache.hadoop.hive.ql.Driver.compile:336
        [junit] org.apache.hadoop.hive.ql.Driver.run:914
        [junit] org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet:73
        [junit] sun.reflect.NativeMethodAccessorImpl.invoke0:-2
        [junit] sun.reflect.NativeMethodAccessorImpl.invoke:57
        [junit] sun.reflect.DelegatingMethodAccessorImpl.invoke:43
        [junit] java.lang.reflect.Method.invoke:616
        [junit] junit.framework.TestCase.runTest:168
        [junit] junit.framework.TestCase.runBare:134
        [junit] junit.framework.TestResult$1.protect:110
        [junit] junit.framework.TestResult.runProtected:128
        [junit] junit.framework.TestResult.run:113
        [junit] junit.framework.TestCase.run:124
        [junit] junit.framework.TestSuite.runTest:243
        [junit] junit.framework.TestSuite.run:238
        [junit] org.junit.internal.runners.JUnit38ClassRunner.run:83

        When coming to:

        ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java

        in the get(HiveConf c) method, the exception happens here:

        for (HiveConf.ConfVars oneVar : HiveConf.metaVars) {
        // Since metaVars are all of different types, use string for comparison
        String newVar = c.get(oneVar.varname, "");
        String oldVar = db.getConf().get(oneVar.varname, "");

        I could check that when looping through the first oneVar, oneVar is actually null. So when referencing varname, an exception happens.

        The bug is due to incorrect initialization of HiveConf.

        findHadoopBinary() is defined in HiveConf.java, but outside of "public static enum ConfVars" definition, the compiler would not initialize ConfVars(and other public static enums, including metaVars) if the enum values initialization is calling a function out of the enum definition.

        So metaVars is null, and caused the bug.

        It should be fixed by putting findHadoopBinary() into enum ConfVars definition.

        Show
        Zhenxiao Luo added a comment - When TestMarkingPartitionSet running its first query: driver.run("drop database if exists tmpdb cascade"); An exception happens, the stack trace is: [junit] org.apache.hadoop.hive.ql.metadata.Hive.get:135 [junit] org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.<init>:225 [junit] org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.<init>:172 [junit] org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get:184 [junit] org.apache.hadoop.hive.ql.Driver.compile:420 [junit] org.apache.hadoop.hive.ql.Driver.compile:336 [junit] org.apache.hadoop.hive.ql.Driver.run:914 [junit] org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet:73 [junit] sun.reflect.NativeMethodAccessorImpl.invoke0:-2 [junit] sun.reflect.NativeMethodAccessorImpl.invoke:57 [junit] sun.reflect.DelegatingMethodAccessorImpl.invoke:43 [junit] java.lang.reflect.Method.invoke:616 [junit] junit.framework.TestCase.runTest:168 [junit] junit.framework.TestCase.runBare:134 [junit] junit.framework.TestResult$1.protect:110 [junit] junit.framework.TestResult.runProtected:128 [junit] junit.framework.TestResult.run:113 [junit] junit.framework.TestCase.run:124 [junit] junit.framework.TestSuite.runTest:243 [junit] junit.framework.TestSuite.run:238 [junit] org.junit.internal.runners.JUnit38ClassRunner.run:83 When coming to: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java in the get(HiveConf c) method, the exception happens here: for (HiveConf.ConfVars oneVar : HiveConf.metaVars) { // Since metaVars are all of different types, use string for comparison String newVar = c.get(oneVar.varname, ""); String oldVar = db.getConf().get(oneVar.varname, ""); I could check that when looping through the first oneVar, oneVar is actually null. So when referencing varname, an exception happens. The bug is due to incorrect initialization of HiveConf. findHadoopBinary() is defined in HiveConf.java, but outside of "public static enum ConfVars" definition, the compiler would not initialize ConfVars(and other public static enums, including metaVars) if the enum values initialization is calling a function out of the enum definition. So metaVars is null, and caused the bug. It should be fixed by putting findHadoopBinary() into enum ConfVars definition.
        Hide
        Carl Steinbach added a comment -

        It looks like the patch for HIVE-2757 is causing failures in the following metastore tests:

        • TestMarkPartition
        • TestMarkPartitionRemote
        • TestMetaStoreEventListener

        I tested the patch for HIVE-2757 before committing and did not encounter these errors. I attribute this oversight to running 'clean' instead of 'very-clean'. In a separate patch we should probably modify the clean target to remove the following directories in the local Ivy repo:

        local/org.apache.hive
        cache/org.apache.hive

        Show
        Carl Steinbach added a comment - It looks like the patch for HIVE-2757 is causing failures in the following metastore tests: TestMarkPartition TestMarkPartitionRemote TestMetaStoreEventListener I tested the patch for HIVE-2757 before committing and did not encounter these errors. I attribute this oversight to running 'clean' instead of 'very-clean'. In a separate patch we should probably modify the clean target to remove the following directories in the local Ivy repo: local/org.apache.hive cache/org.apache.hive

          People

          • Assignee:
            Zhenxiao Luo
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development