Hive
  1. Hive
  2. HIVE-3014

Fix metastore test failures caused by HIVE-2757

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Issue Links

        Activity

        Carl Steinbach created issue -
        Hide
        Carl Steinbach added a comment -

        It looks like the patch for HIVE-2757 is causing failures in the following metastore tests:

        • TestMarkPartition
        • TestMarkPartitionRemote
        • TestMetaStoreEventListener

        I tested the patch for HIVE-2757 before committing and did not encounter these errors. I attribute this oversight to running 'clean' instead of 'very-clean'. In a separate patch we should probably modify the clean target to remove the following directories in the local Ivy repo:

        local/org.apache.hive
        cache/org.apache.hive

        Show
        Carl Steinbach added a comment - It looks like the patch for HIVE-2757 is causing failures in the following metastore tests: TestMarkPartition TestMarkPartitionRemote TestMetaStoreEventListener I tested the patch for HIVE-2757 before committing and did not encounter these errors. I attribute this oversight to running 'clean' instead of 'very-clean'. In a separate patch we should probably modify the clean target to remove the following directories in the local Ivy repo: local/org.apache.hive cache/org.apache.hive
        Carl Steinbach made changes -
        Field Original Value New Value
        Link This issue is broken by HIVE-2757 [ HIVE-2757 ]
        Carl Steinbach made changes -
        Link This issue is duplicated by HIVE-3021 [ HIVE-3021 ]
        Hide
        Zhenxiao Luo added a comment -

        When TestMarkingPartitionSet running its first query:

        driver.run("drop database if exists tmpdb cascade");

        An exception happens, the stack trace is:

        [junit] org.apache.hadoop.hive.ql.metadata.Hive.get:135
        [junit] org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.<init>:225
        [junit] org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.<init>:172
        [junit] org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get:184
        [junit] org.apache.hadoop.hive.ql.Driver.compile:420
        [junit] org.apache.hadoop.hive.ql.Driver.compile:336
        [junit] org.apache.hadoop.hive.ql.Driver.run:914
        [junit] org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet:73
        [junit] sun.reflect.NativeMethodAccessorImpl.invoke0:-2
        [junit] sun.reflect.NativeMethodAccessorImpl.invoke:57
        [junit] sun.reflect.DelegatingMethodAccessorImpl.invoke:43
        [junit] java.lang.reflect.Method.invoke:616
        [junit] junit.framework.TestCase.runTest:168
        [junit] junit.framework.TestCase.runBare:134
        [junit] junit.framework.TestResult$1.protect:110
        [junit] junit.framework.TestResult.runProtected:128
        [junit] junit.framework.TestResult.run:113
        [junit] junit.framework.TestCase.run:124
        [junit] junit.framework.TestSuite.runTest:243
        [junit] junit.framework.TestSuite.run:238
        [junit] org.junit.internal.runners.JUnit38ClassRunner.run:83

        When coming to:

        ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java

        in the get(HiveConf c) method, the exception happens here:

        for (HiveConf.ConfVars oneVar : HiveConf.metaVars) {
        // Since metaVars are all of different types, use string for comparison
        String newVar = c.get(oneVar.varname, "");
        String oldVar = db.getConf().get(oneVar.varname, "");

        I could check that when looping through the first oneVar, oneVar is actually null. So when referencing varname, an exception happens.

        The bug is due to incorrect initialization of HiveConf.

        findHadoopBinary() is defined in HiveConf.java, but outside of "public static enum ConfVars" definition, the compiler would not initialize ConfVars(and other public static enums, including metaVars) if the enum values initialization is calling a function out of the enum definition.

        So metaVars is null, and caused the bug.

        It should be fixed by putting findHadoopBinary() into enum ConfVars definition.

        Show
        Zhenxiao Luo added a comment - When TestMarkingPartitionSet running its first query: driver.run("drop database if exists tmpdb cascade"); An exception happens, the stack trace is: [junit] org.apache.hadoop.hive.ql.metadata.Hive.get:135 [junit] org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.<init>:225 [junit] org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.<init>:172 [junit] org.apache.hadoop.hive.ql.parse.SemanticAnalyzerFactory.get:184 [junit] org.apache.hadoop.hive.ql.Driver.compile:420 [junit] org.apache.hadoop.hive.ql.Driver.compile:336 [junit] org.apache.hadoop.hive.ql.Driver.run:914 [junit] org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet:73 [junit] sun.reflect.NativeMethodAccessorImpl.invoke0:-2 [junit] sun.reflect.NativeMethodAccessorImpl.invoke:57 [junit] sun.reflect.DelegatingMethodAccessorImpl.invoke:43 [junit] java.lang.reflect.Method.invoke:616 [junit] junit.framework.TestCase.runTest:168 [junit] junit.framework.TestCase.runBare:134 [junit] junit.framework.TestResult$1.protect:110 [junit] junit.framework.TestResult.runProtected:128 [junit] junit.framework.TestResult.run:113 [junit] junit.framework.TestCase.run:124 [junit] junit.framework.TestSuite.runTest:243 [junit] junit.framework.TestSuite.run:238 [junit] org.junit.internal.runners.JUnit38ClassRunner.run:83 When coming to: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java in the get(HiveConf c) method, the exception happens here: for (HiveConf.ConfVars oneVar : HiveConf.metaVars) { // Since metaVars are all of different types, use string for comparison String newVar = c.get(oneVar.varname, ""); String oldVar = db.getConf().get(oneVar.varname, ""); I could check that when looping through the first oneVar, oneVar is actually null. So when referencing varname, an exception happens. The bug is due to incorrect initialization of HiveConf. findHadoopBinary() is defined in HiveConf.java, but outside of "public static enum ConfVars" definition, the compiler would not initialize ConfVars(and other public static enums, including metaVars) if the enum values initialization is calling a function out of the enum definition. So metaVars is null, and caused the bug. It should be fixed by putting findHadoopBinary() into enum ConfVars definition.
        Hide
        Phabricator added a comment -

        zhenxiao requested code review of "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".
        Reviewers: JIRA

        HIVE-3014: Fix metastore test failures caused by HIVE-2757

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        AFFECTED FILES
        common/src/java/org/apache/hadoop/hive/conf/HiveConf.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/7275/

        To: JIRA, zhenxiao

        Show
        Phabricator added a comment - zhenxiao requested code review of " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". Reviewers: JIRA HIVE-3014 : Fix metastore test failures caused by HIVE-2757 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D3213 AFFECTED FILES common/src/java/org/apache/hadoop/hive/conf/HiveConf.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/7275/ To: JIRA, zhenxiao
        Phabricator made changes -
        Attachment HIVE-3014.D3213.1.patch [ 12526864 ]
        Zhenxiao Luo made changes -
        Attachment HIVE-3014.1.patch.txt [ 12527364 ]
        Zhenxiao Luo made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Phabricator added a comment -

        cwsteinbach has accepted the revision "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        BRANCH
        HIVE-3014

        To: JIRA, cwsteinbach, zhenxiao

        Show
        Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". REVISION DETAIL https://reviews.facebook.net/D3213 BRANCH HIVE-3014 To: JIRA, cwsteinbach, zhenxiao
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Zhenxiao!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
        Carl Steinbach made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Phabricator added a comment -

        zhenxiao has closed the revision "HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757".

        Closed by cws.

        REVISION DETAIL
        https://reviews.facebook.net/D3213

        COMMIT
        https://reviews.facebook.net/rHIVE1339004

        To: JIRA, cwsteinbach, zhenxiao

        Show
        Phabricator added a comment - zhenxiao has closed the revision " HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 ". Closed by cws. REVISION DETAIL https://reviews.facebook.net/D3213 COMMIT https://reviews.facebook.net/rHIVE1339004 To: JIRA, cwsteinbach, zhenxiao
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1434 (See https://builds.apache.org/job/Hive-trunk-h0.21/1434/)
        HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757
        (Zhenxiao Luo via Carl Steinbach)

        Summary: HIVE-3014: Fix metastore test failures caused by HIVE-2757

        Test Plan: EMPTY

        Reviewers: JIRA, cwsteinbach

        Reviewed By: cwsteinbach

        Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004)

        Result = SUCCESS
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1434 (See https://builds.apache.org/job/Hive-trunk-h0.21/1434/ ) HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3014 : Fix metastore test failures caused by HIVE-2757 Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004) Result = SUCCESS cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Jeff Hammerbacher made changes -
        Link This issue relates to HIVE-3035 [ HIVE-3035 ]
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757
        (Zhenxiao Luo via Carl Steinbach)

        Summary: HIVE-3014: Fix metastore test failures caused by HIVE-2757

        Test Plan: EMPTY

        Reviewers: JIRA, cwsteinbach

        Reviewed By: cwsteinbach

        Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004
        Files :

        • /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3014 [jira] Fix metastore test failures caused by HIVE-2757 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3014 : Fix metastore test failures caused by HIVE-2757 Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D3213 (Revision 1339004) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1339004 Files : /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Ashutosh Chauhan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        4d 15h 1 Zhenxiao Luo 15/May/12 16:46
        Patch Available Patch Available Resolved Resolved
        12h 42m 1 Carl Steinbach 16/May/12 05:29
        Resolved Resolved Closed Closed
        239d 15h 23m 1 Ashutosh Chauhan 10/Jan/13 19:53

          People

          • Assignee:
            Zhenxiao Luo
            Reporter:
            Carl Steinbach
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development