Hive
  1. Hive
  2. HIVE-1441

Extend ivy offline mode to cover metastore downloads

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.6.0
    • Fix Version/s: 0.7.0
    • Component/s: Build Infrastructure
    • Labels:
      None

      Description

      We recently started downloading datanucleus jars via ivy, and the existing ivy offilne mode doesn't cover this, so we still end up trying to contact the ivy repository even with offline mode enabled.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        1d 21h 4m 1 John Sichi 28/Jun/10 23:21
        Patch Available Patch Available Resolved Resolved
        55d 23h 31m 1 Ning Zhang 23/Aug/10 22:53
        Resolved Resolved Closed Closed
        480d 2h 6m 1 Carl Steinbach 16/Dec/11 23:59
        Carl Steinbach made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Ning Zhang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Ning Zhang added a comment -

        Committed. Thanks John!

        Show
        Ning Zhang added a comment - Committed. Thanks John!
        Hide
        Edward Capriolo added a comment -

        Fresh checkout before any after patch. Still looking into it.

         </properties>
          <testcase classname="org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote" name="testPartition" time="8.242">
            <error message="Could not connect to meta store using any of the URIs provided" type="org.apache.hadoop.hive.metastore.api.MetaException">M
        etaException(message:Could not connect to meta store using any of the URIs provided)
        	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:160)
        	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:128)
        	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:71)
        	at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote.setUp(TestHiveMetaStoreRemote.java:64)
        	at junit.framework.TestCase.runBare(TestCase.java:125)
        	at junit.framework.TestResult$1.protect(TestResult.java:106)
        	at junit.framework.TestResult.runProtected(TestResult.java:124)
        	at junit.framework.TestResult.run(TestResult.java:109)
        	at junit.framework.TestCase.run(TestCase.java:118)
        	at junit.framework.TestSuite.runTest(TestSuite.java:208)
        	at junit.framework.TestSuite.run(TestSuite.java:203)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931)
        	at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785)
        </error>
          </testcase>
          <system-out><![CDATA[Running metastore!
        ]]></system-out>
          <system-err><![CDATA[]]></system-err>
        </testsuite>
        
        
        From eclipse:
        Running metastore!
        MetaException(message:hive.metastore.warehouse.dir is not set in the config or blank)
        	at org.apache.hadoop.hive.metastore.Warehouse.<init>(Warehouse.java:58)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:155)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:125)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:1965)
        	at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote$RunMS.run(TestHiveMetaStoreRemote.java:39)
        	at java.lang.Thread.run(Thread.java:619)
        10/07/30 16:03:22 ERROR metastore.HiveMetaStore: Metastore Thrift Server threw an exception. Exiting...
        10/07/30 16:03:22 ERROR metastore.HiveMetaStore: MetaException(message:hive.metastore.warehouse.dir is not set in the config or blank)
        	at org.apache.hadoop.hive.metastore.Warehouse.<init>(Warehouse.java:58)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:155)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:125)
        	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:1965)
        	at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote$RunMS.run(TestHiveMetaStoreRemote.java:39)
        	at java.lang.Thread.run(Thread.java:619)
        
        Show
        Edward Capriolo added a comment - Fresh checkout before any after patch. Still looking into it. </properties> <testcase classname="org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote" name="testPartition" time="8.242"> <error message="Could not connect to meta store using any of the URIs provided" type="org.apache.hadoop.hive.metastore.api.MetaException">M etaException(message:Could not connect to meta store using any of the URIs provided) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:160) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:128) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.&lt;init&gt;(HiveMetaStoreClient.java:71) at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote.setUp(TestHiveMetaStoreRemote.java:64) at junit.framework.TestCase.runBare(TestCase.java:125) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:118) at junit.framework.TestSuite.runTest(TestSuite.java:208) at junit.framework.TestSuite.run(TestSuite.java:203) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:422) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:931) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:785) </error> </testcase> <system-out><![CDATA[Running metastore! ]]></system-out> <system-err><![CDATA[]]></system-err> </testsuite> From eclipse: Running metastore! MetaException(message:hive.metastore.warehouse.dir is not set in the config or blank) at org.apache.hadoop.hive.metastore.Warehouse.<init>(Warehouse.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:155) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:125) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:1965) at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote$RunMS.run(TestHiveMetaStoreRemote.java:39) at java.lang.Thread.run(Thread.java:619) 10/07/30 16:03:22 ERROR metastore.HiveMetaStore: Metastore Thrift Server threw an exception. Exiting... 10/07/30 16:03:22 ERROR metastore.HiveMetaStore: MetaException(message:hive.metastore.warehouse.dir is not set in the config or blank) at org.apache.hadoop.hive.metastore.Warehouse.<init>(Warehouse.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:155) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:125) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:1965) at org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote$RunMS.run(TestHiveMetaStoreRemote.java:39) at java.lang.Thread.run(Thread.java:619)
        Hide
        John Sichi added a comment -

        What is the failure? Look in build/metastore/test/TEST-org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote.xml

        And does it occur only with this patch applied? Offline mode only works after everything has been downloaded for the first time in online mode, so be sure to get a clean test run first in online mode.

        Show
        John Sichi added a comment - What is the failure? Look in build/metastore/test/TEST-org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote.xml And does it occur only with this patch applied? Offline mode only works after everything has been downloaded for the first time in online mode, so be sure to get a clean test run first in online mode.
        Hide
        Edward Capriolo added a comment -

        Testing.

        test:
        [junit] Running org.apache.hadoop.hive.metastore.TestHiveMetaStore
        [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 24.539 sec
        [junit] Running org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote
        [junit] Running metastore!
        [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 8.238 sec
        [junit] Test org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote FAILED

        Can this be skipped?

        Show
        Edward Capriolo added a comment - Testing. test: [junit] Running org.apache.hadoop.hive.metastore.TestHiveMetaStore [junit] Tests run: 10, Failures: 0, Errors: 0, Time elapsed: 24.539 sec [junit] Running org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote [junit] Running metastore! [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 8.238 sec [junit] Test org.apache.hadoop.hive.metastore.TestHiveMetaStoreRemote FAILED Can this be skipped?
        Hide
        Namit Jain added a comment -

        +1

        Show
        Namit Jain added a comment - +1
        John Sichi made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        John Sichi added a comment -

        Tested this out, should be ready.

        Show
        John Sichi added a comment - Tested this out, should be ready.
        John Sichi made changes -
        Field Original Value New Value
        Attachment HIVE-1441.1.patch [ 12448141 ]
        Hide
        John Sichi added a comment -

        I need to test this out some more.

        Show
        John Sichi added a comment - I need to test this out some more.
        John Sichi created issue -

          People

          • Assignee:
            John Sichi
            Reporter:
            John Sichi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development