Hive
  1. Hive
  2. HIVE-7167

Hive Metastore fails to start with SQLServerException

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 0.14.0
    • Component/s: Metastore
    • Labels:

      Description

      In the case that hiveserver2 uses embedded metastore and hiveserver uses remote metastore, this exception comes up when hiveserver2 and hiveserver are started simultaneously.

      metastore service status is running but when I launch hive cli, I get following metastore connection error:

      C:\apps\dist\hive-0.13.0.2.1.2.0-1660\bin>hive.cmd

      14/05/09 17:40:03 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no l
      onger has any effect.  Use hive.hmshandler.retry.* instead
      
      Logging initialized using configuration in file:/C:/apps/dist/hive-0.13.0.2.1.2.
      0-1660/conf/hive-log4j.properties
      Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeExceptio
      n: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
              at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
      a:347)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.h
      ive.metastore.HiveMetaStoreClient
              at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
      Utils.java:1413)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry
      ingMetaStoreClient.java:62)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret
      ryingMetaStoreClient.java:72)
              at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja
      va:2444)
              at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456)
              at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
      a:341)
              ... 7 more
      Caused by: java.lang.reflect.InvocationTargetException
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
      orAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
      onstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
              at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
      Utils.java:1411)
              ... 12 more
      Caused by: MetaException(message:Could not connect to meta store using any of th
      e URIs provided. Most recent failure: org.apache.thrift.transport.TTransportExce
      ption: java.net.ConnectException: Connection refused: connect
              at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto
      reClient.java:336)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS
      toreClient.java:214)
              at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      
              at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstruct
      orAccessorImpl.java:57)
              at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingC
      onstructorAccessorImpl.java:45)
              at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
              at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStore
      Utils.java:1411)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(Retry
      ingMetaStoreClient.java:62)
              at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(Ret
      ryingMetaStoreClient.java:72)
              at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.ja
      va:2444)
              at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2456)
              at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.jav
      a:341)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
      java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
      sorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.net.ConnectException: Connection refused: connect
              at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
              at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
      va:339)
              at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
      Impl.java:200)
              at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
      :182)
              at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:157)
              at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
              at java.net.Socket.connect(Socket.java:579)
              at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
              ... 19 more
      )
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaSto
      reClient.java:382)
              at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaS
      toreClient.java:214)
              ... 17 more
      

      Because of this issue, we are seeing many failures in hive related system tests.

      1. HIVE-7167.1.patch
        3 kB
        Xiaobing Zhou
      2. HIVE-7167.2.patch
        2 kB
        Xiaobing Zhou
      3. HIVE-7167.3.patch
        3 kB
        Xiaobing Zhou

        Issue Links

          Activity

          Hide
          Xiaobing Zhou added a comment -

          The patch is available now. I setup a review for submission, see the link:
          https://reviews.apache.org/r/22191/

          Show
          Xiaobing Zhou added a comment - The patch is available now. I setup a review for submission, see the link: https://reviews.apache.org/r/22191/
          Hide
          Hive QA added a comment -

          Overall: -1 no tests executed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12648081/HIVE-7167.1.patch

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/384/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/384/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-384/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Tests exited with: NonZeroExitCodeException
          Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
          + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
          + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
          + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
          + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
          + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128'
          + cd /data/hive-ptest/working/
          + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-384/source-prep.txt
          + [[ false == \t\r\u\e ]]
          + mkdir -p maven ivy
          + [[ svn = \s\v\n ]]
          + [[ -n '' ]]
          + [[ -d apache-svn-trunk-source ]]
          + [[ ! -d apache-svn-trunk-source/.svn ]]
          + [[ ! -d apache-svn-trunk-source ]]
          + cd apache-svn-trunk-source
          + svn revert -R .
          Reverted 'ql/src/test/results/clientpositive/tez/having.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/transform_ppr1.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/join0.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/count.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/mrr.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/tez_join_tests.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/subquery_in.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union2.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union8.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/bucket2.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/bucket4.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/cross_join.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/tez_union.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/subquery_exists.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/transform_ppr2.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/transform1.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/limit_pushdown.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/auto_join0.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union3.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union5.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union7.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/union9.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/script_pipe.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/bucket3.q.out'
          Reverted 'ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java'
          Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java'
          ++ egrep -v '^X|^Performing status on external'
          ++ awk '{print $2}'
          ++ svn status --no-ignore
          + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
          + svn update
          U    metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java
          U    ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out
          
          Fetching external item into 'hcatalog/src/test/e2e/harness'
          Updated external to revision 1600009.
          
          Updated to revision 1600009.
          + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
          + patchFilePath=/data/hive-ptest/working/scratch/build.patch
          + [[ -f /data/hive-ptest/working/scratch/build.patch ]]
          + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
          + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch
          The patch does not appear to apply with p0, p1, or p2
          + exit 1
          '
          

          This message is automatically generated.

          ATTACHMENT ID: 12648081

          Show
          Hive QA added a comment - Overall : -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12648081/HIVE-7167.1.patch Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/384/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/384/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-384/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]] + export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera + export PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-384/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'ql/src/test/results/clientpositive/tez/having.q.out' Reverted 'ql/src/test/results/clientpositive/tez/transform_ppr1.q.out' Reverted 'ql/src/test/results/clientpositive/tez/join0.q.out' Reverted 'ql/src/test/results/clientpositive/tez/tez_joins_explain.q.out' Reverted 'ql/src/test/results/clientpositive/tez/count.q.out' Reverted 'ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out' Reverted 'ql/src/test/results/clientpositive/tez/mrr.q.out' Reverted 'ql/src/test/results/clientpositive/tez/tez_join_tests.q.out' Reverted 'ql/src/test/results/clientpositive/tez/subquery_in.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union2.q.out' Reverted 'ql/src/test/results/clientpositive/tez/mapjoin_mapjoin.q.out' Reverted 'ql/src/test/results/clientpositive/tez/mapjoin_decimal.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union8.q.out' Reverted 'ql/src/test/results/clientpositive/tez/bucket2.q.out' Reverted 'ql/src/test/results/clientpositive/tez/bucket4.q.out' Reverted 'ql/src/test/results/clientpositive/tez/cross_product_check_1.q.out' Reverted 'ql/src/test/results/clientpositive/tez/cross_join.q.out' Reverted 'ql/src/test/results/clientpositive/tez/tez_union.q.out' Reverted 'ql/src/test/results/clientpositive/tez/subquery_exists.q.out' Reverted 'ql/src/test/results/clientpositive/tez/metadata_only_queries.q.out' Reverted 'ql/src/test/results/clientpositive/tez/transform_ppr2.q.out' Reverted 'ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out' Reverted 'ql/src/test/results/clientpositive/tez/filter_join_breaktask.q.out' Reverted 'ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out' Reverted 'ql/src/test/results/clientpositive/tez/transform1.q.out' Reverted 'ql/src/test/results/clientpositive/tez/limit_pushdown.q.out' Reverted 'ql/src/test/results/clientpositive/tez/auto_join0.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union3.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union5.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union7.q.out' Reverted 'ql/src/test/results/clientpositive/tez/union9.q.out' Reverted 'ql/src/test/results/clientpositive/tez/script_pipe.q.out' Reverted 'ql/src/test/results/clientpositive/tez/bucket3.q.out' Reverted 'ql/src/test/results/clientpositive/tez/cross_product_check_2.q.out' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/MapReduceCompiler.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf target datanucleus.log ant/target shims/target shims/0.20/target shims/0.20S/target shims/0.23/target shims/aggregator/target shims/common/target shims/common-secure/target packaging/target hbase-handler/target testutils/target jdbc/target metastore/target itests/target itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target itests/util/target hcatalog/target hcatalog/core/target hcatalog/streaming/target hcatalog/server-extensions/target hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen contrib/target service/target serde/target beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target + svn update U metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java U ql/src/test/results/clientpositive/display_colstats_tbllvl.q.out Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1600009. Updated to revision 1600009. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0, p1, or p2 + exit 1 ' This message is automatically generated. ATTACHMENT ID: 12648081
          Hide
          Sergey Shelukhin added a comment -

          1) Can you post SQLServerException you are getting?
          2) Why these 3 methods of all methods?
          3) It seems like and a hacky way to solve the problem. It can still fail again, right?

          Show
          Sergey Shelukhin added a comment - 1) Can you post SQLServerException you are getting? 2) Why these 3 methods of all methods? 3) It seems like and a hacky way to solve the problem. It can still fail again, right?
          Hide
          Sushanth Sowmyan added a comment -

          (Removing fix version tag of 0.13.0 - 0.13.0 is a released version that won't be changing, this patch will likely go into 0.14)

          Show
          Sushanth Sowmyan added a comment - (Removing fix version tag of 0.13.0 - 0.13.0 is a released version that won't be changing, this patch will likely go into 0.14)
          Hide
          Xiaobing Zhou added a comment -

          Thanks Sergey Shelukhin,
          1) Here's the exception.
          2) it's because these three methods are potential places that causes contention when two metastores both try to initialize DB in the same time.
          3) I don't think so. If one tried and failed in the first time, JDOException is caught for one more time try, if failed again, simply ignored, means another metastore succeeds.

          2014-05-15 21:06:01,659 WARN server.HiveServer2 (HiveServer2.java:startHiveServer2(112)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds
          java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:292)
          at org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:88)
          at org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:63)
          at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
          at org.apache.hive.service.cli.CLIService.init(CLIService.java:110)
          at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
          at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68)
          at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100)
          at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
          Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
          at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1413)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
          at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
          at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
          at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
          at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
          ... 9 more
          Caused by: java.lang.reflect.InvocationTargetException
          at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
          at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
          at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
          at java.lang.reflect.Constructor.newInstance(Constructor.java:525)
          at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1411)
          ... 14 more
          Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore
          NestedThrowables:
          java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint 'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object 'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE).
          at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
          at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
          at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:406)
          at org.apache.hadoop.hive.metastore.ObjectStore.grantPrivileges(ObjectStore.java:3877)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:601)
          at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108)
          at $Proxy10.grantPrivileges(Unknown Source)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles(HiveMetaStore.java:567)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398)
          at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
          at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
          at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
          at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
          ... 19 more
          Caused by: java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint 'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object 'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE).
          at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1160)
          at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424)
          at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
          at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628)
          at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596)
          at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683)
          at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86)
          at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454)
          at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:203)
          at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:267)
          at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
          ... 34 more
          2014-05-15 21:06:01,667 INFO server.HiveServer2 (HiveServer2.java:startHiveServer2(120)) - Exception caught when calling stop of HiveServer2 before retrying start
          java.lang.NullPointerException
          at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:82)
          at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:116)
          at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)

          Show
          Xiaobing Zhou added a comment - Thanks Sergey Shelukhin , 1) Here's the exception. 2) it's because these three methods are potential places that causes contention when two metastores both try to initialize DB in the same time. 3) I don't think so. If one tried and failed in the first time, JDOException is caught for one more time try, if failed again, simply ignored, means another metastore succeeds. 2014-05-15 21:06:01,659 WARN server.HiveServer2 (HiveServer2.java:startHiveServer2(112)) - Error starting HiveServer2 on attempt 1, will retry in 60 seconds java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:347) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:292) at org.apache.hive.service.cli.session.SessionManager.applyAuthorizationConfigPolicy(SessionManager.java:88) at org.apache.hive.service.cli.session.SessionManager.init(SessionManager.java:63) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.cli.CLIService.init(CLIService.java:110) at org.apache.hive.service.CompositeService.init(CompositeService.java:59) at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:68) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:100) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149) Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1413) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72) at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453) at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341) ... 9 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:525) at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1411) ... 14 more Caused by: javax.jdo.JDODataStoreException: Exception thrown flushing changes to datastore NestedThrowables: java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint 'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object 'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE). at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165) at org.apache.hadoop.hive.metastore.ObjectStore.commitTransaction(ObjectStore.java:406) at org.apache.hadoop.hive.metastore.ObjectStore.grantPrivileges(ObjectStore.java:3877) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:108) at $Proxy10.grantPrivileges(Unknown Source) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultRoles(HiveMetaStore.java:567) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:398) at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54) at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59) at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171) ... 19 more Caused by: java.sql.BatchUpdateException: Violation of UNIQUE KEY constraint 'GLOBALPRIVILEGEINDEX'. Cannot insert duplicate key in object 'dbo.GLOBAL_PRIVS'. The duplicate key value is (admin, ROLE, All, admin, ROLE). at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.executeBatch(SQLServerPreparedStatement.java:1160) at com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:424) at org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372) at org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:628) at org.datanucleus.store.rdbms.SQLController.processStatementsForConnection(SQLController.java:596) at org.datanucleus.store.rdbms.SQLController$1.transactionFlushed(SQLController.java:683) at org.datanucleus.store.connection.AbstractManagedConnection.transactionFlushed(AbstractManagedConnection.java:86) at org.datanucleus.store.connection.ConnectionManagerImpl$2.transactionFlushed(ConnectionManagerImpl.java:454) at org.datanucleus.TransactionImpl.flush(TransactionImpl.java:203) at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:267) at org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98) ... 34 more 2014-05-15 21:06:01,667 INFO server.HiveServer2 (HiveServer2.java:startHiveServer2(120)) - Exception caught when calling stop of HiveServer2 before retrying start java.lang.NullPointerException at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:82) at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:116) at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:149)
          Hide
          Xiaobing Zhou added a comment -

          Thanks Sushanth Sowmyan for pointing-out, new patch HIVE-7167.2.patch is made on trunk and attached.

          Show
          Xiaobing Zhou added a comment - Thanks Sushanth Sowmyan for pointing-out, new patch HIVE-7167 .2.patch is made on trunk and attached.
          Hide
          Xiaobing Zhou added a comment -

          closed by mistake, reopen it.

          Show
          Xiaobing Zhou added a comment - closed by mistake, reopen it.
          Hide
          Xiaobing Zhou added a comment -

          HIVE-7167.2.patch is available.

          Show
          Xiaobing Zhou added a comment - HIVE-7167 .2.patch is available.
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12651723/HIVE-7167.2.patch

          ERROR: -1 due to 5 failed/errored test(s), 5668 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas
          org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes
          org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/545/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/545/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-545/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 5 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12651723

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12651723/HIVE-7167.2.patch ERROR: -1 due to 5 failed/errored test(s), 5668 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13 org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_root_dir_external_table org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_ctas org.apache.hive.hcatalog.pig.TestHCatLoader.testReadDataPrimitiveTypes org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/545/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-Build/545/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-Build-545/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed This message is automatically generated. ATTACHMENT ID: 12651723
          Hide
          Sushanth Sowmyan added a comment -

          Sergey Shelukhin, could you please verify Xiaobing's replies and review the patch?

          Show
          Sushanth Sowmyan added a comment - Sergey Shelukhin , could you please verify Xiaobing's replies and review the patch?
          Hide
          Sergey Shelukhin added a comment -

          Hmm, sure, +1. Sorry for long time taken, I was on vacation

          Show
          Sergey Shelukhin added a comment - Hmm, sure, +1. Sorry for long time taken, I was on vacation
          Hide
          Sergey Shelukhin added a comment -

          Perhaps justification comment can be added about why the retries are done (or log message modified to mention that we assume we have metastore conflict)

          Show
          Sergey Shelukhin added a comment - Perhaps justification comment can be added about why the retries are done (or log message modified to mention that we assume we have metastore conflict)
          Hide
          Xiaobing Zhou added a comment -

          Thanks Sergey Shelukhin, I will do that to make next patch.

          Show
          Xiaobing Zhou added a comment - Thanks Sergey Shelukhin , I will do that to make next patch.
          Hide
          Sushanth Sowmyan added a comment -

          Canceling patch because it has an outstanding issue:

          createDefaultDB() has a catch MetaException removed that should not be removed, since it winds up being caught by the generic catch Exception, which causes the Metastore to fail as it attempts to cast the MetaException to a RuntimeException.

          Show
          Sushanth Sowmyan added a comment - Canceling patch because it has an outstanding issue: createDefaultDB() has a catch MetaException removed that should not be removed, since it winds up being caught by the generic catch Exception, which causes the Metastore to fail as it attempts to cast the MetaException to a RuntimeException.
          Hide
          Sushanth Sowmyan added a comment -

          Marking as patch-available again and withdrawing my objections, apologies for canceling it. I was looking at code in an older branch, and there, the catch Exception below this would cause an issue. However, HIVE-6338 invalidates that concern of mine, and this patch should be good to go.

          Show
          Sushanth Sowmyan added a comment - Marking as patch-available again and withdrawing my objections, apologies for canceling it. I was looking at code in an older branch, and there, the catch Exception below this would cause an issue. However, HIVE-6338 invalidates that concern of mine, and this patch should be good to go.
          Hide
          Vikram Dixit K added a comment -

          Can this be rebased please. I am marking this for 0.14.

          Thanks
          Vikram.

          Show
          Vikram Dixit K added a comment - Can this be rebased please. I am marking this for 0.14. Thanks Vikram.
          Hide
          Xiaobing Zhou added a comment -

          Made the 3rd patch to be reviewed.

          Show
          Xiaobing Zhou added a comment - Made the 3rd patch to be reviewed.
          Hide
          Xiaobing Zhou added a comment -

          Sergey Shelukhin can you review and check it in if possible? Thanks!

          Show
          Xiaobing Zhou added a comment - Sergey Shelukhin can you review and check it in if possible? Thanks!
          Hide
          Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12673219/HIVE-7167.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 6526 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1144/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1144/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1144/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12673219

          Show
          Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12673219/HIVE-7167.3.patch ERROR: -1 due to 1 failed/errored test(s), 6526 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1144/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1144/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1144/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12673219
          Hide
          Sergey Shelukhin added a comment -

          Will commit shortly

          Show
          Sergey Shelukhin added a comment - Will commit shortly
          Hide
          Sergey Shelukhin added a comment -

          committed to trunk and branch-0.14

          Show
          Sergey Shelukhin added a comment - committed to trunk and branch-0.14
          Hide
          Xiaobing Zhou added a comment -
          Show
          Xiaobing Zhou added a comment - Thanks Sergey Shelukhin !
          Hide
          Thejas M Nair added a comment -

          This has been fixed in 0.14 release. Please open new jira if you see any issues.

          Show
          Thejas M Nair added a comment - This has been fixed in 0.14 release. Please open new jira if you see any issues.

            People

            • Assignee:
              Xiaobing Zhou
              Reporter:
              Xiaobing Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development