Hive
  1. Hive
  2. HIVE-2883

Metastore client doesnt close connection properly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      While closing connection, it always fail with following trace. Seemingly, it doesnt have any harmful effects.

      12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore client
      org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
      	at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
      	at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
      	at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
      	at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
      	at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421)
      	at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)
      

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          Looks like client.shutdown() is unnecessarily called after client.close() I guess only one of those calls will suffice.

          Show
          Ashutosh Chauhan added a comment - Looks like client.shutdown() is unnecessarily called after client.close() I guess only one of those calls will suffice.
          Hide
          Ashutosh Chauhan added a comment -

          My analysis is client already knows about transport, so inside client.shutdown() it is calling transport.close() which closes underlying outputstream. So, we should not do transport.close() ourselves in HiveMetaStoreClient.close() ? But, I can't be sure of this hypothesis since implementation of client.shutdown() is in FaceBookService.client (which is in libfb303.jar) I cant seem to find sources for this class anywhere.

          So, I request Facebook contributors of Hive project to either fix this bug or release sources of libfb303.jar so that I can verify and fix this myself.

          Show
          Ashutosh Chauhan added a comment - My analysis is client already knows about transport, so inside client.shutdown() it is calling transport.close() which closes underlying outputstream. So, we should not do transport.close() ourselves in HiveMetaStoreClient.close() ? But, I can't be sure of this hypothesis since implementation of client.shutdown() is in FaceBookService.client (which is in libfb303.jar) I cant seem to find sources for this class anywhere. So, I request Facebook contributors of Hive project to either fix this bug or release sources of libfb303.jar so that I can verify and fix this myself.
          Hide
          Travis Crawford added a comment -

          @ashutosh - fb303 is released as a thrift contrib project:

          http://svn.apache.org/viewvc/thrift/trunk/contrib/fb303/

          Show
          Travis Crawford added a comment - @ashutosh - fb303 is released as a thrift contrib project: http://svn.apache.org/viewvc/thrift/trunk/contrib/fb303/
          Hide
          Ashutosh Chauhan added a comment -

          Thanks, Travis for pointing this out. I will take a look.

          Show
          Ashutosh Chauhan added a comment - Thanks, Travis for pointing this out. I will take a look.
          Hide
          Ashutosh Chauhan added a comment -

          Nopes, this location doesnt have it. It just has abstract class FaceBookBase.java I am looking for its implemetation that is FaceBookService.java Do you know where is that?

          Show
          Ashutosh Chauhan added a comment - Nopes, this location doesnt have it. It just has abstract class FaceBookBase.java I am looking for its implemetation that is FaceBookService.java Do you know where is that?
          Hide
          Phabricator added a comment -

          ashutoshc requested code review of "HIVE-2883 [jira] Metastore client doesnt close connection properly".
          Reviewers: JIRA

          https://issues.apache.org/jira/browse/HIVE-2883

          Here is the patch which gets rid of this problem as per my hypothesis. I also tested it and the above noted stack trace doesnt occur anymore.

          While closing connection, it always fail with following trace. Seemingly, it doesnt have any harmful effects.

          12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore client
          org.apache.thrift.transport.TTransportException: Cannot write to null outputStream
          at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
          at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
          at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
          at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
          at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421)
          at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
          at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D2613

          AFFECTED FILES
          metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/5985/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - ashutoshc requested code review of " HIVE-2883 [jira] Metastore client doesnt close connection properly". Reviewers: JIRA https://issues.apache.org/jira/browse/HIVE-2883 Here is the patch which gets rid of this problem as per my hypothesis. I also tested it and the above noted stack trace doesnt occur anymore. While closing connection, it always fail with following trace. Seemingly, it doesnt have any harmful effects. 12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore client org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421) at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310) TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2613 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/5985/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Travis Crawford added a comment -

          I patched this in and ran a pig query through HCatalog, and the "Unable to shutdown local metastore client" error went away, and the query produced a correct result. LGTM.

          Show
          Travis Crawford added a comment - I patched this in and ran a pig query through HCatalog, and the "Unable to shutdown local metastore client" error went away, and the query produced a correct result. LGTM.
          Hide
          Mithun Radhakrishnan added a comment -

          +1

          Good fix, this. Takes care of HCATALOG-236 as well.

          Show
          Mithun Radhakrishnan added a comment - +1 Good fix, this. Takes care of HCATALOG-236 as well.
          Hide
          Ashutosh Chauhan added a comment -

          Hive Committers,

          This has been independently verified by couple other folks. Can someone review it for me?

          All tests passed with the patch.

          Show
          Ashutosh Chauhan added a comment - Hive Committers, This has been independently verified by couple other folks. Can someone review it for me? All tests passed with the patch.
          Hide
          Ashutosh Chauhan added a comment -

          Unlinking from 0.9.0 release

          Show
          Ashutosh Chauhan added a comment - Unlinking from 0.9.0 release
          Hide
          Phabricator added a comment -

          cwsteinbach has accepted the revision "HIVE-2883 [jira] Metastore client doesnt close connection properly".

          +1. Please commit.

          REVISION DETAIL
          https://reviews.facebook.net/D2613

          BRANCH
          svn

          Show
          Phabricator added a comment - cwsteinbach has accepted the revision " HIVE-2883 [jira] Metastore client doesnt close connection properly". +1. Please commit. REVISION DETAIL https://reviews.facebook.net/D2613 BRANCH svn
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk.

          Show
          Ashutosh Chauhan added a comment - Committed to trunk.
          Hide
          Phabricator added a comment -

          ashutoshc has committed the revision "HIVE-2883 [jira] Metastore client doesnt close connection properly".

          Change committed by hashutosh.

          REVISION DETAIL
          https://reviews.facebook.net/D2613

          COMMIT
          https://reviews.facebook.net/rHIVE1329461

          Show
          Phabricator added a comment - ashutoshc has committed the revision " HIVE-2883 [jira] Metastore client doesnt close connection properly". Change committed by hashutosh. REVISION DETAIL https://reviews.facebook.net/D2613 COMMIT https://reviews.facebook.net/rHIVE1329461
          Hide
          Thomas Weise added a comment -

          Reopening for backport to 0.9 branch.

          Show
          Thomas Weise added a comment - Reopening for backport to 0.9 branch.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1392 (See https://builds.apache.org/job/Hive-trunk-h0.21/1392/)
          HIVE-2883 [jira] Metastore client doesnt close connection properly

          Summary:
          https://issues.apache.org/jira/browse/HIVE-2883

          Here is the patch which gets rid of this problem as per my hypothesis. I also
          tested it and the above noted stack trace doesnt occur anymore.

          While closing connection, it always fail with following trace. Seemingly, it
          doesnt have any harmful effects.

          12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore
          client
          org.apache.thrift.transport.TTransportException: Cannot write to null
          outputStream
          at
          org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
          at
          org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
          at
          org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
          at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
          at
          com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421)
          at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
          at
          org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)

          Test Plan: EMPTY

          Reviewers: JIRA, cwsteinbach

          Reviewed By: cwsteinbach

          Differential Revision: https://reviews.facebook.net/D2613 (Revision 1329461)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329461
          Files :

          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1392 (See https://builds.apache.org/job/Hive-trunk-h0.21/1392/ ) HIVE-2883 [jira] Metastore client doesnt close connection properly Summary: https://issues.apache.org/jira/browse/HIVE-2883 Here is the patch which gets rid of this problem as per my hypothesis. I also tested it and the above noted stack trace doesnt occur anymore. While closing connection, it always fail with following trace. Seemingly, it doesnt have any harmful effects. 12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore client org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421) at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310) Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D2613 (Revision 1329461) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329461 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
          Hide
          Ashutosh Chauhan added a comment -

          Checked-in 0.9 branch.

          Show
          Ashutosh Chauhan added a comment - Checked-in 0.9 branch.
          Hide
          Ashutosh Chauhan added a comment -

          This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

          Show
          Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2883 [jira] Metastore client doesnt close connection properly

          Summary:
          https://issues.apache.org/jira/browse/HIVE-2883

          Here is the patch which gets rid of this problem as per my hypothesis. I also
          tested it and the above noted stack trace doesnt occur anymore.

          While closing connection, it always fail with following trace. Seemingly, it
          doesnt have any harmful effects.

          12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore
          client
          org.apache.thrift.transport.TTransportException: Cannot write to null
          outputStream
          at
          org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142)
          at
          org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163)
          at
          org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91)
          at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62)
          at
          com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421)
          at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415)
          at
          org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310)

          Test Plan: EMPTY

          Reviewers: JIRA, cwsteinbach

          Reviewed By: cwsteinbach

          Differential Revision: https://reviews.facebook.net/D2613 (Revision 1329461)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329461
          Files :

          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2883 [jira] Metastore client doesnt close connection properly Summary: https://issues.apache.org/jira/browse/HIVE-2883 Here is the patch which gets rid of this problem as per my hypothesis. I also tested it and the above noted stack trace doesnt occur anymore. While closing connection, it always fail with following trace. Seemingly, it doesnt have any harmful effects. 12/03/20 10:55:02 ERROR hive.metastore: Unable to shutdown local metastore client org.apache.thrift.transport.TTransportException: Cannot write to null outputStream at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:142) at org.apache.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:163) at org.apache.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:91) at org.apache.thrift.TServiceClient.sendBase(TServiceClient.java:62) at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:421) at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:415) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:310) Test Plan: EMPTY Reviewers: JIRA, cwsteinbach Reviewed By: cwsteinbach Differential Revision: https://reviews.facebook.net/D2613 (Revision 1329461) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329461 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java

            People

            • Assignee:
              Ashutosh Chauhan
              Reporter:
              Ashutosh Chauhan
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development