Hive
  1. Hive
  2. HIVE-2937

TestHiveServerSessions hangs when executed directly

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Trivial Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Metastore
    • Labels:
      None

      Description

      ant test -Doffline=true -Dtestcase=TestHiveServerSessions
      

      Hangs infinitely.

      I couldn't imagine exact cause of the problem, but found that by adding 'new HiveServer.HiveServerHandler();' in setup(), test resulted to success.

        Issue Links

          Activity

          Hide
          Ashutosh Chauhan added a comment -

          @Navis,

          I have also found this test hanging on my machines. Looking at apache build logs, either of HIVE-2929 or HIVE-2858 is the cause. https://builds.apache.org/job/Hive-trunk-h0.21/1360/ Though atleast one of the subsequent build has passed. So, may be there is a race condition somewhere.

          Nsmit was involved in both of those patches, so may have more context.

          Show
          Ashutosh Chauhan added a comment - @Navis, I have also found this test hanging on my machines. Looking at apache build logs, either of HIVE-2929 or HIVE-2858 is the cause. https://builds.apache.org/job/Hive-trunk-h0.21/1360/ Though atleast one of the subsequent build has passed. So, may be there is a race condition somewhere. Nsmit was involved in both of those patches, so may have more context.
          Hide
          Phabricator added a comment -

          navis requested code review of "HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly".
          Reviewers: JIRA

          DPAL-1079 TestHiveServerSessions hangs when executed directly

          ant test -Doffline=true -Dtestcase=TestHiveServerSessions

          Hangs infinitely.

          I couldn't imagine exact cause of the problem, but found that by adding 'new HiveServer.HiveServerHandler();' in setup(), test resulted to success.

          TEST PLAN
          EMPTY

          REVISION DETAIL
          https://reviews.facebook.net/D2697

          AFFECTED FILES
          metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java

          MANAGE HERALD DIFFERENTIAL RULES
          https://reviews.facebook.net/herald/view/differential/

          WHY DID I GET THIS EMAIL?
          https://reviews.facebook.net/herald/transcript/6183/

          Tip: use the X-Herald-Rules header to filter Herald messages in your client.

          Show
          Phabricator added a comment - navis requested code review of " HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly". Reviewers: JIRA DPAL-1079 TestHiveServerSessions hangs when executed directly ant test -Doffline=true -Dtestcase=TestHiveServerSessions Hangs infinitely. I couldn't imagine exact cause of the problem, but found that by adding 'new HiveServer.HiveServerHandler();' in setup(), test resulted to success. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D2697 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/6183/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
          Hide
          Navis added a comment -

          @Ashutosh Chauhan

          It was related to initialization of metastore. Just added static sync in HMSHandler#createDefaultDB and seemed enough to solve this.

          Show
          Navis added a comment - @Ashutosh Chauhan It was related to initialization of metastore. Just added static sync in HMSHandler#createDefaultDB and seemed enough to solve this.
          Hide
          Ashutosh Chauhan added a comment -

          @Navis

          Can you add Namit in reviewer list?

          Show
          Ashutosh Chauhan added a comment - @Navis Can you add Namit in reviewer list?
          Hide
          Kevin Wilfong added a comment -

          I ran into this issue too. It seems to be caused by creating the HiveClient's too quickly. The initialization of the HiveClient initializes the HiveServerHandler which initializes the HMSHandler. The initializations of the HMSHandler's happen in such quick succession that when the second call to create the default db occurs, the first call hasn't finished creating the db yet so it attempts to create the same db and never gets out of this method (possibly a derby issue).

          It could also be fixed by adding a Thread.sleep between creating HiveClient's, but Navis's solution seems much more appropriate.

          @Ashutosh I don't think it is related to either of those patches as previous builds appear to fail for the same reason
          https://builds.apache.org/job/Hive-trunk-h0.21/1355/ also both of those patches should have only affected map reduce jobs, not the metastore.

          Show
          Kevin Wilfong added a comment - I ran into this issue too. It seems to be caused by creating the HiveClient's too quickly. The initialization of the HiveClient initializes the HiveServerHandler which initializes the HMSHandler. The initializations of the HMSHandler's happen in such quick succession that when the second call to create the default db occurs, the first call hasn't finished creating the db yet so it attempts to create the same db and never gets out of this method (possibly a derby issue). It could also be fixed by adding a Thread.sleep between creating HiveClient's, but Navis's solution seems much more appropriate. @Ashutosh I don't think it is related to either of those patches as previous builds appear to fail for the same reason https://builds.apache.org/job/Hive-trunk-h0.21/1355/ also both of those patches should have only affected map reduce jobs, not the metastore.
          Hide
          Phabricator added a comment -

          kevinwilfong has accepted the revision "HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly".

          +1 running tests

          REVISION DETAIL
          https://reviews.facebook.net/D2697

          BRANCH
          DPAL-1079

          Show
          Phabricator added a comment - kevinwilfong has accepted the revision " HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly". +1 running tests REVISION DETAIL https://reviews.facebook.net/D2697 BRANCH DPAL-1079
          Hide
          Ashutosh Chauhan added a comment -

          @Kevin,

          What you said make sense. But, I am still puzzled why this has started to show up recently. I see this hang ~70% of time on my builds starting last week. If the bug is as described, this should have always been the case. Isn't it?

          Show
          Ashutosh Chauhan added a comment - @Kevin, What you said make sense. But, I am still puzzled why this has started to show up recently. I see this hang ~70% of time on my builds starting last week. If the bug is as described, this should have always been the case. Isn't it?
          Hide
          Kevin Wilfong added a comment -

          @Ashutosh

          I dug deeper into the issue. I verified that the second HiveServerHandler does exit the method, just via an exception rather than via the return. This prevents the HiveServerHandler from being created. The test actually gets stuck making the first execute call on the second HiveClient, presumably because there is no handler on the other end of the connection to handle the request, so it gets stuck waiting for a response.

          I admit I am equally confused as to why this started showing up so frequently recently. I had seen this problem a few times before, I had assumed it was caused by running the tests twice on the same machine. Now however, I've run into this problem every time I try to run tests, since yesterday.

          As far as I can tell, there is no, and never was any logic to handle this problem. So Navis's diff seems worth committing to fix that issue, and it is at least a plus that it gets us around this issue, even if the root cause of why this issue suddenly became so prominent is still unsolved.

          I would like to commit this patch, unless you have any objections.

          Show
          Kevin Wilfong added a comment - @Ashutosh I dug deeper into the issue. I verified that the second HiveServerHandler does exit the method, just via an exception rather than via the return. This prevents the HiveServerHandler from being created. The test actually gets stuck making the first execute call on the second HiveClient, presumably because there is no handler on the other end of the connection to handle the request, so it gets stuck waiting for a response. I admit I am equally confused as to why this started showing up so frequently recently. I had seen this problem a few times before, I had assumed it was caused by running the tests twice on the same machine. Now however, I've run into this problem every time I try to run tests, since yesterday. As far as I can tell, there is no, and never was any logic to handle this problem. So Navis's diff seems worth committing to fix that issue, and it is at least a plus that it gets us around this issue, even if the root cause of why this issue suddenly became so prominent is still unsolved. I would like to commit this patch, unless you have any objections.
          Hide
          Ashutosh Chauhan added a comment -

          Go ahead and commit it will unblock us from this failing test. But do file a follow-up jira describing your findings so in case anyone is interested to further pursue this will have some context to begin with.

          Show
          Ashutosh Chauhan added a comment - Go ahead and commit it will unblock us from this failing test. But do file a follow-up jira describing your findings so in case anyone is interested to further pursue this will have some context to begin with.
          Hide
          Phabricator added a comment -

          navis updated the revision "HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly".
          Reviewers: JIRA, njain, kevinwilfong

          1. When failed to create server handler, close transport and make a log
          2. Fix lint warning

          REVISION DETAIL
          https://reviews.facebook.net/D2697

          AFFECTED FILES
          metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          service/src/java/org/apache/hadoop/hive/service/HiveServer.java

          Show
          Phabricator added a comment - navis updated the revision " HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly". Reviewers: JIRA, njain, kevinwilfong 1. When failed to create server handler, close transport and make a log 2. Fix lint warning REVISION DETAIL https://reviews.facebook.net/D2697 AFFECTED FILES metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java service/src/java/org/apache/hadoop/hive/service/HiveServer.java
          Hide
          Navis added a comment -

          Modified to close connection when exception is thrown initializing HiveServerHandler.

          Show
          Navis added a comment - Modified to close connection when exception is thrown initializing HiveServerHandler.
          Hide
          Phabricator added a comment -

          kevinwilfong has commented on the revision "HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly".

          Awesome! I wasn't sure if Thrift supported this, at least with this we won't end up with these mysterious infinite hangs.

          REVISION DETAIL
          https://reviews.facebook.net/D2697

          BRANCH
          DPAL-1079

          Show
          Phabricator added a comment - kevinwilfong has commented on the revision " HIVE-2937 [jira] TestHiveServerSessions hangs when executed directly". Awesome! I wasn't sure if Thrift supported this, at least with this we won't end up with these mysterious infinite hangs. REVISION DETAIL https://reviews.facebook.net/D2697 BRANCH DPAL-1079
          Hide
          Kevin Wilfong added a comment -

          Committed, thanks Navis.

          Show
          Kevin Wilfong added a comment - Committed, thanks Navis.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1373 (See https://builds.apache.org/job/Hive-trunk-h0.21/1373/)
          HIVE-2937. TestHiveServerSessions hangs when executed directly. (navis via kevinwilfong) (Revision 1325830)

          Result = SUCCESS
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325830
          Files :

          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1373 (See https://builds.apache.org/job/Hive-trunk-h0.21/1373/ ) HIVE-2937 . TestHiveServerSessions hangs when executed directly. (navis via kevinwilfong) (Revision 1325830) Result = SUCCESS kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325830 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.

          Show
          Ashutosh Chauhan added a comment - This issue is closed now. It was released with the fix in 0.9.0. If there is a problem, please open a new jira and link this one with that.
          Hide
          Navis added a comment -

          This is not applied to hive-0.9.0 and not consistent with release note.

          Show
          Navis added a comment - This is not applied to hive-0.9.0 and not consistent with release note.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-2937. TestHiveServerSessions hangs when executed directly. (navis via kevinwilfong) (Revision 1325830)

          Result = ABORTED
          kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325830
          Files :

          • /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
          • /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-2937 . TestHiveServerSessions hangs when executed directly. (navis via kevinwilfong) (Revision 1325830) Result = ABORTED kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325830 Files : /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java

            People

            • Assignee:
              Navis
              Reporter:
              Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development