Hive
  1. Hive
  2. HIVE-3723

Hive Driver leaks ZooKeeper connections

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      In certain error cases (i.e.: statement fails to compile, semantic errors) the hive driver leaks zookeeper connections.

      This can be seen in the TestNegativeCliDriver test which accumulates a large number of open file handles and fails if the max allowed number of file handles isn't at least 2048.

        Issue Links

          Activity

          Hide
          Gunther Hagleitner added a comment -

          The patch makes sure that calling "destroy" will always close the lock manager (and thus release the zookeeper connection.)

          Show
          Gunther Hagleitner added a comment - The patch makes sure that calling "destroy" will always close the lock manager (and thus release the zookeeper connection.)
          Hide
          Ashutosh Chauhan added a comment -

          Around line 910 if compile returns non-zero value, then also we releaseLocks(List<Lock>) and then return. There also, we should call destroy() instead of releaseLocks.

          Show
          Ashutosh Chauhan added a comment - Around line 910 if compile returns non-zero value, then also we releaseLocks(List<Lock>) and then return. There also, we should call destroy() instead of releaseLocks.
          Hide
          Gunther Hagleitner added a comment -

          The releaseLocks call happens at various places in the run method. I think that's fine too, since calling destroy would kill the HiveLockManager which is shared between different context objects (i.e.: That should only happen when someone specifically destroys the driver.)

          As a side note, compile doesn't seem to acquire any locks - the call is thus just a protection against future versions that could.

          Show
          Gunther Hagleitner added a comment - The releaseLocks call happens at various places in the run method. I think that's fine too, since calling destroy would kill the HiveLockManager which is shared between different context objects (i.e.: That should only happen when someone specifically destroys the driver.) As a side note, compile doesn't seem to acquire any locks - the call is thus just a protection against future versions that could.
          Hide
          Ashutosh Chauhan added a comment -

          Hmm.. make sense. +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - Hmm.. make sense. +1 will commit if tests pass.
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk and 0.10. Thanks, Gunther!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk and 0.10. Thanks, Gunther!
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1821 (See https://builds.apache.org/job/Hive-trunk-h0.21/1821/)
          HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1821 (See https://builds.apache.org/job/Hive-trunk-h0.21/1821/ ) HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development