Hive
  1. Hive
  2. HIVE-3723

Hive Driver leaks ZooKeeper connections

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      In certain error cases (i.e.: statement fails to compile, semantic errors) the hive driver leaks zookeeper connections.

      This can be seen in the TestNegativeCliDriver test which accumulates a large number of open file handles and fails if the max allowed number of file handles isn't at least 2048.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          5m 36s 1 Gunther Hagleitner 19/Nov/12 22:34
          Patch Available Patch Available Open Open
          3d 6h 45m 1 Ashutosh Chauhan 23/Nov/12 05:19
          Open Open Resolved Resolved
          4d 12h 15m 1 Ashutosh Chauhan 27/Nov/12 17:35
          Resolved Resolved Closed Closed
          44d 2h 17m 1 Ashutosh Chauhan 10/Jan/13 19:53
          Harel Ben Attia made changes -
          Link This issue is blocked by HIVE-5853 [ HIVE-5853 ]
          Ashutosh Chauhan made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Ashutosh Chauhan added a comment -

          This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

          Show
          Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
          HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278)

          Result = ABORTED
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1821 (See https://builds.apache.org/job/Hive-trunk-h0.21/1821/)
          HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278)

          Result = FAILURE
          hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278
          Files :

          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1821 (See https://builds.apache.org/job/Hive-trunk-h0.21/1821/ ) HIVE-3723 : Hive Driver leaks ZooKeeper connections (Gunther Hagleitner via Ashutosh Chauhan) (Revision 1414278) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1414278 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
          Ashutosh Chauhan made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Fix Version/s 0.10.0 [ 12320745 ]
          Resolution Fixed [ 1 ]
          Hide
          Ashutosh Chauhan added a comment -

          Committed to trunk and 0.10. Thanks, Gunther!

          Show
          Ashutosh Chauhan added a comment - Committed to trunk and 0.10. Thanks, Gunther!
          Hide
          Ashutosh Chauhan added a comment -

          Hmm.. make sense. +1 will commit if tests pass.

          Show
          Ashutosh Chauhan added a comment - Hmm.. make sense. +1 will commit if tests pass.
          Hide
          Gunther Hagleitner added a comment -

          The releaseLocks call happens at various places in the run method. I think that's fine too, since calling destroy would kill the HiveLockManager which is shared between different context objects (i.e.: That should only happen when someone specifically destroys the driver.)

          As a side note, compile doesn't seem to acquire any locks - the call is thus just a protection against future versions that could.

          Show
          Gunther Hagleitner added a comment - The releaseLocks call happens at various places in the run method. I think that's fine too, since calling destroy would kill the HiveLockManager which is shared between different context objects (i.e.: That should only happen when someone specifically destroys the driver.) As a side note, compile doesn't seem to acquire any locks - the call is thus just a protection against future versions that could.
          Ashutosh Chauhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Ashutosh Chauhan added a comment -

          Around line 910 if compile returns non-zero value, then also we releaseLocks(List<Lock>) and then return. There also, we should call destroy() instead of releaseLocks.

          Show
          Ashutosh Chauhan added a comment - Around line 910 if compile returns non-zero value, then also we releaseLocks(List<Lock>) and then return. There also, we should call destroy() instead of releaseLocks.
          Gunther Hagleitner made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Gunther Hagleitner added a comment -

          The patch makes sure that calling "destroy" will always close the lock manager (and thus release the zookeeper connection.)

          Show
          Gunther Hagleitner added a comment - The patch makes sure that calling "destroy" will always close the lock manager (and thus release the zookeeper connection.)
          Gunther Hagleitner made changes -
          Field Original Value New Value
          Attachment HIVE-3723.1-r1411423.patch [ 12554248 ]
          Gunther Hagleitner created issue -

            People

            • Assignee:
              Gunther Hagleitner
              Reporter:
              Gunther Hagleitner
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development