Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12790

Metastore connection leaks in HiveServer2

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 2.0.0
    • Component/s: HiveServer2
    • Labels:
      None

      Description

      HiveServer2 keeps opening new connections to HMS each time it launches a task. These connections do not appear to be closed when the task completes thus causing a HMS connection leak. "lsof" for the HS2 process shows connections to port 9083.

      2015-12-03 04:20:56,352 INFO  [HiveServer2-Background-Pool: Thread-424756()]: ql.Driver (SessionState.java:printInfo(558)) - Launching Job 11 out of 41
      2015-12-03 04:20:56,354 INFO  [Thread-405728()]: hive.metastore (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI thrift://<anonymizedURL>:9083
      2015-12-03 04:20:56,360 INFO  [Thread-405728()]: hive.metastore (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, current connections: 14824
      2015-12-03 04:20:56,360 INFO  [Thread-405728()]: hive.metastore (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
      ....
      2015-12-03 04:21:06,355 INFO  [HiveServer2-Background-Pool: Thread-424756()]: ql.Driver (SessionState.java:printInfo(558)) - Launching Job 12 out of 41
      2015-12-03 04:21:06,357 INFO  [Thread-405756()]: hive.metastore (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI thrift://<anonymizedURL>:9083
      2015-12-03 04:21:06,362 INFO  [Thread-405756()]: hive.metastore (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, current connections: 14825
      2015-12-03 04:21:06,362 INFO  [Thread-405756()]: hive.metastore (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
      ...
      2015-12-03 04:21:08,357 INFO  [HiveServer2-Background-Pool: Thread-424756()]: ql.Driver (SessionState.java:printInfo(558)) - Launching Job 13 out of 41
      2015-12-03 04:21:08,360 INFO  [Thread-405782()]: hive.metastore (HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI thrift://<anonymizedURL>:9083
      2015-12-03 04:21:08,364 INFO  [Thread-405782()]: hive.metastore (HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, current connections: 14826
      2015-12-03 04:21:08,365 INFO  [Thread-405782()]: hive.metastore (HiveMetaStoreClient.java:open(400)) - Connected to metastore.
      ... 
      

      The TaskRunner thread starts a new SessionState each time, which creates a new connection to the HMS (via Hive.get(conf).getMSC()) that is never closed.

      Even SessionState.close(), currently not being called by the TaskRunner thread, does not close this connection.

      Attaching a anonymized log snippet where the number of HMS connections reaches north of 25000+ connections.

        Attachments

        1. HIVE-12790.2.patch
          2 kB
          Naveen Gangam
        2. HIVE-12790.3.patch
          2 kB
          Naveen Gangam
        3. HIVE-12790.patch
          2 kB
          Naveen Gangam
        4. snippedLog.txt
          6.01 MB
          Naveen Gangam

          Issue Links

            Activity

              People

              • Assignee:
                ngangam Naveen Gangam
                Reporter:
                ngangam Naveen Gangam
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: