Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21228

Memory leak since AbstractFSWAL caches Thread object and never clean later

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.1.0, 2.0.2, 1.4.7
    • Fix Version/s: 3.0.0, 1.3.3, 1.2.8, 2.2.0, 1.4.8, 2.1.1, 2.0.3
    • Component/s: wal
    • Labels:
      None

      Description

      In AbstractFSWAL(FSHLog in branch-1), we have a map caches thread and SyncFutures.

      /**
         * Map of {@link SyncFuture}s keyed by Handler objects. Used so we reuse SyncFutures.
         * <p>
         * TODO: Reuse FSWALEntry's rather than create them anew each time as we do SyncFutures here.
         * <p>
         * TODO: Add a FSWalEntry and SyncFuture as thread locals on handlers rather than have them get
         * them from this Map?
         */
        private final ConcurrentMap<Thread, SyncFuture> syncFuturesByHandler;
      

      A colleague of mine find a memory leak case caused by this map.

      Every thread who writes WAL will be cached in this map, And no one will clean the threads in the map even after the thread is dead.

      In one of our customer's cluster, we noticed that even though there is no requests, the heap of the RS is almost full and CMS GC was triggered every second.
      We dumped the heap and then found out there were more than 30 thousands threads with Terminated state. which are all cached in this map above. Everything referenced in these threads were leaked. Most of the threads are:
      1.PostOpenDeployTasksThread, which will write Open Region mark in WAL
      2. hconnection-0x1f838e31-shared--pool, which are used to write index short circuit(Phoenix), and WAL will be write and sync in these threads.
      3. Index writer thread(Phoenix), which referenced by RegionCoprocessorHost$RegionEnvironment then by HRegion and finally been referenced by PostOpenDeployTasksThread.

      We should turn this map into a thread local one, let JVM GC the terminated thread for us.

        Attachments

        1. HBASE-21228.branch-2.0.003.patch
          4 kB
          Allan Yang
        2. HBASE-21228.branch-2.0.002.patch
          4 kB
          Allan Yang
        3. HBASE-21228.branch-2.0.001.patch
          3 kB
          Allan Yang

          Issue Links

            Activity

              People

              • Assignee:
                allan163 Allan Yang
                Reporter:
                allan163 Allan Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: