Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-9642

AM ZK Workers stuck doing 100% CPU on HashMap.put

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.96.0
    • Fix Version/s: 0.98.0, 0.96.0
    • Component/s: None
    • Labels:
      None

      Description

      I just noticed on my test cluster that my master is using all my CPUs even though it's completely idle. 5 threads are doing this:

      "AM.ZK.Worker-pool2-t34" daemon prio=10 tid=0x00007f68ac176800 nid=0x5251 runnable [0x00007f688cc83000]
         java.lang.Thread.State: RUNNABLE
      	at java.util.HashMap.put(HashMap.java:374)
      	at org.apache.hadoop.hbase.master.AssignmentManager.handleRegion(AssignmentManager.java:954)
      	at org.apache.hadoop.hbase.master.AssignmentManager$6.run(AssignmentManager.java:1419)
      	at org.apache.hadoop.hbase.master.AssignmentManager$3.run(AssignmentManager.java:1247)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
      	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
      

      Looking at the code, I see HBASE-9095 introduced two HashMaps for tests only but they end up being used concurrently in the AM and are never cleaned up. It seems to me that any master running since that patch was committed has a time bomb in it.

      I'm marking this as a blocker. Devaraj Das and Jimmy Xiang, you guys wanna take a look at this?

        Attachments

        1. 9642-1.txt
          5 kB
          Devaraj Das
        2. 9642-2.txt
          5 kB
          Devaraj Das

          Activity

            People

            • Assignee:
              devaraj Devaraj Das
              Reporter:
              jdcryans Jean-Daniel Cryans
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: