Description
I've been running tests on clusters with "lots" of regions, about 400, and I'm seeing weird contention in the client.
This one shows up a lot around the SoftValueSortedMap.
First I have this blocked thread on I'm not sure what:
"TestClient-12" prio=10 tid=0x00007fb268872000 nid=0x3add waiting for monitor entry [0x00007fb251416000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.util.SoftValueSortedMap.isEmpty(SoftValueSortedMap.java:210) - locked <0x00000000c1b70318> (a java.util.TreeMap) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:1263) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1103) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1036) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:993)
This client waits on it:
"TestClient-14" prio=10 tid=0x00007fb268876000 nid=0x3adf waiting for monitor entry [0x00007fb251214000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.util.SoftValueSortedMap.lowerValueByKey(SoftValueSortedMap.java:189) - waiting to lock <0x00000000c1b70318> (a java.util.TreeMap) - locked <0x00000000c1b82120> (a org.apache.hadoop.hbase.util.SoftValueSortedMap) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:1272) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1103) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1036) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:993)
As you can see it's also holding a lock, which I have 11 other clients waiting on:
"TestClient-13" prio=10 tid=0x00007fb268874000 nid=0x3ade waiting for monitor entry [0x00007fb251315000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hbase.util.SoftValueSortedMap.lowerValueByKey(SoftValueSortedMap.java:189) - waiting to lock <0x00000000c1b82120> (a org.apache.hadoop.hbase.util.SoftValueSortedMap) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getCachedLocation(HConnectionManager.java:1272) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1103) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1036) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:993)