[HBASE-23801] ExecutorService ignores Throwables

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: regionserver
• Assignee: Unassigned
• Reporter: Nick Dimiduk
• Created: 05/Feb/20 18:53
• Updated: 05/Feb/20 18:53

Description

From the test logs of a failed build, I find that the ExecutorService running in the region server appears to ignore exceptions thrown in its executor's tasks. In this case, a fatal RuntimeException was thrown. Instead of aborting the RS, we merely log. This results in the region open operation never completing, and the test simply hangs.

This implementation is ancient, dating back to HBASE-4281 or before, circa 2011. Thus I assume it impacts all active release lines.

2020-02-05 07:45:53,361 FATAL [RS_OPEN_REGION-regionserver/e590165b0ef0:0-1] conf.Configuration(2859): error parsing conf hbase-site.xml
org.xml.sax.SAXParseException; systemId: file:/home/jenkins/jenkins-slave/workspace/HBase_Nightly_branch-2@2/component/hbase-mapreduce/target/test-classes/hbase-site.xml; lineNumber: 1; columnNumber: 1; Premature end of file.
	at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
	at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
	at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2684)
	at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2672)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2746)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2706)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1550)
	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:70)
	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:84)
	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:98)
	at org.apache.hadoop.hbase.util.LossyCounting.<init>(LossyCounting.java:84)
	at org.apache.hadoop.hbase.regionserver.MetricsUserAggregateImpl.<init>(MetricsUserAggregateImpl.java:45)
	at org.apache.hadoop.hbase.regionserver.MetricsUserAggregateFactory.getMetricsUserAggregate(MetricsUserAggregateFactory.java:35)
	at org.apache.hadoop.hbase.regionserver.MetricsRegion.<init>(MetricsRegion.java:40)
	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:849)
	at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:730)
	at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.hadoop.hbase.regionserver.HRegion.newHRegion(HRegion.java:7135)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegionFromTableDir(HRegion.java:7352)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7329)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7287)
	at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:7238)
	at org.apache.hadoop.hbase.regionserver.handler.AssignRegionHandler.process(AssignRegionHandler.java:133)
	at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)