[HBASE-27211] Data race in MonitoredTaskImpl could cause split wal failure

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.5.0, 3.0.0-alpha-4, 2.4.14
• Component/s: monitoring, wal
• Hadoop Flags: Reviewed
• Assignee: Duo Zhang
• Reporter: Duo Zhang
• Created: 16/Jul/22 04:11
• Updated: 19/Jul/22 19:50
• Resolved: 18/Jul/22 12:09

Description

https://ci-hbase.apache.org/job/HBase-Flaky-Tests/job/master/3866/testReport/junit/org.apache.hadoop.hbase.wal/TestWALSplitBoundedLogWriterCreation/testThreading/

java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy49.prettyPrintJournal(Unknown Source)
	at org.apache.hadoop.hbase.wal.WALSplitter.splitWAL(WALSplitter.java:427)
	at org.apache.hadoop.hbase.wal.TestWALSplit.doTestThreading(TestWALSplit.java:1035)
	at org.apache.hadoop.hbase.wal.TestWALSplit.testThreading(TestWALSplit.java:939)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:39)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hbase.monitoring.TaskMonitor$PassthroughInvocationHandler.invoke(TaskMonitor.java:291)
	... 32 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hbase.monitoring.MonitoredTaskImpl.prettyPrintJournal(MonitoredTaskImpl.java:278)
	... 36 more

In general, we do not set any null value in the code but the related data structutre are not thread safe and we use it in multi thread environment when splitting wal, so let's make it thread safe first.

Issue Links

• links to: GitHub Pull Request #4630