[MAPREDUCE-6987] JHS Log Scanner and Cleaner blocked

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.9.0, 3.0.0-alpha1
• Component/s: jobhistoryserver
• Target Version/s: 2.9.0, 3.0.0, 3.1.0
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 19/Oct/17 10:26
• Updated: 31/Oct/17 19:47
• Resolved: 31/Oct/17 19:47

Description

"Log Scanner/Cleaner #1" #81 prio=5 os_prio=0 tid=0x00007fd6c010f000 nid=0x11db waiting on condition [0x00007fd6aa859000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d6c88a80> (a java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
	at org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

"Log Scanner/Cleaner #0" #80 prio=5 os_prio=0 tid=0x00007fd6c010c800 nid=0x11da waiting on condition [0x00007fd6aa95a000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d6c80000> (a java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:47)
	at org.apache.hadoop.util.concurrent.HadoopScheduledThreadPoolExecutor.afterExecute(HadoopScheduledThreadPoolExecutor.java:69)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Both threads waiting on FutureTask.get() for infinite time after first execution

Issue Links

• is blocked by: HADOOP-14966 Handle JDK-8071638 for hadoop-common
• is broken by: MAPREDUCE-6634 Log uncaught exceptions/errors in various thread pools in mapreduce