[HIVE-15936] ConcurrentModificationException in ATSHook

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Target Version/s: 2.2.0
• Hadoop Flags: Reviewed
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 16/Feb/17 02:11
• Updated: 25/Jul/17 20:05
• Resolved: 17/Feb/17 08:23

Description

See ATSHook error:

java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_112]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_112]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_112]
	at java.util.AbstractCollection.toArray(AbstractCollection.java:196) ~[?:1.8.0_112]
	at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:290) ~[guava-14.0.1.jar:?]
	at org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:219) ~[hive-common-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
	at org.apache.hadoop.hive.ql.hooks.ATSHook.createPostHookEvent(ATSHook.java:347) ~[hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
	at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:206) [hive-exec-2.1.0.2.6.0.0-457.jar:2.1.0.2.6.0.0-457]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]

According to jdere, ATSHook is currently accessing the PerfLogger on a separate thread, which means the main query thread can potentially write to the PerfLogger at the same time.
The ATSHook should access the PerfLogger on the main query thread, before it sends the execution to the ATS Logger thread.