Details
Major Description
SQLConf object has a sqlConfEntries map, which has a global lock (since it implements a Collections.synchronizedMap).
Every operation (like get or set) blocks the full object.
Concurrent threads may wait on lock.
An example is the DatatType.sameType method, that queries SQLConf entries map:
if (SQLConf.get.caseSensitiveAnalysis)
...
If this data type check is run in a custom piece of code on an executor with multiple cores (eg: 40), then a lot of time will be lost waiting on the lock.
An easy fix is to use the a ConcurrentHashMap that does not lock on read SQLConf.get): " ... retrieval operations do not entail locking ..."
NOTE: originally discovered by Benson Hon <benson.h@taboola.com> at Taboola while debugging strange slowdowns in production code
