[SPARK-28102] Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 19/Jun/19 02:22
• Updated: 19/Jun/19 22:30
• Resolved: 19/Jun/19 22:26

Description

Spark's use of lz4-java ends up calling LZ4Factory.fastestInstance, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized.

Unfortunately, the LZ4 library does not cache the fact that the JNI library failed to load, so every call to LZ4Factory.fastestInstance ends up repeating this failed load. These initialization attempts are performed in a static synchronized method, so exceptions from failures are thrown while holding a global lock and this causes lock-contention performance issues.

To avoid this problem, we can cache the LZ4Factory ourselves (since it's thread-safe), thereby caching the outcome of the failed load and avoiding performance problems.

Issue Links

• links to: GitHub Pull Request #24905