Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28102

Failed LZ4 JNI initialization is repeatedly re-attempted, causing lock contention issues

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Spark's use of lz4-java ends up calling LZ4Factory.fastestInstance, which attempts to load JNI libraries and falls back on Java implementations in case the JNI library cannot be loaded or initialized.

      Unfortunately, the LZ4 library does not cache the fact that the JNI library failed to load, so every call to LZ4Factory.fastestInstance ends up repeating this failed load. These initialization attempts are performed in a static synchronized method, so exceptions from failures are thrown while holding a global lock and this causes lock-contention performance issues.

      To avoid this problem, we can cache the LZ4Factory ourselves (since it's thread-safe), thereby caching the outcome of the failed load and avoiding performance problems.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joshrosen Josh Rosen
                Reporter:
                joshrosen Josh Rosen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: