[SPARK-24421] Accessing sun.misc.Cleaner in JDK11 - ASF JIRA

XML

Word

Printable

JSON

Docs Text:

Hide
Provisional release notes text:

Spark 3 attempts to avoid the JVM's default limit on total size of memory allocated by direct buffers, for user convenience, by accessing some internal JDK classes directly. This is no longer possible in Java 9 and later, because of the new module encapsulation system.

For many usages of Spark, this will not matter, as the default {{MaxDirectMemorySize}} may be more than sufficient for all direct buffer allocation.

If it isn't, it can be made to work again by allowing the access explicitly with the JVM argument. {{--add-opens java.base/java.lang=ALL-UNNAMED}}. Of course this can also be resolved by explicitly setting {{-XX:MaxDirectMemorySize=}} to a sufficiently large value.

Show
Provisional release notes text: Spark 3 attempts to avoid the JVM's default limit on total size of memory allocated by direct buffers, for user convenience, by accessing some internal JDK classes directly. This is no longer possible in Java 9 and later, because of the new module encapsulation system. For many usages of Spark, this will not matter, as the default {{MaxDirectMemorySize}} may be more than sufficient for all direct buffer allocation. If it isn't, it can be made to work again by allowing the access explicitly with the JVM argument. {{--add-opens java.base/java.lang=ALL-UNNAMED}}. Of course this can also be resolved by explicitly setting {{-XX:MaxDirectMemorySize=}} to a sufficiently large value.

Many internal APIs such as unsafe are encapsulated in JDK9+, see http://openjdk.java.net/jeps/260 for detail.

To use Unsafe, we need to add jdk.unsupported to our code’s module declaration:

module java9unsafe {
    requires jdk.unsupported;
}

is duplicated by

SPARK-27587 No such method error (sun.nio.ch.DirectBuffer.cleaner()) when reading big table from JDBC (with one slow query)

links to

[Github] Pull Request #22993 (srowen)