XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: Build
    • Labels:
    • Docs Text:
      Hide
      Provisional release notes text:

      Spark 3 attempts to avoid the JVM's default limit on total size of memory allocated by direct buffers, for user convenience, by accessing some internal JDK classes directly. This is no longer possible in Java 9 and later, because of the new module encapsulation system.

      For many usages of Spark, this will not matter, as the default {{MaxDirectMemorySize}} may be more than sufficient for all direct buffer allocation.

      If it isn't, it can be made to work again by allowing the access explicitly with the JVM argument. {{--add-opens java.base/java.lang=ALL-UNNAMED}}. Of course this can also be resolved by explicitly setting {{-XX:MaxDirectMemorySize=}} to a sufficiently large value.
      Show
      Provisional release notes text: Spark 3 attempts to avoid the JVM's default limit on total size of memory allocated by direct buffers, for user convenience, by accessing some internal JDK classes directly. This is no longer possible in Java 9 and later, because of the new module encapsulation system. For many usages of Spark, this will not matter, as the default {{MaxDirectMemorySize}} may be more than sufficient for all direct buffer allocation. If it isn't, it can be made to work again by allowing the access explicitly with the JVM argument. {{--add-opens java.base/java.lang=ALL-UNNAMED}}. Of course this can also be resolved by explicitly setting {{-XX:MaxDirectMemorySize=}} to a sufficiently large value.

      Description

      Many internal APIs such as unsafe are encapsulated in JDK9+, see http://openjdk.java.net/jeps/260 for detail.

      To use Unsafe, we need to add jdk.unsupported to our code’s module declaration:

      module java9unsafe {
          requires jdk.unsupported;
      }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                srowen Sean R. Owen
                Reporter:
                dbtsai DB Tsai
              • Votes:
                4 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: