Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25494

Upgrade Spark's use of Janino to 3.0.10

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 3.0.0
    • SQL
    • None

    Description

      This ticket proposes to upgrade Spark's use of Janino from 3.0.9 to 3.0.10.
      Note that 3.0.10 is a out-of-band release specifically for fixing an integer overflow issue in Janino's ClassFile reader. It is otherwise exactly the same as 3.0.9, so it's a low risk and compatible upgrade.

      The integer overflow issue affects Spark SQL's codegen stats collection: when a generated Class file is huge, especially when the constant pool size is above Short.MAX_VALUE, Janino's ClassFile reader will throw an exception when Spark wants to parse the generated Class file to collect stats. So we'll miss the stats of some huge Class files.

      The Janino fix is tracked by this issue: https://github.com/janino-compiler/janino/issues/58

      Attachments

        Activity

          People

            rednaxelafx Kris Mok
            rednaxelafx Kris Mok
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: