Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20871

Only log Janino code in debug mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently if Janino code compilation fails, it will log the entirety of the code in the executors. Because the generated code can often be very large, the logging can cause heap pressure on the driver and cause it to fall over.

      I propose removing the "$formatted" from here: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L964

        Attachments

          Activity

            People

            • Assignee:
              pj.fanning PJ Fanning
              Reporter:
              gtakahashi@palantir.com Glen Takahashi
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: