Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-20871

Only log Janino code in debug mode

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.1.1
    • 2.3.0
    • SQL
    • None

    Description

      Currently if Janino code compilation fails, it will log the entirety of the code in the executors. Because the generated code can often be very large, the logging can cause heap pressure on the driver and cause it to fall over.

      I propose removing the "$formatted" from here: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala#L964

      Attachments

        Activity

          People

            pj.fanning PJ Fanning
            gtakahashi@palantir.com Glen Takahashi
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: