Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24003

Add support to provide spark.executor.extraJavaOptions in terms of App Id and/or Executor Id's

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: Mesos, Spark Core, YARN
    • Labels:
      None

      Description

      Users may want to enable gc logging or heap dump for the executors, but there is a chance of overwriting it by other executors since the paths cannot be expressed dynamically. This improvement would enable to express the spark.executor.extraJavaOptions paths in terms of App Id and Executor Id's to avoid the overwriting by other executors.

      There was a discussion about this in SPARK-3767, but it never fixed.

        Attachments

          Activity

            People

            • Assignee:
              devaraj Devaraj Kavali
              Reporter:
              devaraj Devaraj Kavali
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: