Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21941

Stop storing unused attemptId in SQLTaskMetrics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Web UI
    • Labels:
      None

      Description

      Currently SQLTaskMetrics has a long attemptId field on it that is unused, with a TODO saying to populate the value in the future. We should save this memory by leaving the TODO but taking the unused field out of the class.

      I have a driver that heap dumped on OOM and has 390,105 instances of SQLTaskMetric – removing this 8 bytes field will save roughly 390k*8 = 3.1MB of heap space. It's not going to fix my OOM, but there's no reason to put this pressure on the GC if we don't get anything by storing it.

      https://github.com/apache/spark/blob/v2.2.0/sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala#L485

        Attachments

          Activity

            People

            • Assignee:
              aash Andrew Ash
              Reporter:
              aash Andrew Ash
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: