Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-988

Write PySpark profiling guide

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Documentation, PySpark
    • Labels:
      None

      Description

      Write a guide on profiling PySpark applications. I've done this in the past by modifying the workers to make cProfile dumps, then using various tools to collect and merge those dumps into an overall performance profile.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joshrosen Josh Rosen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: