Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • Documentation, PySpark
    • None

    Description

      1. Python Profiler: https://spark.apache.org/docs/2.3.0/api/python/_modules/pyspark/profiler.html,

      >>> sc._conf.set("spark.python.profile", "true")
      >>> rdd = sc.parallelize(range(100)).map(str)
      >>> rdd.count()
      100
      >>> sc.show_profiles()
      

      2. Python Debugger, e.g.) https://www.jetbrains.com/help/pycharm/remote-debugging-with-product.html#remote-interpreter (?)

      3. Monitoring Python Workers: top, `ls`, etc. (?)

      4. PyCharm setup (?) (at https://issues.apache.org/jira/browse/SPARK-32189)

      Attachments

        Issue Links

          Activity

            People

              hyukjin.kwon Hyukjin Kwon
              hyukjin.kwon Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: