Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28590

Add sort_stats Setter for Custom Profiler

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: PySpark
    • Labels:
      None

      Description

      When I want to use BasicProfiler with different sorters in sort_stats, I sometimes need to create a custom profiler and implement the show() method only to replace the following line: stats.sort_stats("time", "cumulative").print_stats().

      I think it'd be better if the users are able to specify the sorters without creating a custom profiler.

      I implemented the changes in PySpark only.

      To apply the setter and getter methods, one can use the following way:

      conf = SparkConf().set("spark.python.profile", "true")
      
      # use BasicProfiler
      sc = SparkContext('local', 'test', conf=conf)
      sc.profiler_collector.profiler_cls.set_sort_stats_sorters(BasicProfiler, ['ncalls', 'tottime', 'name']
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                albertus-kelvin Albertus Kelvin
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: