[SPARK-22043] Python profile, show_profiles() and dump_profiles(), should throw an error with a better message

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.1.2, 2.2.1, 2.3.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 17/Sep/17 11:23
• Updated: 12/Dec/22 18:11
• Resolved: 18/Sep/17 04:21

Description

I mistakenly missed spark.python.profile enabled today while profiling and met this unfriendly messages:

>>> sc.show_profiles()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark/python/pyspark/context.py", line 1000, in show_profiles
    self.profiler_collector.show_profiles()
AttributeError: 'NoneType' object has no attribute 'show_profiles'
>>> sc.dump_profiles("/tmp/abc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles
    self.profiler_collector.dump_profiles(path)
AttributeError: 'NoneType' object has no attribute 'dump_profiles'

It looks we should give better information that says spark.python.profile should be enabled.

Issue Links

• links to: [Github] Pull Request #19260 (HyukjinKwon)