Major Description
When calling toPandas, usually Query UI shows each plan node's metric and corresponding Stage ID and Task ID:
>>> df = spark.createDataFrame([(1, 10, 'abc'), (2, 20, 'def')], schema=['x', 'y', 'z']) >>> df.toPandas() x y z 0 1 10 abc 1 2 20 def
but if Arrow execution is enabled, it shows only plan nodes and the duration is not correct:
>>> spark.conf.set('spark.sql.execution.arrow.pyspark.enabled', True)
>>> df.toPandas()
x y z
0 1 10 abc
1 2 20 def
