In Scala APIs, we have added Arrow integration and developer APIs in Apache Spark, for example, at ArrowConverters but this is again inconvenient to use out of the box.
In PySpark, in order to use Arrow format to connect to other external systems, they should manually convert pandas DataFrame in pandas UDF to the Arrow batch, which is inconvenient.
pandas UDFs are used in other use cases too. For example, they convert it back to
Arrow batch, and integrate with other systems, see also https://github.com/apache/spark/pull/26783#issue-534127514.
However, it doesn’t vectorize operations because pandas doesn’t support nested structure
natively, and the performance impact seems non-trivial.
In addition, it requires virtually copying during the conversion between pandas and Arrow format that consumes computation (Spark internal format -> Arrow format -> pandas DataFrame -> Arrow format). See https://github.com/apache/spark/pull/26783#issue-534127514 for performance impact.
I would like to propose an API DataFrame.mapInArrow like DataFrame.mapInPandas, and RDD.mapPartitions.
The API shape would look like:
|Implement DataFrame.mapInArrow in Scala||Open||Unassigned|