Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30153

Extend data exchange options for vectorized UDF functions with vanilla Arrow serialization

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • 3.1.0
    • None
    • PySpark
    • None

    Description

      Spark has introduced vectorized UDF with pandas_udf and this provides considerable speed up by reducing the overhead due to serialization and deserialization, where applciable.
      The current implementation of pandas_udf uses Arrow for fast serialization and then Pandas Series (or Pandas DF) for processing.
      There are opportunities to improve UDF performance, in certain cases, by bypaasing the conversion to and from Pandas and using Arrow Tables, directly with the help of specialized libraries able to process Arrow Tables and Arrays.
      One such case is for scientific computing of high energy physics data, where processing of arrays of data is of key importance.
      A test case using such approach has shown an increase of performance of about 3x, compared to the equivalent processing with pandas_udf, for a UDF based on plain Arrow serialization using a custom-developed extension of pandas_udf.  Processing of Arrow data in the test case was done via the "awkward arrays" library (https://github.com/scikit-hep/awkward-array).

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lucacanali Luca Canali
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: