Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22024 [pySpark] Speeding up internal conversion for Spark SQL
  3. SPARK-22032

Speed up StructType.fromInternal

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • PySpark
    • None

    Description

      StructType.fromInternal is calling f.fromInternal(v) for every field.
      We can use needConversion method to limit the number of function calls.

      Attachments

        Activity

          People

            maver1ck Maciej Bryński
            maver1ck Maciej Bryński
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: