Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36733

Perf issue in SchemaPruning when a struct has many fields

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.2
    • 3.3.0
    • SQL
    • None

    Description

      Seeing a significant performance degradation in query processing when a table contains a significantly large number of fields (>10K).

      Here's the stacktraces while processing a query

         java.lang.Thread.State: RUNNABLE   java.lang.Thread.State: RUNNABLE at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:285) at scala.collection.TraversableLike$$Lambda$296/874023329.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.map(TraversableLike.scala:285) at scala.collection.TraversableLike.map$(TraversableLike.scala:278) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:198) at org.apache.spark.sql.types.StructType.fieldNames(StructType.scala:108) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$1(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$1$adapted(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$$$Lambda$3963/249742655.apply(Unknown Source) at scala.collection.TraversableLike.$anonfun$filterImpl$1(TraversableLike.scala:303) at scala.collection.TraversableLike$$Lambda$403/465534593.apply(Unknown Source) at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36) at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198) at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:302) at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:296) at scala.collection.mutable.ArrayOps$ofRef.filterImpl(ArrayOps.scala:198) at scala.collection.TraversableLike.filter(TraversableLike.scala:394) at scala.collection.TraversableLike.filter$(TraversableLike.scala:394) at scala.collection.mutable.ArrayOps$ofRef.filter(ArrayOps.scala:198) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.sortLeftFieldsByRight(SchemaPruning.scala:70) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$.$anonfun$sortLeftFieldsByRight$3(SchemaPruning.scala:75) at org.apache.spark.sql.catalyst.expressions.SchemaPruning$$$Lambda$3965/461314749.apply(Unknown Source) 

      Attachments

        Activity

          People

            sarutak Kousuke Saruta
            taroplus Kohki Nishio
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: