Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39545

Override `concat` method for `ExpressionSet` in Scala 2.13 to improve the performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      ExpressionSet ++ method in the master branch a little slower than the branch-3.3 with Scala-2.13

       

      For example, write a microbenchmark as follows and run with Scala 2.13:

      val valuesPerIteration = 100000
          val benchmark = new Benchmark("Test ExpressionSet ++ ", valuesPerIteration, output = output)
          val aUpper = AttributeReference("A", IntegerType)(exprId = ExprId(1))
          val initialSet = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
          val setToAddWithSameDeterministicExpression = ExpressionSet(aUpper + 1 :: Rand(0) :: Nil)
      
          benchmark.addCase("Test ++") { _: Int =>
            for (_ <- 0L until valuesPerIteration) {
              initialSet ++ setToAddWithSameDeterministicExpression
            }
          }
      
          benchmark.run() 

      branch-3.3 result:
       

      OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
      Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
      Test ExpressionSet ++ :                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
      ------------------------------------------------------------------------------------------------------------------------
      Test ++                                              14             16           4          7.2         139.1       1.0X
       

       
      master result :

      OpenJDK 64-Bit Server VM 1.8.0_312-b07 on Linux 4.14.0_1-0-0-45
      Intel(R) Xeon(R) Gold 6XXXC CPU @ 2.60GHz
      Test ExpressionSet ++ :                   Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
      ------------------------------------------------------------------------------------------------------------------------
      Test ++                                              16             19           5          6.1         163.9       1.0X
       

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: