Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21041

With whole-stage codegen, SparkSession.range()'s behavior is inconsistent with SparkContext.range()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.1, 2.2.0
    • Fix Version/s: 2.2.0, 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      When whole-stage codegen is enabled, in face of integer overflow, SparkSession.range()'s behavior is inconsistent with when codegen is turned off, while the latter is consistent with SparkContext.range()'s behavior.

      The following Spark Shell session shows the inconsistency:

      scala> sc.range
         def range(start: Long,end: Long,step: Long,numSlices: Int): org.apache.spark.rdd.RDD[Long]
      
      scala> spark.range
                                                                                                           
      def range(start: Long,end: Long,step: Long,numPartitions: Int): org.apache.spark.sql.Dataset[Long]   
      def range(start: Long,end: Long,step: Long): org.apache.spark.sql.Dataset[Long]                      
      def range(start: Long,end: Long): org.apache.spark.sql.Dataset[Long]                                 
      def range(end: Long): org.apache.spark.sql.Dataset[Long] 
      
      scala> sc.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
      res1: Array[Long] = Array()
      
      scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
      res2: Array[Long] = Array(9223372036854775804, 9223372036854775805, 9223372036854775806)
      
      scala> spark.conf.set("spark.sql.codegen.wholeStage", false)
      
      scala> spark.range(java.lang.Long.MAX_VALUE - 3, java.lang.Long.MIN_VALUE + 2, 1).collect
      res5: Array[Long] = Array()
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dongjoon Dongjoon Hyun
                Reporter:
                rednaxelafx Kris Mok
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: