Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19451

rangeBetween method should accept Long value as boundary

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.6.1, 2.0.2
    • 2.3.0
    • SQL
    • None

    Description

      Hi there,

      there seems to be a major limitation in spark window functions and rangeBetween method.

      If I have the following code :

      Exemple
          val tw =  Window.orderBy("date")
            .partitionBy("id")
            .rangeBetween( from , 0)
      

      Everything seems ok, while from value is not too large... Even if the rangeBetween() method supports Long parameters.
      But.... If i set -2160000000L value to from it does not work !

      It is probably related to this part of code in the between() method, of the WindowSpec class, called by rangeBetween()

      between() method
          val boundaryStart = start match {
            case 0 => CurrentRow
            case Long.MinValue => UnboundedPreceding
            case x if x < 0 => ValuePreceding(-start.toInt)
            case x if x > 0 => ValueFollowing(start.toInt)
          }
      

      ( look at this .toInt )

      Does anybody know it there's a way to solve / patch this behavior ?

      Any help will be appreciated

      Thx

      Attachments

        Activity

          People

            jiangxb1987 Xingbo Jiang
            jchamp Julien Champ
            Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: