Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19451

rangeBetween method should accept Long value as boundary

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.1, 2.0.2
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Hi there,

      there seems to be a major limitation in spark window functions and rangeBetween method.

      If I have the following code :

      Exemple
          val tw =  Window.orderBy("date")
            .partitionBy("id")
            .rangeBetween( from , 0)
      

      Everything seems ok, while from value is not too large... Even if the rangeBetween() method supports Long parameters.
      But.... If i set -2160000000L value to from it does not work !

      It is probably related to this part of code in the between() method, of the WindowSpec class, called by rangeBetween()

      between() method
          val boundaryStart = start match {
            case 0 => CurrentRow
            case Long.MinValue => UnboundedPreceding
            case x if x < 0 => ValuePreceding(-start.toInt)
            case x if x > 0 => ValueFollowing(start.toInt)
          }
      

      ( look at this .toInt )

      Does anybody know it there's a way to solve / patch this behavior ?

      Any help will be appreciated

      Thx

        Attachments

          Activity

            People

            • Assignee:
              jiangxb1987 Xingbo Jiang
              Reporter:
              jchamp Julien Champ
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: