[SPARK-31517] SparkR::orderBy with multiple columns descending produces error

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.5
• Fix Version/s: 3.1.0, 3.2.0
• Component/s: SparkR

Environment

Databricks Runtime 6.5

• Assignee: Michael Chirico
• Reporter: Ross Bowen
• Created: 22/Apr/20 08:33
• Updated: 12/Dec/22 18:11
• Resolved: 17/Dec/20 08:22

Description

When specifying two columns within an `orderBy()` function, to attempt to get an ordering by two columns in descending order, an error is returned.

library(magrittr) 
library(SparkR) 
cars <- cbind(model = rownames(mtcars), mtcars) 
carsDF <- createDataFrame(cars) 

carsDF %>% 
  mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")), desc(column("mpg")), desc(column("disp"))))) %>% 
  head() 

This returns an error:

 Error in ns[[i]] : subscript out of bounds

This seems to be related to the more general issue that the following code, excluding the use of the `desc()` function also fails:

carsDF %>% 
  mutate(rank = over(rank(), orderBy(windowPartitionBy(column("cyl")), column("mpg"), column("disp")))) %>% 
  head()

 

Issue Links

• duplicates: SPARK-26199 Long expressions cause mutate to fail
• is duplicated by: SPARK-26199 Long expressions cause mutate to fail
• links to: [Github] Pull Request #28386 (MichaelChirico)