[SPARK-33043] RowMatrix is incompatible with spark.driver.maxResultSize=0

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0, 3.0.1
• Fix Version/s: 3.0.2, 3.1.0
• Component/s: MLlib
• Assignee: Sean R. Owen
• Reporter: Karen Feng
• Created: 01/Oct/20 00:31
• Updated: 06/Oct/20 23:58
• Resolved: 03/Oct/20 18:13

Description

RowMatrix does not work if spark.driver.maxResultSize=0, as this requirement breaks:

 

require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,      s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "        + s"as it's bigger than maxResultSize ($maxDriverResultSizeInBytes Bytes)")

 

https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala#L795.

 

This check should likely only happen if maxDriverResultSizeInBytes > 0.

Issue Links

• links to: [Github] Pull Request #29925 (srowen)
• links to: [Github] Pull Request #29925 (srowen)