[SPARK-17306] QuantileSummaries doesn't compress

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: SQL
• Assignee: Sean R. Owen
• Reporter: Sean Zhong
• Created: 30/Aug/16 00:34
• Updated: 11/Sep/16 08:58
• Resolved: 11/Sep/16 08:58

Description

compressThreshold was not referenced anywhere

class QuantileSummaries(
    val compressThreshold: Int,
    val relativeError: Double,
    val sampled: ArrayBuffer[Stats] = ArrayBuffer.empty,
    private[stat] var count: Long = 0L,
    val headSampled: ArrayBuffer[Double] = ArrayBuffer.empty) extends Serializable

And, it causes memory leak, QuantileSummaries takes unbounded memory

val summary = new QuantileSummaries(10000, relativeError = 0.001)
// Results in creating an array of size 100000000 !!! 
(1 to 100000000).foreach(summary.insert(_))

Issue Links

• links to: [Github] Pull Request #14976 (srowen)