[SPARK-31217] Unnecessary persist on cumulativeCounts in BinaryClassificationMetrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.4, 2.4.5
• Component/s: ML, MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: IcySanwitch
• Created: 22/Mar/20 15:32
• Updated: 25/May/21 01:52
• Resolved: 25/May/21 01:40

Description

In mllib.evaluation.BinaryClassificationMetrics, cumulativeCounts is cached in a lazy initialization. But when I run LogisticRegressionSummaryExample as well as ModelSelectionViaCrossValidationExample, I find that cached cumulativeCounts only used by one action during execution.
So I think it should not be cached in initilization, we can set an extra persist() API in this class, just as that the unpersist() API in BinaryClassificationMetrics releases cached cumulativeCounts.