[SPARK-22949] Reduce memory requirement for TrainValidationSplit

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.0, 2.4.0
• Component/s: ML
• Target Version/s: 2.3.0, 2.4.0
• Assignee: Bago Amirbekian
• Reporter: Bago Amirbekian
• Created: 03/Jan/18 23:21
• Updated: 05/Jan/18 06:45
• Resolved: 05/Jan/18 06:45

Description

There was a fix in {{ CrossValidator }} to reduce memory usage on the driver, the same patch to be applied to {{ TrainValidationSplit }}.

Issue Links

• relates to: SPARK-21535 Reduce memory requirement for CrossValidator and TrainValidationSplit
• relates to: SPARK-22707 Optimize CrossValidator memory occupation by models in fitting
• links to: [Github] Pull Request #20143 (MrBago)