[SPARK-20542] Add an API into Bucketizer that can bin a lot of columns all at once

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: ML
• Assignee: L. C. Hsieh Assign to me
• Reporter: L. C. Hsieh
• Created: 01/May/17 02:54
• Updated: 18/Sep/19 08:56
• Resolved: 09/Nov/17 14:37

Description

Current ML's Bucketizer can only bin a column of continuous features. If a dataset has thousands of of continuous columns needed to bin, we will result in thousands of ML stages. It is very inefficient regarding query planning and execution.

We should have a type of bucketizer that can bin a lot of columns all at once. It would need to accept an list of arrays of split points to correspond to the columns to bin, but it might make things more efficient by replacing thousands of stages with just one.

Issue Links

• Is contained by: SPARK-8418 Add single- and multi-value support to ML Transformers
• is related to: SPARK-20392 Slow performance when calling fit on ML pipeline for dataset with many columns but few rows
• relates to: SPARK-12225 Support adding or replacing multiple columns at once in DataFrame API
• relates to: SPARK-22799 Bucketizer should throw exception if single- and multi-column params are both set
• links to: [Github] Pull Request #17819 (viirya)
• links to: [Github] Pull Request #19892 (zhengruifeng)