[SPARK-23562] RFormula handleInvalid should handle invalid values in non-string columns.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: ML
• Target Version/s: 2.4.0
• Assignee: Unassigned
• Reporter: Bago Amirbekian
• Shepherd: Joseph K. Bradley
• Created: 02/Mar/18 01:28
• Updated: 11/Apr/18 01:30
• Resolved: 11/Apr/18 01:29

Description

Currently when handleInvalid is set to 'keep' or 'skip' this only applies to String fields. Numeric fields that are null will either cause the transformer to fail or might be null in the resulting label column.

I'm not sure what the semantics of keep might be for numeric columns with null values, but we should be able to at least support skip for these types.
--> Discussed offline: null values can be converted to NaN values for "keep"

Issue Links

• relates to: SPARK-19947 RFormulaModel always throws Exception on transforming data with NULL or Unseen labels
• links to: [Github] Pull Request #20970 (yogeshg)

Sub-Tasks

• VectorAssembler should have handleInvalid to handle columns with null values: Resolved
• Forward RFormula handleInvalid Param to VectorAssembler: Resolved
• add python api for VectorAssembler handleInvalid: Resolved