[SPARK-13964] Feature hashing improvements

• Type: Umbrella
• Status: Resolved
• Priority: Minor
• Resolution: Incomplete
• Component/s: ML, MLlib
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Nicholas Pentreath
• Created: 17/Mar/16 08:55
• Updated: 21/May/19 04:11
• Resolved: 21/May/19 04:11

Description

Investigate improvements to Spark ML feature hashing (see e.g. http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.FeatureHasher.html#sklearn.feature_extraction.FeatureHasher).

Issue Links

• contains: SPARK-21748 Migrate the implementation of HashingTF from MLlib to ML
• contains: SPARK-21481 Add indexOf method in ml.feature.HashingTF similar to mllib.feature.HashingTF
• relates to: SPARK-22801 Allow FeatureHasher to specify numeric columns to treat as categorical

Sub-Tasks

• Add binary toggle Param to ml.HashingTF: Resolved
• Use MurmurHash3 for hashing String features: Closed
• Extend input format that feature hashing can handle: Resolved
• HashingTF should extend UnaryTransformer: Resolved
• HashingTF should use MurmurHash3: Resolved
• Add binary toggle Param to PySpark HashingTF in ML & MLlib: Resolved
• PySpark HashingTF hashAlgorithm param + docs: Resolved