[SPARK-13969] Extend input format that feature hashing can handle

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.3.0
• Component/s: ML, MLlib
• Assignee: Nicholas Pentreath
• Reporter: Nicholas Pentreath
• Created: 17/Mar/16 09:44
• Updated: 17/Jan/18 08:55
• Resolved: 16/Aug/17 08:55

Description

Currently HashingTF works like CountVectorizer (the equivalent in scikit-learn is HashingVectorizer). That is, it works on a sequence of strings and computes term frequencies.

The use cases for feature hashing extend to arbitrary feature values (binary, count or real-valued). For example, scikit-learn's FeatureHasher can accept a sequence of (feature_name, value) pairs (e.g. a map, list). In this way, feature hashing can operate as both "one-hot encoder" and "vector assembler" at the same time.

Investigate adding a more generic feature hasher (that in turn can be used by HashingTF).

Issue Links

• relates to: SPARK-21468 FeatureHasher Python API
• relates to: SPARK-21469 Add doc and example for FeatureHasher
• relates to: SPARK-23127 Update FeatureHasher user guide for catCols parameter
• links to: [Github] Pull Request #18513 (MLnick)