Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13964 Feature hashing improvements
  3. SPARK-13968

Use MurmurHash3 for hashing String features

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: ML, MLlib
    • Labels:
      None

      Description

      Typically feature hashing is done on strings, i.e. feature names (or in the case of raw feature indexes, either the string representation of the numerical index can be used, or the index used "as-is" and not hashed).

      It is common to use a well-distributed hash function such as MurmurHash3. This is the case in e.g. Scikit-learn.

      Currently Spark's HashingTF uses the object's hash code. Look at using MurmurHash3 (at least for String which is the common case).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                yanboliang Yanbo Liang
                Reporter:
                mlnick Nicholas Pentreath
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: