Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18334

What hashDistance should MinHash use?

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • None
    • None
    • None

    Description

      MinHash currently is using the same `hashDistance` function as RandomProjection. This does not make sense for MinHash because the Jaccard distance of two sets is not relevant to the absolute distance of their hash buckets indices.

      This bug could affect accuracy of multi probing NN search for MinHash.

      Attachments

        Issue Links

          Is contained by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18454 Changes to improve Nearest Neighbor Search for LSH

          • Major - Major loss of function.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. SPARK-5992 Locality Sensitive Hashing (LSH)

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-18392 LSH API, algorithm, and documentation follow-ups

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15800 [Github] Pull Request #15800 (Yunni)

          Activity

            People

              Unassigned Unassigned
              yunn Yun Ni
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: