[SPARK-18409] LSH approxNearestNeighbors should use approxQuantile instead of sort

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Component/s: ML
• Labels: bulk-closed
• Assignee: Huaxin Gao
• Reporter: Joseph K. Bradley
• Created: 11/Nov/16 02:40
• Updated: 17/Sep/20 03:05
• Resolved: 21/May/19 04:13

Description

LSHModel.approxNearestNeighbors sorts the full dataset on the hashDistance in order to find a threshold. It should use approxQuantile instead.

Issue Links

• Is contained by: SPARK-18454 Changes to improve Nearest Neighbor Search for LSH
• is related to: SPARK-30120 LSH approxNearestNeighbors should use BoundedPriorityQueue when numNearestNeighbors is small
• links to: [Github] Pull Request #16966 (Yunni)
• links to: [Github] Pull Request #29778 (zhengruifeng)
• links to: GitHub Pull Request #16966
• links to: GitHub Pull Request #26415
• links to: GitHub Pull Request #26990