[SPARK-4038] Outlier Detection Algorithm for MLlib

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Component/s: MLlib
• Assignee: Unassigned
• Reporter: Ashutosh Trivedi
• Created: 21/Oct/14 19:30
• Updated: 15/Mar/18 17:54
• Resolved: 08/Apr/17 09:55

Description

The aim of this JIRA is to discuss about which parallel outlier detection algorithms can be included in MLlib.
The one which I am familiar with is Attribute Value Frequency (AVF). It scales linearly with the number of data points and attributes, and relies on a single data scan. It is not distance based and well suited for categorical data. In original paper a parallel version is also given, which is not complected to implement. I am working on the implementation and soon submit the initial code for review.
Here is the Link for the paper
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4410382

As pointed out by Xiangrui in discussion
http://apache-spark-developers-list.1001551.n3.nabble.com/MLlib-Contributing-Algorithm-for-Outlier-Detection-td8880.html
There are other algorithms also. Lets discuss about which will be more general and easily paralleled.

Sub-Tasks

• KMeans-based outlier detection: Resolved