[SPARK-11302] Multivariate Gaussian Model with Covariance matrix returns incorrect answer in some cases

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.1, 1.6.0
• Fix Version/s: 1.3.2, 1.4.2, 1.5.2, 1.6.0
• Component/s: MLlib
• Target Version/s: 1.3.2, 1.4.1, 1.5.2, 1.6.0
• Assignee: Sean R. Owen
• Reporter: eyal sharon
• Created: 25/Oct/15 16:10
• Updated: 28/Oct/15 06:15
• Resolved: 28/Oct/15 06:14

Description

I have been trying to apply an Anomaly Detection model using Spark MLib.

As an input, I feed the model with a mean vector and a Covariance matrix. ,assuming my features contain Co-variance.

Here are my input for the model ,and the model returns zero for each data point for this input.

MU vector -
1054.8, 1069.8, 1.3 ,1040.1
Cov' matrix -
165496.0 , 167996.0, 11.0 , 163037.0
167996.0, 170631.0, 19.0, 165405.0
11.0, 19.0 , 0.0, 2.0
163037.0, 165405.0 2.0 , 160707.0

Conversely, for the non covariance case, represented by this matrix ,the model is working and returns results as expected
165496.0, 0.0 , 0.0, 0.0
0.0, 170631.0, 0.0, 0.0
0.0 , 0.0 , 0.8, 0.0
0.0 , 0.0, 0.0, 160594.2

Issue Links

• links to: [Github] Pull Request #9293 (srowen)
• links to: [Github] Pull Request #9309 (srowen)