[SPARK-18166] GeneralizedLinearRegression Wrong Value Range for Poisson Distribution

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.1.0
• Component/s: ML
• Flags: Important
• External issue URL: https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala
• Assignee: Wayne Zhang
• Reporter: Wayne Zhang
• Created: 28/Oct/16 22:01
• Updated: 14/Nov/16 11:14
• Resolved: 14/Nov/16 11:14

Description

The current implementation of Poisson GLM seems to allow only positive values (See below). This is not correct since the support of Poisson includes the origin.

override def initialize(y: Double, weight: Double): Double = {
require(y > 0.0, "The response variable of Poisson family " +
s"should be positive, but got $y")
y
}

The fix is easy, just change it to
require(y >= 0.0, "The response variable of Poisson family " +

Issue Links

• links to: [Github] Pull Request #15683 (actuaryzhang)