[SPARK-24431] wrong areaUnderPR calculation in BinaryClassificationEvaluator

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: ML
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Xinyong Tian
• Created: 30/May/18 16:21
• Updated: 08/Oct/19 05:42
• Resolved: 08/Oct/19 05:42

Description

My problem, I am using CrossValidator(estimator=LogisticRegression(...), ...,  evaluator=BinaryClassificationEvaluator(metricName='areaUnderPR'))  to select best model. when the regParam in logistict regression is very high, no variable will be selected (no model), ie every row 's prediction is same ,eg. equal event rate (baseline frequency). But at this point,  BinaryClassificationEvaluator set the areaUnderPR highest. As a result  best model seleted is a no model. 

the reason is following.  at time of no model, precision recall curve will be only two points: at recall =0, precision should be set to  zero , while the software set it to 1. at recall=1, precision is the event rate. As a result, the areaUnderPR will be close 0.5 (my even rate is very low), which is maximum .

the solution is to set precision =0 when recall =0.

Issue Links

• relates to: SPARK-21806 BinaryClassificationMetrics pr(): first point (0.0, 1.0) is misleading