[SPARK-32060] Huber loss Convergence

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Resolved
• Affects Version/s: 3.1.0
• Component/s: ML
• Assignee: Unassigned
• Reporter: Ruifeng Zheng
• Created: 23/Jun/20 01:29
• Updated: 18/Mar/21 06:38
• Resolved: 18/Mar/21 06:38

Description

performace test in https://issues.apache.org/jira/browse/SPARK-31783, Huber loss seems start to diverge since 70 iters.

 

 for (size <- Seq(1, 4, 16, 64); iter <- Seq(10, 50, 100)) {
    Thread.sleep(10000)
    val hlir = new LinearRegression().setLoss("huber").setSolver("l-bfgs").setMaxIter(iter).setTol(0)
    val start = System.currentTimeMillis
    val model = hlir.setBlockSize(size).fit(df)
    val end = System.currentTimeMillis
    println((model.uid, size, iter, end - start, model.summary.objectiveHistory.last, model.summary.totalIterations, model.coefficients.toString.take(100)))
}

Issue Links

• relates to: SPARK-3181 Add Robust Regression Algorithm with Huber Estimator