Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16470

ml.regression.LinearRegression training data do not check whether the result actually reach convergence

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 2.0.0
    • Fix Version/s: 2.0.0
    • Component/s: ML, Optimizer
    • Labels:
      None

      Description

      In `ml.regression.LinearRegression`, it use breeze `LBFGS` and `OWLQN` optimizer to do data training, but do not check whether breeze's optimizer returned result actually reached convergence.

      In fact, the `LBFGS` and `OWLQN` optimizer in breeze finish iteration may result in the following situations:

      1) reach max iteration number
      2) function reach value convergence
      3) objective function stop improving
      4) gradient reach convergence
      5) search failed(due to some internal numerical error)

      only (2) & (4) situations above can be regarded as convergence result.
      in other situation, we should at least print some warning to remind spark user.

        Attachments

          Activity

            People

            • Assignee:
              weichenxu123 Weichen Xu
              Reporter:
              weichenxu123 Weichen Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified