[SPARK-13448] Document MLlib behavior changes in Spark 2.0 - ASF JIRA

XML

Word

Printable

JSON

This JIRA keeps a list of MLlib behavior changes in Spark 2.0. So we can remember to add them to the migration guide / release notes.

~~SPARK-13429~~: change convergenceTol in LogisticRegressionWithLBFGS from 1e-4 to 1e-6.
~~SPARK-7780~~: Intercept will not be regularized if users train binary classification model with L1/L2 Updater by LogisticRegressionWithLBFGS, because it calls ML LogisticRegresson implementation. Meanwhile if users set without regularization, training with or without feature scaling will return the same solution by the same convergence rate(because they run the same code route), this behavior is different from the old API.
~~SPARK-12363~~: Bug fix for PowerIterationClustering which will likely change results
~~SPARK-13048~~: LDA using the EM optimizer will keep the last checkpoint by default, if checkpointing is being used.
~~SPARK-12153~~: Word2Vec now respects sentence boundaries. Previously, it did not handle them correctly.
~~SPARK-10574~~: HashingTF uses MurmurHash3 by default in both spark.ml and spark.mllib
~~SPARK-14768~~: Remove expectedType arg for PySpark Param
~~SPARK-14931~~: Mismatched default Param values between pipelines in Spark and PySpark
~~SPARK-13600~~: QuantileDiscretizer now uses approxQuantile from DataFrame stats (previously used custom sampling logic). Buckets will differ for same input data and params.

is duplicated by

SPARK-14847 ML/MLlib breaking changes between 1.6 & 2.0

is related to

SPARK-14817 ML, Graph, R 2.0 QA: Programming guide update and migration guide

SPARK-15643 ML 2.0 QA: migration guide update

SPARK-17692 Document ML/MLlib behavior changes in Spark 2.1

relates to

SPARK-13429 Unify Logistic Regression convergence tolerance of ML & MLlib