Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14657

RFormula output wrong features when formula w/o intercept

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.3.0
    • ML
    • None

    Description

      SparkR::glm output different features compared with R glm when fit w/o intercept and having string/category features. Take the following example, SparkR output three features compared with four features for native R.
      SparkR::glm

      training <- suppressWarnings(createDataFrame(sqlContext, iris))
      model <- glm(Sepal_Width ~ Sepal_Length + Species - 1, data = training)
      summary(model)

      Coefficients:
      Estimate Std. Error t value Pr(>|t|)
      Sepal_Length 0.67468 0.0093013 72.536 0
      Species_versicolor -1.2349 0.07269 -16.989 0
      Species_virginica -1.4708 0.077397 -19.003 0

      stats::glm

      summary(glm(Sepal.Width ~ Sepal.Length + Species - 1, data = iris))

      Coefficients:
      Estimate Std. Error t value Pr(>|t|)
      Sepal.Length 0.3499 0.0463 7.557 4.19e-12 ***
      Speciessetosa 1.6765 0.2354 7.123 4.46e-11 ***
      Speciesversicolor 0.6931 0.2779 2.494 0.0137 *
      Speciesvirginica 0.6690 0.3078 2.174 0.0313 *

      The encoder for string/category feature is different. R did not drop any category but SparkR drop the last one.
      I searched online and test some other cases, found when we fit R glm model(or other models powered by R formula) w/o intercept on a dataset including string/category features, one of the categories in the first category feature is being used as reference category, we will not drop any category for that feature.

      I think we should keep consistent semantics between Spark RFormula and R formula.

      cc Xiangrui Meng

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            yanboliang Yanbo Liang
            yanboliang Yanbo Liang
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment