Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31332

Proposal to add Proximity Measure in Random Forest

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.5
    • None
    • ML

    Description

      Background

      The RandomForest model does not provide proximity measure as described in [Breiman|https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm]. There are many important use cases of proximity:

      • more accurate replacement for missing data
      • identify outliers
      • clustering or multi-dimensional scaling
      • compute the proximities of test set in the training set
      • unsupervised learning

      Performance and storage concerns are among reasons that proximities are not computed and kept during prediction, as mentioned in https://dzone.com/articles/classification-using-random-forest-with-spark-20.

      Proposal

      RF in Spark is optimized for massive scalability on large-scale dataset where the number of data points, features and trees can be very big. Even with optimized storage, proximity requires O(NxT) memory, and it may still not fit in memory: where N is number of data points and T is number of trees in the forest.

      We propose to add a column in the prediction output to return the node-id (or hash) of the terminal node for each sample data point.

      The required changes on the current RF implementation will not increase the computation and storage by significant amounts. And it will leave the possibility open for computing some form of proximity after prediction. It us up to the users how to use the extra column of node-ids. Without this, currently there is no work around to compute proximity measure.

      Experiment on Spark 2.3.1 and 2.4.5

      In one prototype, we output the terminal node id for each prediction from RandomForestClassificationModel. And then we use Spark’s LSHModel to cluster prediction results by terminal node ids. The performance of the whole pipeline was reasonable for the size of our dataset.

      References

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            spoon Stanley Poon
            Sean R. Owen Sean R. Owen
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Issue deployment