Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14043

Remove restriction on maxDepth for decision trees

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Incomplete
    • None
    • None
    • ML

    Description

      We currently restrict decision trees (DecisionTree, GBT, RandomForest) to be of maxDepth <= 30. We should remove this restriction to support deep (imbalanced) trees.

      Trees store an index for each node, where each index corresponds to a unique position in a binary tree. (I.e., the first index of row 0 is 1, the first of row 1 is 2, the first of row 2 is 4, etc., IIRC)

      With some careful thought, we could probably avoid using indices altogether.

      Attachments

        Issue Links

          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-14045 DecisionTree improvement umbrella

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-3162 Train DecisionTree locally when possible

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              josephkb Joseph K. Bradley
              Votes:
              5 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: