Major Description
2 major efficiency issues in computation and storage:
(1) DecisionTree aggregation involves reshaping data unnecessarily.
E.g., the internal methods extractNodeInfo() and getBinDataForNode() involve reshaping the data multiple times without real computation.
(2) DecisionTree splits and aggregate bins can include many unused bins/splits.
The same number of splits/bins are used for all features. E.g., if there is a continuous feature which uses 100 bins, then there will also be 100 bins allocated for all binary features, even though only 2 are necessary.
Attachments
Issue Links
- contains
-
SPARK-3157 Avoid duplicated stats in DecisionTree extractLeftRightNodeAggregates
-
- Closed
-
- links to
