Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
On stratstats 10K, the codegen optimizer gets stuck during plan enumeration of a DAG partition with 274,877,906,944 plans. During initial compilation, the same partition was processed in 2.8s as most candidate plans have been successfully pruned by cost or structure.
After a detailed analysis, we can characterize the problem as follows: there is a fusion partition with 38 interesting points, resulting in the huge search space of 274 billion plans. Furthermore, all these materialization points are extremely small while at the same time there are large inputs and compute-intensive operations as well as many common subexpression, which renders the pruning by costs starting from the fuse-all heuristic ineffective.
We can address this as follows: First, we should run both heuristics (fuse-all and fuse-no-redundancy) which are the first and last plan upfront to quickly obtain a good lower bound for the costs. Second, we should only consider enumerating large partitions (say with more than 20 interesting points) if the opening heuristics show costs that is further than an epsilon (say 1%) away from the minimal static costs. Together these rules will make the optimizer significantly more robust without missing any meaningful plan choices.
Attachments
Issue Links
- relates to
-
SYSTEMDS-1284 Code generation for operator fusion
-
- Closed
-