Affects Version/s: 2.1.0
Fix Version/s: None
There is a bit complicated Spark App using DataSet api run once a day, and I noticed the app will hang once in a while,
I add some log and compare two driver log which one belong to successful app, another belong to faied, and here is some results of investigation
1. Usually the app will running correctly, but sometime it will hang after finishing job 1
2. According to log I append , the successful app always reach the fixed point when iteration is 7 on Batch Operator Optimizations, but failed app never reached this fixed point.
3. The difference between two logical plan appear in BooleanSimplification on iteration, before this rule, two logical plan is same:
4. after BooleanSimplification, There is only one difference: Filter's constraints format like (A || B) && (A || C) on successful app but A && (B || C) on failed app.
5. hereafter, error occurred with rule InferFiltersFromConstraints:
Failed app plan's left child has an additional constraint:
5. Soon after the gap between two app's plan is getting bigger, one will successful, another hang. It seems there are two possibly reason :
1. BooleanSimplification is not idempotent
2. InferFiltersFromConstraints's behavior not correct when child's constraints has A || ( B && C) instead of (A || B) && (A || C)
I'm not sure which is root casue, could someone follow this question?