Na Yang Thanks for working on this. This task is fairly large, and I think breaking the task into smaller ones would help in sharing the load and tracking progress. Could you please create smaller JIRAs for this? I'd image that supporting union would require work in the following area:
1. SparkCompiler changes: generate a SparkWork that contains UnionWork from logical operator tree.
2. SparkPlan modeling: represent the spark job in terms of a graph (rather than) list of SparkTran instances. We may need to enhance SparkTran interface.
3. SparkPlanGenerator: need to generate a plan from SparkWork, which needs to use Spark's union transformation to achieve the functionality..
4. other earas.
Tez can be a good reference point
Please feel free to create JIRAs for those or other areas.