[KYLIN-4384] Spark build failed in EMR when KYLIN-4224 introduced

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: v3.1.0
• Component/s: Spark Engine
• Labels: aws-emr aws-glue
• Environment: AWS EMR 5.28 with Glue support
• Assignee: Xiaoxiang Yu
• Reporter: Xiaoxiang Yu
• Created: 21/Feb/20 12:16
• Updated: 31/Jul/20 12:25
• Resolved: 26/Feb/20 07:17

Description

I built a package using the latest code of master branch to test the Kylin Glue integration. 

 

I create a EMR cluster(5.28) with glue enabled. In my test, MR build engine works fine.  But spark build engine failed at the first step, and I found the code change is introduced in https://issues.apache.org/jira/browse/KYLIN-4224.

 

The new added step change the way that how intermediate table was created. And it is mandatory, so nobody can disable or change it without rebuild the binary. It is not good in my side. Currently, I can not find the root cause and I am not good at spark indeed. But I suggest we should add a config to make it easy to disable that feature.

 

Issue Links

• is blocked by: KYLIN-4224 Create flat table wich spark sql
• links to: GitHub Pull Request #1118