[GRIFFIN-290] Fix bug for submitting job to livy

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Assignee: Unassigned
• Reporter: wan kun
• Created: 12/Sep/19 10:42
• Updated: 17/Sep/19 23:29
• Resolved: 17/Sep/19 23:29

Description

When griffin submit multiple DQ jobs to livy, the http parameter `name` is always griffin.
So livy will reject them.

job request :

 

[owner: null, request: [proxyUser: None, file: hdfs://nameservice-standby/user/kun.wan/measure-0.6.0-SNAPSHOT.jar,
args:

Unknown macro: { "spark" }

,
"sinks" : [
Unknown macro: { .... }
],
"griffin.checkpoint" : [ ]
},

Unknown macro: { "measure.type" }

],
"evaluate.rule" :

,
"measure.type" : "griffin"
},raw,raw, driverMemory: 1g, executorMemory: 6g, executorCores: 2, numExecutors: 6, queue: root.users.kun_dot_wan, name: griffin]]

 

livy Response : 

400 Bad Request
[Date:"Thu, 12 Sep 2019 10:00:00 GMT", Content-Type:"application/json;charset=utf-8", Content-Length:"47", Server:"Jetty(9.3.24.v20180605)"]

Unknown macro: {"msg"}

Issue Links

• links to: GitHub Pull Request #534