[KYLIN-3912] Support cube level mapreduce queue config for BeelineHiveClient

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v2.6.1
• Fix Version/s: v3.0.0-alpha2
• Component/s: Job Engine
• Assignee: Unassigned
• Reporter: Shaohui Liu
• Created: 27/Mar/19 05:22
• Updated: 21/Jan/20 07:43
• Resolved: 23/Apr/19 06:59

Description

To support multi tenants, we set different mapreduce queue config for different projects and cubes, but BeelineHiveClient don't use those configs. So the getHiveTableRows api always run on same queue in kylin_hive_conf or jdbc url, which cause computing resource competition.

 

2018-11-28 15:37:27,261 ERROR [Scheduler 1950398337 Job 08b3ee43-c84d-4039-84c5-a36ecb2cff18-228] execution.AbstractExecutable:383 : job:08b3ee43-c84d-4039-84c5-a36ecb2cff18-01 execute finished with exception
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
downstreamError is null.
Query log: http://zjy-hadoop-prc-ct14.bj:28911/log?qid=a05e1629-2072-46dd-9d71-b5722d04b2aa
        at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
        at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376)
        at org.apache.kylin.source.hive.BeelineHiveClient.getHiveTableRows(BeelineHiveClient.java:108)
        at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.computeRowCount(HiveMRInput.java:304)
        at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:354)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
        at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
        at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
        at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #556
• links to: GitHub Pull Request #590