Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3912

Support cube level mapreduce queue config for BeelineHiveClient

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v2.6.1
    • Fix Version/s: v3.0.0-alpha2
    • Component/s: Job Engine
    • Labels:
      None

      Description

      To support multi tenants, we set different mapreduce queue config for different projects and cubes, but BeelineHiveClient don't use those configs. So the getHiveTableRows api always run on same queue in kylin_hive_conf or jdbc url, which cause computing resource competition.

       

      2018-11-28 15:37:27,261 ERROR [Scheduler 1950398337 Job 08b3ee43-c84d-4039-84c5-a36ecb2cff18-228] execution.AbstractExecutable:383 : job:08b3ee43-c84d-4039-84c5-a36ecb2cff18-01 execute finished with exception
      java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
      downstreamError is null.
      Query log: http://zjy-hadoop-prc-ct14.bj:28911/log?qid=a05e1629-2072-46dd-9d71-b5722d04b2aa
              at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
              at org.apache.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:376)
              at org.apache.kylin.source.hive.BeelineHiveClient.getHiveTableRows(BeelineHiveClient.java:108)
              at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.computeRowCount(HiveMRInput.java:304)
              at org.apache.kylin.source.hive.HiveMRInput$RedistributeFlatHiveTableStep.doWork(HiveMRInput.java:354)
              at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
              at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
              at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
              at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              liushaohui Shaohui Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: