Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1912

Create Intermediate Flat Hive Table failed when using beeline

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: v1.5.2
    • Fix Version/s: v1.5.3
    • Component/s: Job Engine
    • Labels:
      None

      Description

      When beeline is enabled with parameters:
      kylin.hive.client = beeline
      kylin.hive.beeline.params=-n root --hiveconf hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.|dfs.' -u 'jdbc:hive2://localhost:10000'

      The build cube will fail with following exceptions:

      java.lang.IllegalArgumentException: Map input splits are 0 bytes, something is wrong!
      	at org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
      	at org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
      	at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
      	at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
      	at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
      	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
      	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      2016-07-21 15:05:38,615 INFO  [pool-5-thread-6] common.AbstractHadoopJob:479 : tempMetaFileString is : file:///opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/kylin_job_meta5547890224035214519/meta
      2016-07-21 15:05:38,619 ERROR [pool-5-thread-6] execution.AbstractExecutable:128 : error execute MapReduceExecutable{id=782765e2-5357-4cc3-8430-b4ee9ffc69de-05, name=Build Base Cuboid Data, state=RUNNING}
      java.lang.IllegalArgumentException: Map input splits are 0 bytes, something is wrong!
      	at org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:528)
      	at org.apache.kylin.engine.mr.steps.CuboidJob.setReduceTaskNum(CuboidJob.java:183)
      	at org.apache.kylin.engine.mr.steps.CuboidJob.run(CuboidJob.java:139)
      	at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:91)
      	at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:121)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
      	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
      	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      After checking the detail logs, I found the intermediate table may not create succeed.

      2016-07-21 15:15:50,361 INFO  [pool-5-thread-1] execution.AbstractExecutable:100 : Executing >>>>>>>>>>>>>   Create Intermediate Flat Hive Table   <<<<<<<<<<<<<
      2016-07-21 15:15:50,376 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from oldTs: 1469085306322, to newTs: 1469085350362, operation result: true
      2016-07-21 15:15:50,376 INFO  [pool-5-thread-1] manager.ExecutableManager:274 : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from READY to RUNNING
      2016-07-21 15:15:50,376 INFO  [pool-5-thread-1] execution.AbstractExecutable:49 : executing:beeline  -f /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql;rm -f /opt/client/kylin/apache-kylin-1.5.2.1-bin/bin/./../tomcat/temp/beeline_346879081651464139.hql
      2016-07-21 15:15:53,999 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from oldTs: 1469085350362, to newTs: 1469085353989, operation result: true
      2016-07-21 15:15:54,028 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from oldTs: 1469085353989, to newTs: 1469085354001, operation result: true
      2016-07-21 15:15:54,062 DEBUG [pool-5-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from oldTs: 1469085354001, to newTs: 1469085354030, operation result: true
      2016-07-21 15:15:54,062 INFO  [pool-5-thread-1] manager.ExecutableManager:274 : job id:8bd42c24-9fef-4d9f-887f-7de1dac57f7e-00 from RUNNING to SUCCEED
      

      Now use beeline command to create the intermediate table, return SUCCEED in only 4 seconds, normally build is 40+ seconds with Hive CLI.

      Then check the code, found beeline could not process ";" properly. Beeline execute each statement for each line, not split by ";". But Kylin has the variable 'useDatabaseHql' with value 'USE default;" without the 'new line'. That's the root cause. Not a difficult problem, buy may stop many new beeline user.

        Attachments

        1. KYLIN-1912.patch
          3 kB
          Billy Liu

          Activity

            People

            • Assignee:
              liuyiming.vip Yiming Liu
              Reporter:
              yimingliu Billy Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: