Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3465

kylin Kafka stream incremental build error

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Do
    • Affects Version/s: v2.4.0
    • Fix Version/s: None
    • Component/s: Environment , NRT Streaming
    • Labels:
      None

      Description

      Development environment:

      • Centos 7
      • Hadoop 2.7.2
      • Hbase 1.2.5
      • Hive 1.2.2
      • Kylin-2.4.0-bin-hbase1x
      • Kafka 0.10.2.2

      I built a cluster consisting of three machines that build the cube from the Kafka stream, but each time a new incremental build is made, the following error is displayed.

       
      {{2018-07-24 14:27:09,269 WARN [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] config.KafkaConsumerProperties:142 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable 2018-07-24 14:27:09,295 ERROR [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] hadoop.KafkaFlatTableJob:131 : error in KafkaFlatTableJob java.lang.NullPointerException at org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.setupMapper(KafkaFlatTableJob.java:143) at org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.run(KafkaFlatTableJob.java:118) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-07-24 14:27:09,296 ERROR [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] common.MapReduceExecutable:138 : error execute MapReduceExecutable{id=faf6db81-bc76-4b25-b838-8ca993e889d5-00, name=Save data from Kafka, state=RUNNING} java.lang.NullPointerException at org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.setupMapper(KafkaFlatTableJob.java:143) at org.apache.kylin.source.kafka.hadoop.KafkaFlatTableJob.run(KafkaFlatTableJob.java:118) at org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2018-07-24 14:27:09,302 INFO [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] execution.ExecutableManager:411 : job id:faf6db81-bc76-4b25-b838-8ca993e889d5-00 from RUNNING to ERROR 2018-07-24 14:27:09,310 INFO [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] execution.ExecutableManager:411 : job id:faf6db81-bc76-4b25-b838-8ca993e889d5 from RUNNING to ERROR 2018-07-24 14:27:09,310 DEBUG [Scheduler 268704591 Job faf6db81-bc76-4b25-b838-8ca993e889d5-602] execution.AbstractExecutable:310 : no need to send email, user list is empty }}
      My current solution is to re-start the service of the job node every time, and build the incremental build again.

      Kylin.properties

      kylin.server.mode=job
      kylin.server.cluster-servers=99.48.1.2:7070,99.48.1.3:7070

      kylin.server.mode=query
      kylin.server.cluster-servers=99.48.1.2:7070,99.48.1.3:7070

      kylin.server.mode=query
      kylin.server.cluster-servers=99.48.1.2:7070,99.48.1.3:7070

      99.48.1.1:7070 job
      99.48.1.2:7070 query
      99.48.1.3:7070 query

      Thanks

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              chenwen.wang chenwen
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: