[KYLIN-2281] Pending on the first step(save data from kafka) when building streaming cube

• Type: Test
• Status: Closed
• Priority: Critical
• Resolution: Pending Closed
• Affects Version/s: v1.6.0
• Component/s: Client - CLI
• Environment: CDH5.8 + Kylin1.6 + Kafka0.10
• Flags: Important
• Assignee: Unassigned
• Reporter: QiLiFei
• Created: 14/Dec/16 13:45
• Updated: 05/Jan/17 06:31
• Resolved: 05/Jan/17 06:31

Description

Hi engineer,

1.
I tried several times to build the streaming cube based on kafka source . However , it will always pending on the first step(already take about 160 mins ) to save data from kafka. I checked the data from kafka . There are only about one thousand data . Which is confused for me . Anyone who can help me ? I'm not sure what to do next . Is there configuration for kafka or kylin need to reset or not ?

2.Command
curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '

{ "sourceOffsetStart": 0, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}

' http://172.31.18.62:7070/kylin/api/cubes/StreamingCube27/build2
3. config and log
I set the available mb for kylin 16G of 20G. I can see the continously log info by command tail -f kylin.log as below . Currenty I've stop the kafka producer for this topic.
2016-12-15 11:19:41,457 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
2016-12-15 11:19:41,462 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771971443, to newTs: 1481771981457, operation result: true
2016-12-15 11:19:51,505 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
2016-12-15 11:19:51,510 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771981457, to newTs: 1481771991505, operation result: true
4.Kafka souce data count （sourceOffset & endOffset）
phadoopdn1:/opt/kafka_2.11-0.10.1.0 # bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.31.18.62:9092 --topic kylin_demo30 --time -1
kylin_demo30: 0:70572
phadoopdn1:/opt/kafka_2.11-0.10.1.0 #

Anyone who can help me to figure out this issue ?