Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-2281

Pending on the first step(save data from kafka) when building streaming cube

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Closed
    • Priority: Critical
    • Resolution: Pending Closed
    • Affects Version/s: v1.6.0
    • Fix Version/s: None
    • Component/s: Client - CLI
    • Labels:
      None
    • Environment:
      CDH5.8 + Kylin1.6 + Kafka0.10
    • Flags:
      Important

      Description

      Hi engineer,

      1.
      I tried several times to build the streaming cube based on kafka source . However , it will always pending on the first step(already take about 160 mins ) to save data from kafka. I checked the data from kafka . There are only about one thousand data . Which is confused for me . Anyone who can help me ? I'm not sure what to do next . Is there configuration for kafka or kylin need to reset or not ?

      2.Command
      curl -X PUT --user ADMIN:KYLIN -H "Content-Type: application/json;charset=utf-8" -d '

      { "sourceOffsetStart": 0, "sourceOffsetEnd": 9223372036854775807, "buildType": "BUILD"}

      ' http://172.31.18.62:7070/kylin/api/cubes/StreamingCube27/build2
      3. config and log
      I set the available mb for kylin 16G of 20G. I can see the continously log info by command tail -f kylin.log as below . Currenty I've stop the kafka producer for this topic.
      2016-12-15 11:19:41,457 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
      2016-12-15 11:19:41,462 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771971443, to newTs: 1481771981457, operation result: true
      2016-12-15 11:19:51,505 DEBUG [pool-9-thread-1] dao.ExecutableDao:210 : updating job output, id: dfb72486-50b8-44c1-bee5-4ad0393ef66e-00
      2016-12-15 11:19:51,510 DEBUG [pool-9-thread-1] hbase.HBaseResourceStore:262 : Update row /execute_output/dfb72486-50b8-44c1-bee5-4ad0393ef66e-00 from oldTs: 1481771981457, to newTs: 1481771991505, operation result: true
      4.Kafka souce data count (sourceOffset & endOffset)
      phadoopdn1:/opt/kafka_2.11-0.10.1.0 # bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 172.31.18.62:9092 --topic kylin_demo30 --time -1
      kylin_demo30: 0:70572
      phadoopdn1:/opt/kafka_2.11-0.10.1.0 #

      Anyone who can help me to figure out this issue ?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              BoyceZhang1 QiLiFei
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: