Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-404

Can't get cube source record size.

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.0, v1.4.0
    • Component/s: None
    • Labels:

      Description

      Dear All!
      I deploy kylin on Apache Hadoop, when i build the job , it failed on the last step. here is the some log message:
      ```java
      [QuartzScheduler_Worker-4]:[2014-11-24 19:24:47,280][INFO][com.kylinolap.job.flow.JobFlowListener.jobWasExecuted(JobFlowListener.java:93)] - cube_job_group.test_kylin_cube_without_slr_empty.30bbd69c-33ae-4983-aaf7-f46ab42b88dd.14 status: FINISHED
      [QuartzScheduler_Worker-4]:[2014-11-24 19:24:47,280][DEBUG][com.kylinolap.cube.CubeManager.loadCubeInstance(CubeManager.java:604)] - Loading CubeInstance kylin_metadata_qa(key='/cube/test_kylin_cube_without_slr_empty.json')@kylin_metadata_qa@hbase:192.168.44.21:2181:/hbase9827
      [QuartzScheduler_Worker-4]:[2014-11-24 19:24:47,295][DEBUG][com.kylinolap.common.persistence.ResourceStore.putResource(ResourceStore.java:166)] - Saving resource /job/30bbd69c-33ae-4983-aaf7-f46ab42b88dd (Store kylin_metadata_qa@hbase:192.168.44.21:2181:/hbase9827)
      [QuartzScheduler_Worker-4]:[2014-11-24 19:24:47,300][INFO][com.kylinolap.job.flow.JobFlowListener.updateCubeSegmentInfoOnSucceed(JobFlowListener.java:238)] - Updating cube segment FULL_BUILD for cube test_kylin_cube_without_slr_empty
      [QuartzScheduler_Worker-4]:[2014-11-24 19:24:47,301][ERROR][com.kylinolap.job.flow.JobFlowListener.jobWasExecuted(JobFlowListener.java:117)] - Can't get cube source record size.
      java.lang.RuntimeException: Can't get cube source record size.
      at com.kylinolap.job.flow.JobFlowListener.updateCubeSegmentInfoOnSucceed(JobFlowListener.java:274)
      at com.kylinolap.job.flow.JobFlowListener.jobWasExecuted(JobFlowListener.java:99)
      at org.quartz.core.QuartzScheduler.notifyJobListenersWasExecuted(QuartzScheduler.java:1985)
      at org.quartz.core.JobRunShell.notifyJobListenersComplete(JobRunShell.java:340)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:224)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
      ```
      here is some version info:
      Kylin 0.6.2
      hadoop 2.4.1
      hbase 0.98.4-hadoop2
      zookeeper 3.4.5
      hive 0.13.0

      I find some config info on hbase
      ```
      get 'kylin_metadata_qa_job', '/job/1eaf05f8-e9f5-4dd7-9748-ab9b3098a512'
      {
      "uuid": "1eaf05f8-e9f5-4dd7-9748-ab9b3098a512",
      "name": "test_kylin_cube_without_slr_empty - FULL_BUILD - BUILD - PST 2014-11-24 23:00:15",
      "type": "BUILD",
      "duration": 461,
      "steps": [

      { "name": "Create Intermediate Flat Hive Table", "info": null, "interruptCmd": null, "sequence_id": 0, "exec_cmd": "hive -e "DROPTABLEIFEXISTSkylin_intermediate_test_kylin_cube_without_slr_desc_FULL_BUILD_1eaf05f8_e9f5_4dd7_9748_ab9b3098a512;nCREATEEXTERNALTABLEIFNOTEXISTSkylin_intermediate_test_kylin_cube_without_slr_desc_FULL_BUILD_1eaf05f8_e9f5_4dd7_9748_ab9b3098a512n(nCAL_DTdaten, LEAF_CATEG_IDintn, LSTG_SITE_IDintn, META_CATEG_NAMEstringn, CATEG_LVL2_NAMEstringn, CATEG_LVL3_NAMEstringn, LSTG_FORMAT_NAMEstringn, SLR_SEGMENT_CDsmallintn, PRICEdecimal(38, 16)n, SELLER_IDbigintn) ...}

      ]
      ```
      in source code: com.kylinolap.job.flow.JobFlowListener.updateCubeSegmentInfoOnSucceed(JobFlowListener.java:272)
      ```
      JobStep createFlatTableStep = jobInstance.findStep(JobConstants.STEP_NAME_CREATE_FLAT_HIVE_TABLE);
      if (null != createFlatTableStep) {
      String sourceRecordsSize = createFlatTableStep.getInfo(JobInstance.SOURCE_RECORDS_SIZE);
      if (sourceRecordsSize == null || sourceRecordsSize.equals(""))

      { throw new RuntimeException("Can't get cube source record size."); }

      sourceSize = Long.parseLong(sourceRecordsSize);
      } else

      { log.info("No step with name '" + JobConstants.STEP_NAME_CREATE_FLAT_HIVE_TABLE + "' is found"); }

      ```

      the "info" is null, cased throw a RuntimeException, some one have idea with it??

      ---------------- Imported from GitHub ----------------
      Url: https://github.com/KylinOLAP/Kylin/issues/101
      Created by: Yancey1989
      Labels: help wanted,
      Assignee: binmahone
      Created at: Tue Nov 25 14:22:13 CST 2014
      State: closed

        Attachments

          Activity

            People

            • Assignee:
              shaofengshi Shao Feng Shi
              Reporter:
              lukehan Luke Han

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment