Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-1402

StringIndexOutOfBoundsException in Kylin Hive Column Cardinality Job

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: v1.2
    • Fix Version/s: v1.3.0, v1.4.0
    • Component/s: None
    • Labels:
      None
    • Environment:
      - Tested against apache-kylin-1.2-HBase1.1-incubating-SNAPSHOT-bin and apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
      - Environment is HDP 2.3.4
      - Hive version: hive-1.2.1.2.3.4.0
      - HBase version: HBase 1.1.2.2.3.4.0-3485

      Description

      When running a cube build/refresh and Kylin is unable to retrieve the cardinality of the underlying table (outputfile is empty), it throws a StringIndexOutOfBoundsException.

      output=/tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM

      Unfortunately, this exception is not caught and the frontend states the build ran without problems!

      [2016-02-03 12:45:41,984][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)] - Job Fetcher: 0 running, 2 actual running, 2 ready, 218 others
      [2016-02-03 12:46:01,245][DEBUG][org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:125)] - Job 'Kylin Hive Column Cardinality Job table=PROJECT_KYLIN.DIM_DTM output=/tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM' finished successfully in 18sec
      [2016-02-03 12:46:01,263][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-00 from RUNNING to SUCCEED
      [2016-02-03 12:46:01,276][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from RUNNING to READY
      [2016-02-03 12:46:01,653][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:102)] - DefaultChainedExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8, name=Hive Column Cardinality calculation for table 'PROJECT_KYLIN.DIM_DTM', state=READY} prepare to schedule
      [2016-02-03 12:46:01,653][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)] - DefaultChainedExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8, name=Hive Column Cardinality calculation for table 'PROJECT_KYLIN.DIM_DTM', state=READY} scheduled
      [2016-02-03 12:46:01,663][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from READY to RUNNING
      [2016-02-03 12:46:01,675][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-01 from READY to RUNNING
      [2016-02-03 12:46:01,676][INFO][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:57)] - parameters of the HadoopShellExecutable:
      [2016-02-03 12:46:01,676][INFO][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:58)] - -table PROJECT_KYLIN.DIM_DTM -output /tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM
      [2016-02-03 12:46:01,734][ERROR][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64)] - error execute HadoopShellExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8-01, name=null, state=RUNNING}
      java.lang.StringIndexOutOfBoundsException: String index out of range: -1
      	at java.lang.String.substring(String.java:1911)
      	at org.apache.kylin.job.hadoop.cardinality.HiveColumnCardinalityUpdateJob.updateKylinTableExd(HiveColumnCardinalityUpdateJob.java:113)
      	at org.apache.kylin.job.hadoop.cardinality.HiveColumnCardinalityUpdateJob.run(HiveColumnCardinalityUpdateJob.java:80)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
      	at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
      	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
      	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
      	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      [2016-02-03 12:46:01,750][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-01 from RUNNING to ERROR
      [2016-02-03 12:46:01,765][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from RUNNING to ERROR
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shaofengshi Shao Feng Shi
                Reporter:
                hd2 Sebastian Zimmermann
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: