[KYLIN-1402] StringIndexOutOfBoundsException in Kylin Hive Column Cardinality Job

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v1.2
• Fix Version/s: v1.3.0, v1.4.0
• Environment: - Tested against apache-kylin-1.2-HBase1.1-incubating-SNAPSHOT-bin and apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin; - Environment is HDP 2.3.4 ; - Hive version: hive-1.2.1.2.3.4.0; - HBase version: HBase 1.1.2.2.3.4.0-3485
• Assignee: Shao Feng Shi
• Reporter: Sebastian Zimmermann
• Created: 04/Feb/16 09:28
• Updated: 14/Mar/16 01:55
• Resolved: 07/Mar/16 03:30

Description

When running a cube build/refresh and Kylin is unable to retrieve the cardinality of the underlying table (outputfile is empty), it throws a StringIndexOutOfBoundsException.

output=/tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM

Unfortunately, this exception is not caught and the frontend states the build ran without problems!

[2016-02-03 12:45:41,984][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)] - Job Fetcher: 0 running, 2 actual running, 2 ready, 218 others
[2016-02-03 12:46:01,245][DEBUG][org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:125)] - Job 'Kylin Hive Column Cardinality Job table=PROJECT_KYLIN.DIM_DTM output=/tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM' finished successfully in 18sec
[2016-02-03 12:46:01,263][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-00 from RUNNING to SUCCEED
[2016-02-03 12:46:01,276][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from RUNNING to READY
[2016-02-03 12:46:01,653][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:102)] - DefaultChainedExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8, name=Hive Column Cardinality calculation for table 'PROJECT_KYLIN.DIM_DTM', state=READY} prepare to schedule
[2016-02-03 12:46:01,653][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:106)] - DefaultChainedExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8, name=Hive Column Cardinality calculation for table 'PROJECT_KYLIN.DIM_DTM', state=READY} scheduled
[2016-02-03 12:46:01,663][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from READY to RUNNING
[2016-02-03 12:46:01,675][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-01 from READY to RUNNING
[2016-02-03 12:46:01,676][INFO][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:57)] - parameters of the HadoopShellExecutable:
[2016-02-03 12:46:01,676][INFO][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:58)] - -table PROJECT_KYLIN.DIM_DTM -output /tmp/kylin/cardinality/PROJECT_KYLIN.DIM_DTM
[2016-02-03 12:46:01,734][ERROR][org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:64)] - error execute HadoopShellExecutable{id=b2944302-a2e3-4847-b672-7bcd9489d0f8-01, name=null, state=RUNNING}
java.lang.StringIndexOutOfBoundsException: String index out of range: -1
	at java.lang.String.substring(String.java:1911)
	at org.apache.kylin.job.hadoop.cardinality.HiveColumnCardinalityUpdateJob.updateKylinTableExd(HiveColumnCardinalityUpdateJob.java:113)
	at org.apache.kylin.job.hadoop.cardinality.HiveColumnCardinalityUpdateJob.run(HiveColumnCardinalityUpdateJob.java:80)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
	at org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
	at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
	at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
	at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
[2016-02-03 12:46:01,750][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8-01 from RUNNING to ERROR
[2016-02-03 12:46:01,765][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] - job id:b2944302-a2e3-4847-b672-7bcd9489d0f8 from RUNNING to ERROR

Issue Links

• is related to: KYLIN-1403 Kylin Hive Column Cardinality Job unable to read bucketed table