[KYLIN-4624] kylin3.1.0使用flink1.10作为engine报错问题

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: v3.1.0
• Component/s: Job Engine
• Environment: centos 7 ; 使用的官网提供的docker镜像，然后我给镜像加了flink ; 跑的的官网的样例数据，spark和mr都ok了 ; flink1.10 ; kylin3.1.0
• Assignee: Unassigned
• Reporter: wangfei
• Created: 08/Jul/20 09:26
• Updated: 31/Jul/20 12:13

Description

在执行到如下步骤出错：
#6 Step Name: Build Cube with Flink
Duration: 0.02 mins Waiting: 0 seconds
报错：
java.lang.NoClassDefFoundError: Could not initialize class org.apache.kylin.engine.flink.FlinkOnYarnConfigMapping
然后我去查找hadoop container的日志
2020-07-08 08:40:34,395 ERROR [main] org.apache.kylin.common.persistence.ResourceStore: Error reading resource /cube/.order_cube_clone.json.crc
org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 start byte 0x8f
at [Source: (DataInputStream); line: 1, column: 10]
 

2020-07-08 08:40:34,482 INFO [main] org.apache.kylin.common.KylinConfig: Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager
2020-07-08 08:40:34,483 INFO [main] org.apache.kylin.metadata.project.ProjectManager: Initializing ProjectManager with metadata url kylin_metadata@ifile,path=/data/hadoop/nm-local-dir/usercache/root/appcache/application_1594187410626_0016/container_1594187410626_0016_01_000007/meta
2020-07-08 08:40:34,504 ERROR [main] org.apache.kylin.common.persistence.ResourceStore: Error reading resource /project/.test_flink.json.crc
org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'crc^@^@@^B@': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (DataInputStream); line: 1, column: 11]

 

 

2020-07-08 08:40:34,512 INFO [main] org.apache.kylin.common.KylinConfig: Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
2020-07-08 08:40:34,524 INFO [main] org.apache.kylin.common.KylinConfig: Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager
2020-07-08 08:40:34,534 INFO [main] org.apache.kylin.common.KylinConfig: Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager
2020-07-08 08:40:34,543 ERROR [main] org.apache.kylin.common.persistence.ResourceStore: Error reading resource /table/.TEST.CUSTOMER--test_flink.json.crc
org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0xe4
at [Source: (DataInputStream); line: 1, column: 11]
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3573)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeCharForError(UTF8StreamJsonParser.java:3318)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3544)

at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2651)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:856)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:753)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4340)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4189)
at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3242)
at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:71)
at org.apache.kylin.common.persistence.JsonSerializer.deserialize(JsonSerializer.java:46)
at org.apache.kylin.common.persistence.ContentReader.readContent(ContentReader.java:40)
at org.apache.kylin.common.persistence.ResourceStore$3.visit(ResourceStore.java:259)
at org.apache.kylin.common.persistence.FileResourceStore.visitFolderImpl(FileResourceStore.java:87)
at org.apache.kylin.common.persistence.ResourceStore.visitFolderInner(ResourceStore.java:766)
at org.apache.kylin.common.persistence.ResourceStore.visitFolderAndContent(ResourceStore.java:751)
at org.apache.kylin.common.persistence.ResourceStore.lambda$getAllResourcesMap$0(ResourceStore.java:255)
at org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at org.apache.kylin.common.persistence.ResourceStore.getAllResourcesMap(ResourceStore.java:253)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
at org.apache.kylin.metadata.TableMetadataManager.initSrcTable(TableMetadataManager.java:126)
at org.apache.kylin.metadata.TableMetadataManager.<init>(TableMetadataManager.java:98)
at org.apache.kylin.metadata.TableMetadataManager.newInstance(TableMetadataManager.java:73)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)