[KYLIN-4547] parse crc file error when building cube with mapreduce

• Type: Bug
• Status: In Review
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: v3.0.0
• Component/s: Metadata
• Environment: hadoop version: CDH-5.12.2-1.cdh5.12.2
• Assignee: qinrui
• Reporter: qinrui
• Created: 08/Jun/20 10:34
• Updated: 03/Mar/22 14:27

Description

It can not parse crc file when building cube with mapreduce.

Here is the exeception log:

// code placeholder
2020-06-08 10:08:23,821 INFO [main] org.apache.kylin.common.KylinConfig: Creating new manager instance of class org.apache.kylin.cube.CubeManager
2020-06-08 10:08:23,844 INFO [main] org.apache.kylin.cube.CubeManager: Initializing CubeManager with config kylin_metadata30@ifile,path=/yarn/nm/usercache/kylin/appcache/application_1590134125851_1782/container_e55_1590134125851_1782_01_000003/meta
2020-06-08 10:08:23,847 INFO [main] org.apache.kylin.common.persistence.ResourceStore: Using metadata url kylin_metadata30@ifile,path=/yarn/nm/usercache/kylin/appcache/application_1590134125851_1782/container_e55_1590134125851_1782_01_000003/meta for resource store
2020-06-08 10:08:24,303 ERROR [main] org.apache.kylin.common.persistence.ResourceStore: Error reading resource /cube/.kylin_sales_cube.json.crc
org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0xd2
 at [Source: (DataInputStream); line: 1, column: 11]
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1804)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:663)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3543)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeCharForError(UTF8StreamJsonParser.java:3288)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3514)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._handleUnexpectedValue(UTF8StreamJsonParser.java:2621)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:826)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:723)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:4129)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3988)
	at org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3058)
	at org.apache.kylin.common.util.JsonUtil.readValue(JsonUtil.java:73)
	at org.apache.kylin.common.persistence.JsonSerializer.deserialize(JsonSerializer.java:46)
	at org.apache.kylin.common.persistence.ContentReader.readContent(ContentReader.java:40)
	at org.apache.kylin.common.persistence.ResourceStore$3.visit(ResourceStore.java:259)
	at org.apache.kylin.common.persistence.FileResourceStore.visitFolderImpl(FileResourceStore.java:87)
	at org.apache.kylin.common.persistence.ResourceStore.visitFolderInner(ResourceStore.java:766)
	at org.apache.kylin.common.persistence.ResourceStore.visitFolderAndContent(ResourceStore.java:751)
	at org.apache.kylin.common.persistence.ResourceStore.lambda$getAllResourcesMap$0(ResourceStore.java:255)
	at org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
	at org.apache.kylin.common.persistence.ResourceStore.getAllResourcesMap(ResourceStore.java:253)
	at org.apache.kylin.metadata.cachesync.CachedCrudAssist.reloadAll(CachedCrudAssist.java:127)
	at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:152)
	at org.apache.kylin.cube.CubeManager.newInstance(CubeManager.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.kylin.common.KylinConfig.getManager(KylinConfig.java:478)
	at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:104)
	at org.apache.kylin.engine.mr.steps.FactDistinctColumnPartitioner.setConf(FactDistinctColumnPartitioner.java:56)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:707)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:776)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Issue Links

• is duplicated by: KYLIN-4546 parse crc file error when building cube with mapreduce
• links to: PR #1245