[KYLIN-2651] Couldn't find global dict index file

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Assignee: Unassigned
• Reporter: Shao Feng Shi
• Created: 31/May/17 00:56
• Updated: 31/May/17 12:04
• Resolved: 31/May/17 11:01

Description

Today I got an error when run a tes cube with Spark engine:

17/05/31 08:50:28 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, sandbox): java.lang.RuntimeException: java.io.FileNotFoundException: File does not exist: /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/.index
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:652)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)

	at org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:129)
	at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:264)
	at org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:329)
	at org.apache.kylin.cube.CubeSegment.buildDictionaryMap(CubeSegment.java:321)
	at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:205)
	at org.apache.kylin.engine.spark.SparkCubingByLayer$2.call(SparkCubingByLayer.java:193)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
	at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1018)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
	at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:191)
	at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:64)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
	at org.apache.spark.scheduler.Task.run(Task.scala:89)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Then I checked the file system, there is only the ".index_v2" file:

[root@sandbox ~]# hadoop fs -ls /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/
Found 2 items
-rw-r--r--   3 shishaofeng hdfs         93 2017-05-31 08:49 /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/.index_v2
-rw-r--r--   3 shishaofeng hdfs       8036 2017-05-31 08:49 /kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/version_1496191754004/cached_1496191753923_1

Here is the metadata of the dictionary:

[root@sandbox DEFAULT.TEST_KYLIN_FACT]# cd TEST_COUNT_DISTINCT_BITMAP
[root@sandbox TEST_COUNT_DISTINCT_BITMAP]# ls
c74d7586-b085-4b12-9f03-9f92a882f8f6.dict
[root@sandbox TEST_COUNT_DISTINCT_BITMAP]# cat c74d7586-b085-4b12-9f03-9f92a882f8f6.dict
`{
  "uuid" : "c74d7586-b085-4b12-9f03-9f92a882f8f6",
  "last_modified" : 0,
  "version" : "2.1.0",
  "source_table" : "DEFAULT.TEST_KYLIN_FACT",
  "source_column" : "TEST_COUNT_DISTINCT_BITMAP",
  "source_column_index" : 10,
  "data_type" : "varchar(256)",
  "input" : {
    "path" : "hdfs:///kylin/kylin_metadata/kylin-c62f0db4-e669-4f34-9337-8a978156fe82/ci_inner_join_cube_clone/fact_distinct_columns/TEST_KYLIN_FACT.TEST_COUNT_DISTINCT_BITMAP",
    "size" : 16268,
    "last_modified_time" : 1496220553650
  },
  "dictionary_class" : "org.apache.kylin.dict.AppendTrieDictionary",
  "cardinality" : 1001
}jhdfs:///kylin/kylin_metadata/resources/GlobalDict/dict/DEFAULT.TEST_KYLIN_FACT/TEST_COUNT_DISTINCT_BITMAP/