[KYLIN-3156] Failed to delete meta path in SparkCubingByLayer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Spark Engine
• Assignee: Yifei Wu
• Reporter: Shaohui Liu
• Created: 08/Jan/18 05:13
• Updated: 30/May/18 10:19
• Resolved: 30/May/18 10:19

Description

After KYLIN-2945, the meta url in SparkCubingByLayer will be a string of StorageURL not string with path@hdfs format. This will make the deleteHDFSMeta method failed in SparkCubingByLayer.

2018-01-08,11:51:50,903 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer
java.lang.RuntimeException: error execute org.apache.kylin.engine.spark.SparkCubingByLayer
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:653)
Caused by: java.lang.IllegalArgumentException: Cannot create FileSystem from URI: kylin_tst:kylin_metadata
at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:98)
at org.apache.kylin.common.util.HadoopUtil.getFileSystem(HadoopUtil.java:78)
at org.apache.kylin.engine.spark.SparkCubingByLayer.deleteHDFSMeta(SparkCubingByLayer.java:484)
at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:207)
at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
... 6 more
Caused by: java.net.URISyntaxException: Illegal character in scheme name at index 5: kylin_tst:kylin_metadata
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parse(URI.java:3048)
at java.net.URI.<init>(URI.java:588)
at org.apache.kylin.common.util.HadoopUtil.makeURI(HadoopUtil.java:96)
... 10 more

Issue Links

• duplicates: KYLIN-3368 "/kylin/kylin_metadata/metadata/" has many garbage for spark cubing