[KYLIN-5177] Kylin4.0.1使用S3存储无法清理数据

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v4.0.1
• Fix Version/s: v4.0.2
• Component/s: Storage - Parquet
• Environment: aws emr5.33.1
• Assignee: mukvin
• Reporter: peilun.zheng
• Created: 12/Apr/22 07:54
• Updated: 31/Aug/22 11:55
• Resolved: 29/Apr/22 03:09

Description

kylin4.0.1在aws emr5.33.1版本上使用bin/kylin.sh org.apache.kylin.tool.StorageCleanupJob

无法清理S3上的存储

log:

Retrieving hadoop conf dir...
Retrieving Spark dependency...
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/ext/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/kylin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2022-04-12 07:35:09,765 INFO  [main] common.KylinConfig:117 : Loading kylin-defaults.properties from file:/home/hadoop/kylin/tool/kylin-tool-4.0.1.jar!/kylin-defaults.properties
2022-04-12 07:35:09,776 INFO  [main] common.KylinConfig:369 : Use KYLIN_HOME=/home/hadoop/kylin
2022-04-12 07:35:09,819 INFO  [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.metadata.url.identifier : kylin_instance
2022-04-12 07:35:09,822 INFO  [main] common.KylinConfigBase:263 : Kylin Config was updated with kylin.log.spark-executor-properties-file : /home/hadoop/kylin/conf/spark-executor-log4j.properties
2022-04-12 07:35:09,822 INFO  [main] common.KylinConfig:160 : Initialized a new KylinConfig from getInstanceFromEnv : 1845904670
2022-04-12 07:35:10,011 WARN  [main] util.NativeCodeLoader:62 : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
    at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:313)
    at org.apache.kylin.common.util.HadoopUtil.getWorkingFileSystem(HadoopUtil.java:100)
    at org.apache.kylin.rest.job.StorageCleanupJob.<init>(StorageCleanupJob.java:121)
    at org.apache.kylin.tool.StorageCleanupJob.main(StorageCleanupJob.java:26)
Caused by: org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "s3"
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3273)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3293)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:120)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3344)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3312)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:478)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
    at org.apache.kylin.common.KylinConfigBase.getHdfsWorkingDirectory(KylinConfigBase.java:310)
    ... 3 more