[KYLIN-5179] Kylin4 can not delete invalid segments on HDFS by StorageCleanupJob

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v4.0.1
• Fix Version/s: v4.0.1
• Component/s: Storage - Parquet
• Assignee: mukvin
• Reporter: Sunbuck
• Created: 22/Apr/22 08:16
• Updated: 09/Nov/22 04:40
• Resolved: 29/Apr/22 03:06

Description

在测试环境中通过KylinWeb中对cube进行数据构建后，想通过kylin.sh org.apache.kylin.tool.StorageCleanupJob进行失效segment进行清理，目前有效的cube为0RI的parquet storage：

在hdfs中查看该cube的segment情况：

其中MV6和C54的segment是我已经在Web中进行delete segment操作后的遗留垃圾数据，在按照Kylin Wiki（https://cwiki.apache.org/confluence/display/KYLIN/How+to+clean+up+storage+in+Kylin+4）中的说明中执行 kylin.sh org.apache.kylin.tool.StorageCleanupJob时，报错如下：

 

2022-04-22 16:26:13,328 INFO  [main] persistence.JDBCResourceStore:115 : Table [kylin_metadata] already exists
2022-04-22 16:26:13,332 INFO  [main] persistence.JDBCResourceStore:115 : Table [kylin_metadata_log] already exists
2022-04-22 16:26:13,561 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster
Running org.apache.kylin.rest.job.StorageCleanupJob
2022-04-22 16:26:13,571 INFO  [main] job.StorageCleanupJob:143 : options: ''
2022-04-22 16:26:13,571 INFO  [main] job.StorageCleanupJob:160 : ===================================================================
delete : false; cleanupTableSnapshot : true; cleanupGlobalDict : true; cleanupJobTmp : false; cleanBeforeDate : Fri Apr 15 16:26:13 CST 2022.
2022-04-22 16:26:13,574 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager
2022-04-22 16:26:13,576 INFO  [main] project.ProjectManager:81 : Initializing ProjectManager with metadata url kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
2022-04-22 16:26:13,602 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.cube.CubeManager
2022-04-22 16:26:13,608 INFO  [main] cube.CubeManager:122 : Initializing CubeManager with config kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
2022-04-22 16:26:13,651 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager
2022-04-22 16:26:13,652 INFO  [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config kylin_metadata@jdbc,url=jdbc:mysql://10.62.0.245:3306/kylin4,username=xxx,password=xxx,maxActive=10,maxIdle=10
2022-04-22 16:26:13,705 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager
2022-04-22 16:26:13,709 INFO  [main] common.KylinConfig:493 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager
2022-04-22 16:26:13,866 INFO  [main] job.StorageCleanupJob:174 : Start to clean up unreferenced projects and cubes ...
2022-04-22 16:26:13,867 INFO  [main] job.StorageCleanupJob:200 : Start to clean up no unreferenced segments ...
2022-04-22 16:26:13,867 WARN  [main] job.StorageCleanupJob:222 : Cube path doesn't exist! The path is file:/kylin4/kylin_metadata/storage/parquet/mobile_core_analysis_cube
2022-04-22 16:26:13,868 WARN  [main] job.StorageCleanupJob:222 : Cube path doesn't exist! The path is file:/kylin4/kylin_metadata/storage/parquet/storage_cube