[KYLIN-998] Finish the hive intermediate table clean up job in org.apache.kylin.job.hadoop.cube.StorageCleanupJob

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v0.7.1, v0.7.2
• Fix Version/s: v1.1
• Component/s: Storage - HBase
• Assignee: nichunen
• Reporter: nichunen
• Created: 05/Sep/15 13:31
• Updated: 17/Oct/18 01:18
• Resolved: 17/Oct/18 01:18

Description

Current kylin has its last cube building job step named “Garbage Collection” to remove the intermediate data in hdfs/hbase/hive. But if the job is accidentally stopped like problem in hadoop cluster, bad cube design, discarded by user, the data was left un-deleted.

In such cases, we can run "hbase org.apache.hadoop.util.RunJar $KYLIN_HOME/lib/kylin-job-0.8.1-incubating-SNAPSHOT.jar org.apache.kylin.job.hadoop.cube.StorageCleanupJob --delete true" to remove the data. But the method "cleanUnusedIntermediateHiveTable" is unfinished.

My first patch is to finish the method, it will remove unused hive tables with names begin with "kylin_intermediate_".

My second patch add some methods to enable deleting unused data with uuids in command line, or stored in a file.

I don't know whether the second patch is useful to you, it's used in our kylin server to remove data after one cube is deleted.