[KYLIN-4002] Cleanup job does not delete Hive intermediate VIEW tables

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: v2.6.1
• Component/s: Job Engine
• Assignee: Unassigned
• Reporter: Iñigo Martinez
• Created: 14/May/19 09:53
• Updated: 29/Jul/19 02:34
• Resolved: 29/Jul/19 02:34

Description

Kylin org.apache.kylin.tool.StorageCleanupJob does not delete hive intermediate tables if they are view based.

After several weeks running Kylin 2.6.1 after a successful upgrade from 2.4.1, we have a lot of temporary tables that have not been deleted. All of them come from a VIEW.

However, those tables built directly from a table and not from a view, are properly deleted.

Tables from view not deleted: 263079

Tables not form view not deleted: 0

 

At log level, we can see:

2019-05-14 08:10:00,293 DEBUG [main] job.StorageCleanupJob:326 : Checking if table is garbage -- kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986
2019-05-14 10:17:53,235 INFO [main] job.StorageCleanupJob:407 : Deleting Hive table kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986
drop table if exists kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986;

So, it's properly detected and drop sentence properly created. But it's not executed and table is still there.

We have launched the 263079 drop sentences via script and all are being removed.