Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4002

Cleanup job does not delete Hive intermediate VIEW tables

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: v2.6.1
    • Fix Version/s: None
    • Component/s: Job Engine
    • Labels:
      None

      Description

      Kylin org.apache.kylin.tool.StorageCleanupJob does not delete hive intermediate tables if they are view based.

      After several weeks running Kylin 2.6.1 after a successful upgrade from 2.4.1, we have a lot of temporary tables that have not been deleted. All of them come from a VIEW.

      However, those tables built directly from a table and not from a view, are properly deleted.

      Tables from view not deleted: 263079

      Tables not form view not deleted: 0

       

      At log level, we can see:

      2019-05-14 08:10:00,293 DEBUG [main] job.StorageCleanupJob:326 : Checking if table is garbage -- kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986
      2019-05-14 10:17:53,235 INFO [main] job.StorageCleanupJob:407 : Deleting Hive table kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986
      drop table if exists kylin_intermediate_dw_di_paises_view_238e3004_13e8_2be6_5f0e_df545b481986;

      So, it's properly detected and drop sentence properly created. But it's not executed and table is still there.

      We have launched the 263079 drop sentences via script and all are being removed.  

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              inigoml Iñigo Martinez
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: