[KYLIN-4341] by-level cuboid intermediate files are left behind and not cleaned up after job is complete

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: v2.6.4
• Component/s: Job Engine
• Environment: Kylin 2.6.4, CenOS 7.6, HDP 2.6.5
• Assignee: Wang Rupeng
• Reporter: Vsevolod Ostapenko
• Created: 15/Jan/20 22:23
• Updated: 02/Mar/20 16:16

Description

Setup: MR as a cube build engine and by-level cube build strategy (auto picked).
Upon completion of a cube segment build job a number of intermediate files are still left behind.
Namely, output of the MR-jobs that produce the base cuboid, subsequent level cuboids, as well as rowkey_stats from the hfile creation step.
The files in question consume about the same amount of space in HDFS as the final hfile.
This lead to wasted space in HDFS that is not released for as long as the corresponding cube segment is online. The only point the leaked space is released, is when segment is offlined and cleaned up as part of the segment retention.

Sample output is as follows.

$ hadoop fs -ls -R /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/
drwxr-xr-x - kylin hdfs 0 2020-01-07 04:44 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89
drwxr-xr-x - kylin hdfs 0 2020-01-07 04:26 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid
drwxr-xr-x - kylin hdfs 0 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid
rw-rr- 2 kylin hdfs 0 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/_SUCCESS
rw-rr- 2 kylin hdfs 51570048 2020-01-07 04:35 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00000
rw-rr- 2 kylin hdfs 51477377 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00001
rw-rr- 2 kylin hdfs 51615162 2020-01-07 04:35 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00002
rw-rr- 2 kylin hdfs 51591031 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00003
rw-rr- 2 kylin hdfs 51648914 2020-01-07 04:35 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00004
rw-rr- 2 kylin hdfs 51532761 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00005
rw-rr- 2 kylin hdfs 51455652 2020-01-07 04:35 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00006
rw-rr- 2 kylin hdfs 51552752 2020-01-07 04:36 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_1_cuboid/part-r-00007
drwxr-xr-x - kylin hdfs 0 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid
rw-rr- 2 kylin hdfs 0 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/_SUCCESS
rw-rr- 2 kylin hdfs 16293012 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-00000
rw-rr- 2 kylin hdfs 16283730 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-00001
rw-rr- 2 kylin hdfs 16288965 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-00002
rw-rr- 2 kylin hdfs 16270572 2020-01-07 04:25 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/cuboid/level_base_cuboid/part-r-00003
drwxr-xr-x - kylin hdfs 0 2020-01-07 04:23 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/rowkey_stats
rw-rr- 3 kylin hdfs 155 2020-01-07 04:23 /user/kylin/26x/test_kylin_26x_metadata/kylin-d2cff21e-9f59-6ac8-df34-656a3479433f/vl_aggs_cube_89/rowkey_stats/part-r-00000_hfile

 

Removing the job metadata using (metastore.sh clean --jobThreshold Ndays) does not help. Information about the job is removed, but no intermediate files are cleaned up.

Storage cleanup does not work either (kylin.sh org.apache.kylin.tool.StorageCleanupJob --delete true), because the corresponding segment is still online.

 

It looks like cleanup of the intermediate files that generated by MR job as part of the base and level cuboid builds is either not implemented or commented out for whatever reason.