[HIVE-28051] LLAP: cleanup local folders on startup and periodically - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.0.0
Component/s: None
Labels:
- pull-request-available

Description

When LLAP daemon crashes, there might be leftovers in the hive llap local dirs, which are good candidates to be cleaned up on startup/regularly. Under normal circumstances, these are cleaned up, however, as the LLAP damons is responsible for that, it cannot help in the event of a crash, so it would make sense to make it able to clean up old files.

A file listing of a problematic cluster where LLAP local dir is configured to /apps/llap/work:

du -sh /apps/llap/work/usercache/hive/appcache/*
261M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0005
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0043
28K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0049
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0052
2.0M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0060
5.9G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0069
1.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0072
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0083
1.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0085
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0087
356M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0089
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0091
429M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0095
48K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0096
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0097
166M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0099
1.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0100
369M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0101
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0102
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0103
2.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0104
16K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0105
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0106
592K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0107
2.9G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0108
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0109
60K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0110
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0111
1.6G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0112
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0113
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0114
11G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0115
3.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0116
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0117
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0118
646M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0119
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0120
8.0K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0121
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0122
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0123
164K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0124
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0125
115M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0126
406M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0127
988K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0128
8.0K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0129
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0130
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0131
94M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0132
1.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0133
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0134
1.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0135
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0136
771M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0137
2.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0138
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0139
2.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0140
1.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0141
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0142
2.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0143
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0144
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0145
13G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0146
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0147
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0148
1.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0149
3.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0150
12G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0151
6.6G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0152
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0153
2.7M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0154
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0155
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0156
27G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0157
11G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0158
1.8M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0159
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0160
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0161
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0162
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0163
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0164
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0165
1.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0166
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0167
2.1M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0168
1.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0169
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0170
289M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0171
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0172
18M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0173
5.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0174
8.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0175
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0176
13G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0177
6.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0178
7.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0179
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0180
28K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0181
976M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0182
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0183
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0184
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0185
65M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0186
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0187
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0188
940K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0189
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0190
2.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0191
102M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0192
4.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0193
654M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0194
3.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0195
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0196
830M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0197
1.2M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0198
332M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0199
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0200
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0201
5.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0202
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0203
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0204
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0205
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0206
8.0K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0207
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0208
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0209
5.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0210
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0211
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0212
3.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0213
43G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0214
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0215
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0216
3.4M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0217
5.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0218
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0219
37M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0220
1.8M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0221
665M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0222
4.6G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0223
1.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0224
1.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0225
5.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0226
12K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0227
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0228
106M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0229
1.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0230
16G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0231
1.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0232
940K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0233
4.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0234
41M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0235
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0236
6.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0237
9.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0238
6.8G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0239
1.2M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0240
2.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0241
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0242
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0243
20K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0244
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0245
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0246
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0247
431M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0248
3.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0249
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0250
1.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0251
15G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0252
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0253
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0254
100K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0255
20K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0256
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0257
2.7G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0258
987M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0259
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0260
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0261
672M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0262
8.0K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0263
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0264
7.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0265
1.6G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0266
5.4M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0267
5.6G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0268
1000M   /apps/llap/work/usercache/hive/appcache/application_1702138790934_0269
13G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0270
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0271
169M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0272
3.0M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0273
5.7G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0274
628K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0275
2.0G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0276
2.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0277
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0278
12K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0279
95M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0280
5.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0281
32K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0282
2.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0283
244M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0284
4.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0285
3.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0286
5.7G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0287
1.3G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0288
15G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0289
848M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0290
19M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0291
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0292
38G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0293
326M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0294
2.2G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0295
4.1G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0296
38G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0297
444K    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0298
28K     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0299
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0300
241M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0301
45G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0302
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0303
36M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0304
679M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0305
27G     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0306
6.4G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0307
23M     /apps/llap/work/usercache/hive/appcache/application_1702138790934_0308
490M    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0309
0       /apps/llap/work/usercache/hive/appcache/application_1702138790934_0310
1.5G    /apps/llap/work/usercache/hive/appcache/application_1702138790934_0311

regarding dates:

ls -lrth /apps/llap/work/usercache/hive/appcache/
total 4.0K
drwxr-xr-x  2 hive hive    6 Dec 29 11:17 application_1702138790934_0102
drwxr-xr-x  3 hive hive   17 Jan  1 01:37 application_1702138790934_0107
drwxr-xr-x  3 hive hive   17 Jan  1 09:04 application_1702138790934_0100
drwxr-xr-x  2 hive hive    6 Jan  2 05:13 application_1702138790934_0087
drwxr-xr-x  3 hive hive   17 Jan  2 05:14 application_1702138790934_0101
drwxr-xr-x  3 hive hive   16 Jan  2 12:31 application_1702138790934_0109
drwxr-xr-x  2 hive hive    6 Jan  2 12:50 application_1702138790934_0106
drwxr-xr-x  3 hive hive   18 Jan  2 19:58 application_1702138790934_0043
drwxr-xr-x  3 hive hive   17 Jan  2 20:05 application_1702138790934_0089
drwxr-xr-x  4 hive hive   30 Jan  2 21:45 application_1702138790934_0072
drwxr-xr-x  3 hive hive   16 Jan  2 21:56 application_1702138790934_0115
drwxr-xr-x  3 hive hive   17 Jan  3 08:48 application_1702138790934_0096
drwxr-xr-x  3 hive hive   17 Jan  3 08:56 application_1702138790934_0105
drwxr-xr-x  3 hive hive   17 Jan  3 09:05 application_1702138790934_0060
drwxr-xr-x  2 hive hive    6 Jan  3 09:08 application_1702138790934_0114
drwxr-xr-x  2 hive hive    6 Jan  3 09:08 application_1702138790934_0113
drwxr-xr-x  2 hive hive    6 Jan  3 09:28 application_1702138790934_0118
drwxr-xr-x  2 hive hive    6 Jan  3 09:40 application_1702138790934_0117
drwxr-xr-x  3 hive hive   17 Jan  3 11:51 application_1702138790934_0110
drwxr-xr-x  2 hive hive    6 Jan  3 12:03 application_1702138790934_0091
drwxr-xr-x  2 hive hive    6 Jan  3 12:05 application_1702138790934_0122
drwxr-xr-x  3 hive hive   17 Jan  3 12:42 application_1702138790934_0095

so e.g. application_1702138790934_0069 has ~6G stale data from Jan3 (file listing was made at the end of January)

these local files are typically the ones below (assuming that llap local dirs point to /apps/llap/work)

/apps/llap/work/usercache/hive/appcache/application_1707917402901_0001/3/output/attempt_1707917402901_0001_3_05_000002_0_10414:
total 16
drwxr-xr-x  2 hive hive   44 Feb 14 13:41 .
drwxr-xr-x 63 hive hive 4096 Feb 14 13:41 ..
-rw-r--r--  1 hive hive  425 Feb 14 13:41 file.out
-rw-r--r--  1 hive hive   32 Feb 14 13:41 file.out.index

/apps/llap/work/usercache/hive/appcache/application_1707917402901_0001/3/output/attempt_1707917402901_0001_3_05_000005_0_10416:
total 16
drwxr-xr-x  2 hive hive   44 Feb 14 13:41 .
drwxr-xr-x 63 hive hive 4096 Feb 14 13:41 ..
-rw-r--r--  1 hive hive  383 Feb 14 13:41 file.out
-rw-r--r--  1 hive hive   32 Feb 14 13:41 file.out.index

Attachments

Issue Links

is related to

HIVE-24272 LLAP: local directories should be cleaned up on ContainerRunnerImpl.queryFailed

Open

links to

GitHub Pull Request #5054

Activity

People

Assignee:: László Bodor

Reporter:: László Bodor

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 31/Jan/24 14:33

Updated:: 29/Mar/24 17:49

Resolved:: 08/Mar/24 08:32