[MESOS-440] Allow for headroom in the GC algorithm.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Vinod Kone
• Reporter: Benjamin Mahler
• Created: 19/Apr/13 19:37
• Updated: 16/May/13 00:19
• Resolved: 16/May/13 00:19

Description

On a heavily loaded slave, we don't GC aggressively enough to ensure we keep some headroom.

This was discovered by Vinod:

This slave started ~8hours ago, the disk is currently hovering at 95% => max allowed age of ~8 hours.

I0418 22:59:24.970710 24127 slave.cpp:2480] Current disk usage 94.75%. Max allowed age: 8.811656935171770hrs
I0418 23:00:24.973837 24121 slave.cpp:2480] Current disk usage 94.77%. Max allowed age: 8.781359149313847hrs
I0418 23:01:24.976848 24120 slave.cpp:2480] Current disk usage 94.79%. Max allowed age: 8.745520512138819hrs
I0418 23:02:24.978328 24122 slave.cpp:2480] Current disk usage 94.81%. Max allowed age: 8.714769072645744hrs
I0418 23:03:24.980588 24131 slave.cpp:2480] Current disk usage 94.83%. Max allowed age: 8.684486979209147hrs
I0418 23:04:24.983655 24122 slave.cpp:2480] Current disk usage 94.85%. Max allowed age: 8.654217725026378hrs
I0418 23:05:24.985623 24127 slave.cpp:2480] Current disk usage 94.87%. Max allowed age: 8.617881210119714hrs
I0418 23:06:24.987745 24120 slave.cpp:2480] Current disk usage 94.89%. Max allowed age: 8.581586066142052hrs
I0418 23:07:24.991423 24132 slave.cpp:2480] Current disk usage 94.91%. Max allowed age: 8.551309679040479hrs
I0418 23:08:24.993568 24129 slave.cpp:2480] Current disk usage 94.93%. Max allowed age: 8.521038998273946hrs
I0418 23:09:24.995787 24129 slave.cpp:2480] Current disk usage 94.95%. Max allowed age: 8.484731015042456hrs
I0418 23:10:24.998302 24123 slave.cpp:2480] Current disk usage 94.97%. Max allowed age: 8.448363115293104hrs
I0418 23:11:25.002799 24124 slave.cpp:2480] Current disk usage 94.99%. Max allowed age: 8.418062476267657hrs
I0418 23:12:25.019284 24117 slave.cpp:2480] Current disk usage 95.01%. Max allowed age: 8.388263994725111hrs
I0418 23:13:25.071542 24118 slave.cpp:2480] Current disk usage 95.03%. Max allowed age: 8.357983327872256hrs
I0418 23:14:25.075816 24127 slave.cpp:2480] Current disk usage 95.04%. Max allowed age: 8.327712647105725hrs
I0418 23:15:25.084130 24120 slave.cpp:2480] Current disk usage 95.06%. Max allowed age: 8.297423420750345hrs
I0418 23:16:25.084964 24132 slave.cpp:2480] Current disk usage 95.08%. Max allowed age: 8.266706219267464hrs
I0418 23:17:25.089365 24119 slave.cpp:2480] Current disk usage 95.10%. Max allowed age: 8.236432685333412hrs
I0418 23:18:25.100533 24117 slave.cpp:2480] Current disk usage 95.12%. Max allowed age: 8.206157724815618hrs
I0418 23:19:25.647794 24132 slave.cpp:2480] Current disk usage 95.14%. Max allowed age: 8.172391913841343hrs
I0418 23:20:26.041347 24132 slave.cpp:2480] Current disk usage 95.15%. Max allowed age: 8.142340926973567hrs
I0418 23:21:26.042716 24122 slave.cpp:2480] Current disk usage 95.17%. Max allowed age: 8.106048636163425hrs

I0418 23:22:26.048478 24125 gc.cpp:167] Pruning directories with remaining removal time 6.663156089446059days
I0418 23:22:26.543171 24125 gc.cpp:167] Pruning directories with remaining removal time 6.663150365405061days
I0418 23:22:26.550261 24125 gc.cpp:167] Pruning directories with remaining removal time 6.663150285115397days
I0418 23:22:26.676244 24125 gc.cpp:134] Deleting /var/lib/mesos/slaves/201303281614-1937777162-5050-34776-36