I would like to point one fine point with the above implementation:
removeOldFilesToControlCumulativeUsage(); // remove very old logs if total usage is alarming irrespective of retain.hours
With the above, retain.hours is no longer a proper contract for tasks, logs can get cleaned-up at arbitrary times. So the 4th requirement is broken -
(4) Clean up of a particular task's user logs should be deterministic - users should have an idea of how long they can see their logs.
The only way I think of in which this can be addressed is by taking care of the cumulative disk-usage of TaskTracker while scheduling a new task. Should this be done too?