Details
-
Improvement
-
Status: Reopened
-
Major
-
Resolution: Unresolved
-
2.0.0
-
None
-
None
Description
One of our local dev Hadoop devs noticed our storm user was by far creating the heaviest load on our production Hadoop cluster. Looking at one of the heaviest supervisor nodes, and comparing debug logs to the Hadoop audit log, it looks like LocalizedResourceRetentionSet cleanup was constantly doing opens and never deleting any files.
The frequency can be addressed by supervisor.localizer.cleanup.interval.ms, but even so, it seems we will continually look for files to delete even when the target size is acceptable, resulting in unnecessary calls to Hadoop.
