[YARN-9629] Support configurable MIN_LOG_ROLLING_INTERVAL

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0, 3.2.1, 3.1.3
• Component/s: log-aggregation, nodemanager, yarn
• Target Version/s: 3.3.0, 3.2.1, 3.1.3
• Hadoop Flags: Reviewed
• Assignee: Adam Antal
• Reporter: Adam Antal
• Created: 17/Jun/19 08:05
• Updated: 12/Jul/19 15:12
• Resolved: 04/Jul/19 08:29

Description

One of the log-aggregation parameter, the minimum valid value for yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds is MIN_LOG_ROLLING_INTERVAL - it has been hardcoded since its addition in YARN-2583.

It has been empirically set as 1 hour, as lower values would too frequently put the NodeManagers under pressure. For bigger clusters that is indeed a valid limitation, but for smaller clusters it makes sense and a valid customer usecase to use lower values, even like not so lower 30 mins. At this point this can only be achieved by setting yarn.nodemanager.log-aggregation.debug-enabled, which I believe should be kept as debug purposes.

I'm suggesting to make this min configurable, although a warning should be logged in the NodeManager startup when this value is lower than 1 hour.