[SPARK-15158] Too aggressive logging in SizeBasedRollingPolicy?

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 1.6.1
• Fix Version/s: 2.0.0
• Component/s: Spark Core
• Target Version/s: 2.0.0
• Assignee: Kai Wang
• Reporter: Kai Wang
• Created: 05/May/16 13:05
• Updated: 05/May/16 21:41
• Resolved: 05/May/16 21:39

Description

The questionable line is this: https://github.com/apache/spark/blob/3e27940a19e7bab448f1af11d2065ecd1ec66197/core/src/main/scala/org/apache/spark/util/logging/RollingPolicy.scala#L116

This will output a message whenever anything is logged at executor level. Like the following:
SizeBasedRollingPolicy:59 83 + 140796 > 1048576
SizeBasedRollingPolicy:59 83 + 140879 > 1048576
SizeBasedRollingPolicy:59 83 + 140962 > 1048576
...

This seems to aggressive. Should this be at least downgrade to debug level?

Issue Links

• links to: [Github] Pull Request #12931 (depend)