[HIVE-6314] The logging (progress reporting) is too verbose

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: logger
• Assignee: Navis Ryu
• Reporter: Sam Steingold
• Created: 27/Jan/14 16:14
• Updated: 06/Jan/15 17:46
• Resolved: 28/Mar/14 20:36

Description

The progress report is issued every second even when no progress have been made:

2014-01-27 10:35:55,209 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.68 sec
2014-01-27 10:35:56,678 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.68 sec
2014-01-27 10:35:59,344 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 6.68 sec
2014-01-27 10:36:01,268 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.67 sec
2014-01-27 10:36:03,149 Stage-1 map = 100%,  reduce = 100%, Cumulative CPU 8.67 sec

This pollutes the logs and the screen, and people do not appreciate it as much as the designers might have thought (How do I limit log verbosity of hive?, controlling the level of verbosity in Hive).
It would be nice to be able to control the level of verbosity (but not by the -v switch!):

1. Make sure that the progress report is only issued where there is something new to report; or
2. Remove all the progress messages; or
3. Make sure that progress is reported only every X sec (instead of every 1 second)