[HIVE-20168] ReduceSinkOperator Logging Hidden

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0, 4.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: Operators
• Labels: newbie noob
• Assignee: Bharath Krishna
• Reporter: David Mollitor
• Created: 13/Jul/18 14:42
• Updated: 17/Nov/22 08:54
• Resolved: 30/Jul/18 17:47

Description

https://github.com/apache/hive/blob/ac6b2a3fb195916e22b2e5f465add2ffbcdc7430/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java

 

if (LOG.isTraceEnabled()) {
  if (numRows == cntr) {
    cntr = logEveryNRows == 0 ? cntr * 10 : numRows + logEveryNRows;
    if (cntr < 0 || numRows < 0) {
      cntr = 0;
      numRows = 1;
    }
    LOG.info(toString() + ": records written - " + numRows);
  }
}

...

if (LOG.isTraceEnabled()) {
  LOG.info(toString() + ": records written - " + numRows);
}

There are logging guards here checking for TRACE level debugging but the logging is actually INFO. This is important logging for detecting data skew. Please change guards to check for INFO... or I would prefer that the guards are removed altogether since it's very rare that a service is running with only WARN level logging.

Issue Links

• is related to: HIVE-10078 Optionally allow logging of records processed in fixed intervals