Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-20168

ReduceSinkOperator Logging Hidden

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.0, 4.0.0
    • Fix Version/s: 4.0.0, 3.2.0
    • Component/s: Operators
    • Labels:

      Description

      https://github.com/apache/hive/blob/ac6b2a3fb195916e22b2e5f465add2ffbcdc7430/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java

       

      if (LOG.isTraceEnabled()) {
        if (numRows == cntr) {
          cntr = logEveryNRows == 0 ? cntr * 10 : numRows + logEveryNRows;
          if (cntr < 0 || numRows < 0) {
            cntr = 0;
            numRows = 1;
          }
          LOG.info(toString() + ": records written - " + numRows);
        }
      }
      
      ...
      
      if (LOG.isTraceEnabled()) {
        LOG.info(toString() + ": records written - " + numRows);
      }
      

      There are logging guards here checking for TRACE level debugging but the logging is actually INFO. This is important logging for detecting data skew. Please change guards to check for INFO... or I would prefer that the guards are removed altogether since it's very rare that a service is running with only WARN level logging.

        Attachments

        1. HIVE-20168.1.patch
          2 kB
          Bharath Krishna
        2. HIVE-20168.2.patch
          2 kB
          Bharath Krishna
        3. HIVE-20168.3.patch
          2 kB
          Bharath Krishna

          Issue Links

            Activity

              People

              • Assignee:
                bharos92 Bharath Krishna
                Reporter:
                belugabehr David Mollitor
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: