Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-732

node health check script should not log "UNHEALTHY" status for every heartbeat in INFO mode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed log level of addition of blacklisted reason in the JobTracker log to debug instead of INFO

      Description

      Currently, when a TT is blacklisted by the node health check script, for every heartbeat a message such as the following is being logged.

      <date> <time> INFO org.apache.hadoop.mapred.JobTracker: Adding blacklisted reason for tracker : <blacklisted TT> Reason for blacklisting is : NODE_UNHEALTHY
      

      Due to this, the the JT logs fill up rapidly clogging the logdirs. Hence this message should be logged in DEBUG mode instead of INFO mode.

      1. MAPREDUCE-732-2.patch
        1 kB
        Hemanth Yamijala
      2. mapreduce-732-1.patch
        1.0 kB
        Sreekanth Ramakrishnan
      3. mapreduce-732.patch
        0.8 kB
        Sreekanth Ramakrishnan
      4. MAPRED-732-ydist.patch
        1 kB
        Sreekanth Ramakrishnan

        Activity

        Ramya Sunil created issue -
        Sreekanth Ramakrishnan made changes -
        Field Original Value New Value
        Assignee Sreekanth Ramakrishnan [ sreekanth ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        A trival patch, changed the log statement to debug level

        Show
        Sreekanth Ramakrishnan added a comment - A trival patch, changed the log statement to debug level
        Sreekanth Ramakrishnan made changes -
        Attachment mapreduce-732.patch [ 12412842 ]
        Hide
        Hong Tang added a comment -

        It would be better we add log entries every time a node changes its health status (healthy -> not healthy and vice versa).

        Show
        Hong Tang added a comment - It would be better we add log entries every time a node changes its health status (healthy -> not healthy and vice versa).
        Hide
        Sreekanth Ramakrishnan added a comment -

        We log everytime when node changes it health status. But currently when a node becomes unhealthy, every subsequent heart beat we log "Adding blacklisted reason for tracker :". This has to be changed as we might end up logging a line per blacklisted tracker every heartbeat, filling up the log files.

        The logging of the change in health status is done in line number 648 and 623 of JobTracker

        Show
        Sreekanth Ramakrishnan added a comment - We log everytime when node changes it health status. But currently when a node becomes unhealthy, every subsequent heart beat we log "Adding blacklisted reason for tracker :". This has to be changed as we might end up logging a line per blacklisted tracker every heartbeat, filling up the log files. The logging of the change in health status is done in line number 648 and 623 of JobTracker
        Hide
        Hong Tang added a comment -

        Got it. Minor improvement: if the reason-for-blacklisting is changed, we probably want to log it as INFO.

        Show
        Hong Tang added a comment - Got it. Minor improvement: if the reason-for-blacklisting is changed, we probably want to log it as INFO.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch incorporating Hong's comment:

        • Whenever a new blacklisting reason is added we log the addition of reason.
        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch incorporating Hong's comment: Whenever a new blacklisting reason is added we log the addition of reason.
        Sreekanth Ramakrishnan made changes -
        Attachment mapreduce-732-1.patch [ 12412953 ]
        Hide
        Hemanth Yamijala added a comment -

        The attached patch file did not apply to trunk. I created a new one with the same fix.

        Show
        Hemanth Yamijala added a comment - The attached patch file did not apply to trunk. I created a new one with the same fix.
        Hemanth Yamijala made changes -
        Attachment MAPREDUCE-732-2.patch [ 12412985 ]
        Hide
        Hemanth Yamijala added a comment -

        Since this is a trivial fix and a change only to LOG messages, I am going to commit this. I ran TestTaskTrackerBlacklisting just to be sure.

        Show
        Hemanth Yamijala added a comment - Since this is a trivial fix and a change only to LOG messages, I am going to commit this. I ran TestTaskTrackerBlacklisting just to be sure.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Sreekanth !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Sreekanth !
        Hemanth Yamijala made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Sreekanth Ramakrishnan added a comment -

        Yahoo distribution patch for the issue.

        Show
        Sreekanth Ramakrishnan added a comment - Yahoo distribution patch for the issue.
        Sreekanth Ramakrishnan made changes -
        Attachment MAPRED-732-ydist.patch [ 12413001 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/ )
        Sreekanth Ramakrishnan made changes -
        Release Note Changed log level of addition of blacklisted reason in the JobTracker log to debug instead of INFO
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 1h 24m 1 Hemanth Yamijala 09/Jul/09 10:42
        Resolved Resolved Closed Closed
        411d 10h 31m 1 Tom White 24/Aug/10 21:14

          People

          • Assignee:
            Sreekanth Ramakrishnan
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development