Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-732

node health check script should not log "UNHEALTHY" status for every heartbeat in INFO mode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Changed log level of addition of blacklisted reason in the JobTracker log to debug instead of INFO

      Description

      Currently, when a TT is blacklisted by the node health check script, for every heartbeat a message such as the following is being logged.

      <date> <time> INFO org.apache.hadoop.mapred.JobTracker: Adding blacklisted reason for tracker : <blacklisted TT> Reason for blacklisting is : NODE_UNHEALTHY
      

      Due to this, the the JT logs fill up rapidly clogging the logdirs. Hence this message should be logged in DEBUG mode instead of INFO mode.

      1. MAPRED-732-ydist.patch
        1 kB
        Sreekanth Ramakrishnan
      2. mapreduce-732.patch
        0.8 kB
        Sreekanth Ramakrishnan
      3. mapreduce-732-1.patch
        1.0 kB
        Sreekanth Ramakrishnan
      4. MAPREDUCE-732-2.patch
        1 kB
        Hemanth Yamijala

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #20 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/20/ )
        Hide
        Sreekanth Ramakrishnan added a comment -

        Yahoo distribution patch for the issue.

        Show
        Sreekanth Ramakrishnan added a comment - Yahoo distribution patch for the issue.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Sreekanth !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Sreekanth !
        Hide
        Hemanth Yamijala added a comment -

        Since this is a trivial fix and a change only to LOG messages, I am going to commit this. I ran TestTaskTrackerBlacklisting just to be sure.

        Show
        Hemanth Yamijala added a comment - Since this is a trivial fix and a change only to LOG messages, I am going to commit this. I ran TestTaskTrackerBlacklisting just to be sure.
        Hide
        Hemanth Yamijala added a comment -

        The attached patch file did not apply to trunk. I created a new one with the same fix.

        Show
        Hemanth Yamijala added a comment - The attached patch file did not apply to trunk. I created a new one with the same fix.
        Hide
        Sreekanth Ramakrishnan added a comment -

        Attaching patch incorporating Hong's comment:

        • Whenever a new blacklisting reason is added we log the addition of reason.
        Show
        Sreekanth Ramakrishnan added a comment - Attaching patch incorporating Hong's comment: Whenever a new blacklisting reason is added we log the addition of reason.
        Hide
        Hong Tang added a comment -

        Got it. Minor improvement: if the reason-for-blacklisting is changed, we probably want to log it as INFO.

        Show
        Hong Tang added a comment - Got it. Minor improvement: if the reason-for-blacklisting is changed, we probably want to log it as INFO.
        Hide
        Sreekanth Ramakrishnan added a comment -

        We log everytime when node changes it health status. But currently when a node becomes unhealthy, every subsequent heart beat we log "Adding blacklisted reason for tracker :". This has to be changed as we might end up logging a line per blacklisted tracker every heartbeat, filling up the log files.

        The logging of the change in health status is done in line number 648 and 623 of JobTracker

        Show
        Sreekanth Ramakrishnan added a comment - We log everytime when node changes it health status. But currently when a node becomes unhealthy, every subsequent heart beat we log "Adding blacklisted reason for tracker :". This has to be changed as we might end up logging a line per blacklisted tracker every heartbeat, filling up the log files. The logging of the change in health status is done in line number 648 and 623 of JobTracker
        Hide
        Hong Tang added a comment -

        It would be better we add log entries every time a node changes its health status (healthy -> not healthy and vice versa).

        Show
        Hong Tang added a comment - It would be better we add log entries every time a node changes its health status (healthy -> not healthy and vice versa).
        Hide
        Sreekanth Ramakrishnan added a comment -

        A trival patch, changed the log statement to debug level

        Show
        Sreekanth Ramakrishnan added a comment - A trival patch, changed the log statement to debug level

          People

          • Assignee:
            Sreekanth Ramakrishnan
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development