Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4227

TimeWindow statistics are not updated for TaskTrackers which have been restarted.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Whenever a TaskTracker is restarted after the JobTracker has been running for a while (an hour / day maybe), the TimeWindow statistics on the JobTracker Active nodes page are stuck at 0.

        Issue Links

          Activity

          Ravi Prakash made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Ravi Prakash made changes -
          Link This issue is duplicated by MAPREDUCE-4963 [ MAPREDUCE-4963 ]
          Hide
          Ravi Prakash added a comment -
          Show
          Ravi Prakash added a comment - https://issues.apache.org/jira/browse/MAPREDUCE-4963 has been committed.
          Matt Foley made changes -
          Target Version/s 1.1.0 [ 12317960 ] 1.2.0 [ 12321661 ]
          Hide
          Matt Foley added a comment -

          Moved to 1.2.0 upon release of 1.1.0.

          Show
          Matt Foley added a comment - Moved to 1.2.0 upon release of 1.1.0.
          Hide
          Ravi Prakash added a comment -

          -1 patch. The patch command could not apply the patch.

          Yeah! This patch applies only to branch-1

          Show
          Ravi Prakash added a comment - -1 patch. The patch command could not apply the patch. Yeah! This patch applies only to branch-1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12525873/MAPREDUCE-4227.branch-1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2396//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525873/MAPREDUCE-4227.branch-1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2396//console This message is automatically generated.
          Ravi Prakash made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Ravi Prakash added a comment -

          I ran test-patch:

          -1 overall.  
              +1 @author.  The patch does not contain any @author tags.
              +1 tests included.  The patch appears to include 3 new or modified tests.
              +1 javadoc.  The javadoc tool did not generate any warning messages.
              +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
              -1 findbugs.  The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings.
          

          The findbugs warning is spurious. We've been through this before https://issues.apache.org/jira/browse/MAPREDUCE-4088?focusedCommentId=13263990&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13263990 .

          Show
          Ravi Prakash added a comment - I ran test-patch: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. The findbugs warning is spurious. We've been through this before https://issues.apache.org/jira/browse/MAPREDUCE-4088?focusedCommentId=13263990&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13263990 .
          Ravi Prakash made changes -
          Attachment MAPREDUCE-4227.branch-1.patch [ 12525873 ]
          Hide
          Ravi Prakash added a comment -

          Hi Matt,

          I'm attaching the patch. It makes the following changes
          1. buckets are removed only if there are more in the current TaskTrackers than are required
          2. Earlier the TimeWindow statistics were not being updated asap. They would lag behind by one "update cycle" I've fixed that as well.
          3. I took the liberty to add some much needed javadocs in this JIRA.

          Disclaimer: StatisticsCollector is being referenced in some AspectJ files. Unfortunately I don't know how I can check if my changes affected anything in there. Any clues?

          Show
          Ravi Prakash added a comment - Hi Matt, I'm attaching the patch. It makes the following changes 1. buckets are removed only if there are more in the current TaskTrackers than are required 2. Earlier the TimeWindow statistics were not being updated asap. They would lag behind by one "update cycle" I've fixed that as well. 3. I took the liberty to add some much needed javadocs in this JIRA. Disclaimer: StatisticsCollector is being referenced in some AspectJ files. Unfortunately I don't know how I can check if my changes affected anything in there. Any clues?
          Matt Foley made changes -
          Field Original Value New Value
          Target Version/s 1.0.3 [ 12320250 ] 1.1.0 [ 12317960 ]
          Hide
          Matt Foley added a comment -

          Hi, Ravi and Koji, could you please convert this issue into a patch? Thanks.

          Show
          Matt Foley added a comment - Hi, Ravi and Koji, could you please convert this issue into a patch? Thanks.
          Hide
          Ravi Prakash added a comment -

          I'd been banging my head on this bug for weeks and then Koji swept in like an angel that he is, and rescued me.

          Courtesy Koji!

          src/mapred/org/apache/hadoop/mapred/StatisticsCollector.java

          276     synchronized void update() {
          277       updates++;
          278       if (updates == updatesPerBucket) {
          279         for(TimeStat stat : statToCollect.values()) {
          280           stat.addBucket();
          281         }
          282         updates = 0;
          283         buckets++;
          284         if (buckets > collectBuckets) {
          285           for (TimeStat stat : statToCollect.values()) {
          286             stat.removeBucket();
          287           }
          288           buckets--;
          289         }
          290       }
          291     }
          292   }
          

          'bucket' keeps track of how many buckets ALL the TimeStat has for that Range (Hourly,Daily) so after 1 hour, buckets == collectBuckets and each time it adds, it takes out 1.
          i believe the incorrect assumption they are making here is ALL the TimeStat would have same number of buckets

          One of these days I'll start a Koji fan club! Please consider joining.

          Show
          Ravi Prakash added a comment - I'd been banging my head on this bug for weeks and then Koji swept in like an angel that he is, and rescued me. Courtesy Koji! src/mapred/org/apache/hadoop/mapred/StatisticsCollector.java 276 synchronized void update() { 277 updates++; 278 if (updates == updatesPerBucket) { 279 for(TimeStat stat : statToCollect.values()) { 280 stat.addBucket(); 281 } 282 updates = 0; 283 buckets++; 284 if (buckets > collectBuckets) { 285 for (TimeStat stat : statToCollect.values()) { 286 stat.removeBucket(); 287 } 288 buckets--; 289 } 290 } 291 } 292 } 'bucket' keeps track of how many buckets ALL the TimeStat has for that Range (Hourly,Daily) so after 1 hour, buckets == collectBuckets and each time it adds, it takes out 1. i believe the incorrect assumption they are making here is ALL the TimeStat would have same number of buckets One of these days I'll start a Koji fan club! Please consider joining.
          Ravi Prakash created issue -

            People

            • Assignee:
              Ravi Prakash
              Reporter:
              Ravi Prakash
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development