Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2777

Backport MAPREDUCE-220 to Hadoop 20 security branch

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0
    • Fix Version/s: 0.20.205.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      Adds cumulative cpu usage and total heap usage to task counters. This is a backport of MAPREDUCE-220 and MAPREDUCE-2469.
      Show
      Adds cumulative cpu usage and total heap usage to task counters. This is a backport of MAPREDUCE-220 and MAPREDUCE-2469 .

      Issue Links

        Activity

        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Matt Foley added a comment -

        Closed upon release of 0.20.205.0

        Show
        Matt Foley added a comment - Closed upon release of 0.20.205.0
        Matt Foley made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Target Version/s 0.20.206.0 [ 12317960 ] 0.20.205.0 [ 12316391 ]
        Fix Version/s 0.20.205.0 [ 12316391 ]
        Resolution Fixed [ 1 ]
        Hide
        Matt Foley added a comment -

        Merged back into 0.20-security-205, along with r1177389 to delete the now-empty test file TestTTMemoryReporting.

        Show
        Matt Foley added a comment - Merged back into 0.20-security-205, along with r1177389 to delete the now-empty test file TestTTMemoryReporting.
        Hide
        Amar Kamat added a comment -

        I re-ran TestTrackerDistributedCacheManager with and without my patch and it passed.
        I suggest we commit this patch to 0.20.205.

        Show
        Amar Kamat added a comment - I re-ran TestTrackerDistributedCacheManager with and without my patch and it passed. I suggest we commit this patch to 0.20.205.
        Hide
        Amar Kamat added a comment -

        test-patch output:

         [exec] +1 overall.  
             [exec] 
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec] 
             [exec]     +1 tests included.  The patch appears to include 21 new or modified tests.
             [exec] 
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec] 
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec] 
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.8) warnings.
        
        Show
        Amar Kamat added a comment - test-patch output: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 21 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.8) warnings.
        Hide
        Amar Kamat added a comment -

        Matt,
        The 2 failures are due to
        1. TestTrackerDistributedCacheManager : This error seems unrelated to this patch. This patch doesn't change anything in DistributedCache. I checked the stack trace and found NPEs. I will look into it further.
        2. TestTTMemoryReporting: This testcase got deleted and rewritten as part of this patch. We only need to delete the stray file.

        I uploaded the patch only after test-patch and ant-tests passed. I am re-running them to make sure nothing changed after I generated the last patch.

        Show
        Amar Kamat added a comment - Matt, The 2 failures are due to 1. TestTrackerDistributedCacheManager : This error seems unrelated to this patch. This patch doesn't change anything in DistributedCache. I checked the stack trace and found NPEs. I will look into it further. 2. TestTTMemoryReporting: This testcase got deleted and rewritten as part of this patch. We only need to delete the stray file. I uploaded the patch only after test-patch and ant-tests passed. I am re-running them to make sure nothing changed after I generated the last patch.
        Matt Foley made changes -
        Fix Version/s 0.20.206.0 [ 12317960 ]
        Target Version/s 0.20.206.0 [ 12317960 ]
        Matt Foley made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Matt Foley added a comment -

        In build #38, https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/38/ (r1176934), this patch introduced two new unit test failures not present after the preceding patch:
        org.apache.hadoop.filecache.TestTrackerDistributedCacheManager.testDeleteCache
        org.apache.hadoop.mapred.TestTTMemoryReporting

        Reverting from 0.20.205. Amar, please fix in 0.20-security. Thank you.

        Show
        Matt Foley added a comment - In build #38, https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-0.20.205-Build/38/ (r1176934), this patch introduced two new unit test failures not present after the preceding patch: org.apache.hadoop.filecache.TestTrackerDistributedCacheManager.testDeleteCache org.apache.hadoop.mapred.TestTTMemoryReporting Reverting from 0.20.205. Amar, please fix in 0.20-security. Thank you.
        Hide
        Matt Foley added a comment -

        BTW, that request is because you want this patch in 0.20.205, and I'm just about to cut a release candidate for that release, today. Thanks, --Matt (RM for 0.20.205)

        Show
        Matt Foley added a comment - BTW, that request is because you want this patch in 0.20.205, and I'm just about to cut a release candidate for that release, today. Thanks, --Matt (RM for 0.20.205)
        Hide
        Matt Foley added a comment -

        Amar, thank you for providing this backport. For a non-trivial change like this, please run test-patch locally and paste the results summary into the Jira as a comment (since we don't have HADOOP-7435 yet). Thanks.

        Show
        Matt Foley added a comment - Amar, thank you for providing this backport. For a non-trivial change like this, please run test-patch locally and paste the results summary into the Jira as a comment (since we don't have HADOOP-7435 yet). Thanks.
        Hide
        Amar Kamat added a comment -

        I also committed this to 0.20-205.

        Show
        Amar Kamat added a comment - I also committed this to 0.20-205.
        Matt Foley made changes -
        Fix Version/s 0.20.206.0 [ 12317960 ]
        Amar Kamat made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Release Note Adds cumulative cpu usage and total heap usage to task counters. This is a backport of MAPREDUCE-220 and MAPREDUCE-2469.
        Resolution Fixed [ 1 ]
        Hide
        Amar Kamat added a comment -

        I just committed this to the 0.20 security branch.

        Show
        Amar Kamat added a comment - I just committed this to the 0.20 security branch.
        Hide
        Jonathan Eagles added a comment -

        +1 patch looks good to me.

        Show
        Jonathan Eagles added a comment - +1 patch looks good to me.
        Hide
        Amar Kamat added a comment -

        test-patch and ant tests passed with the attached patch.

        Show
        Amar Kamat added a comment - test-patch and ant tests passed with the attached patch.
        Amar Kamat made changes -
        Attachment mapreduce-2777-v1.3.patch [ 12493004 ]
        Hide
        Amar Kamat added a comment -

        Attaching a patch that backports the MAPREDUCE-220 and MAPREDUCE-2469 features to the hadoop-0.20-security branch. MAPREDUCE-220 and MAPREDUCE-2469 have dependencies which also got selectively backported. Here is the list of JIRA's that got backported:
        MR-220, MR-1201, MR-1762, MR-1218 & MR-2469. I have resolved most of the hunks w.r.t parameter naming, missing APIs and missing backports (e.g GC_TIME_MILLIS counters etc).

        We have tested this patch at scale and also benchmarked this patch for regressions. It seems that the overhead is negligible and the newly added counters are correctly captured and recorded.

        Show
        Amar Kamat added a comment - Attaching a patch that backports the MAPREDUCE-220 and MAPREDUCE-2469 features to the hadoop-0.20-security branch. MAPREDUCE-220 and MAPREDUCE-2469 have dependencies which also got selectively backported. Here is the list of JIRA's that got backported: MR-220, MR-1201, MR-1762, MR-1218 & MR-2469. I have resolved most of the hunks w.r.t parameter naming, missing APIs and missing backports (e.g GC_TIME_MILLIS counters etc). We have tested this patch at scale and also benchmarked this patch for regressions. It seems that the overhead is negligible and the newly added counters are correctly captured and recorded.
        Amar Kamat made changes -
        Assignee Amar Kamat [ amar_kamat ]
        Amar Kamat made changes -
        Link This issue is related to MAPREDUCE-2469 [ MAPREDUCE-2469 ]
        Jonathan Eagles made changes -
        Field Original Value New Value
        Link This issue is related to MAPREDUCE-220 [ MAPREDUCE-220 ]
        Jonathan Eagles created issue -

          People

          • Assignee:
            Amar Kamat
            Reporter:
            Jonathan Eagles
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development