Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2327

MapTask doesn't need to put username information in SpillRecord

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is an amendment to MAPREDUCE-2096 that's found in Yahoo's 0.20.100 branch.

      This bug causes task failures in the following case:

      • Cluster is not set up with LinuxTaskController (ie not secured cluster)
      • Job submitter is not the same as the user running the TT
      • Map output is more than one spill's worth

      The issue is that UserGroupInformation's view of the current user is the job submitter, but on disk the spill files will be owned by the TT user. SecureIO will then fail when constructing the spill record.

      1. mr-2327.txt
        0.7 kB
        Todd Lipcon
      2. mapreduce-2327.txt
        5 kB
        Todd Lipcon
      3. mapreduce-2327.txt
        5 kB
        Todd Lipcon
      4. mapreduce-2327.txt
        5 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #682 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/)
          MAPREDUCE-2327. MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #682 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/682/ ) MAPREDUCE-2327 . MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #52 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/)
          Merge -r 1103989:1103990 from trunk to branch-0.22. Fixes: MAPREDUCE-2327

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #52 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/52/ ) Merge -r 1103989:1103990 from trunk to branch-0.22. Fixes: MAPREDUCE-2327
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #678 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/678/)
          MAPREDUCE-2327. MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #678 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/678/ ) MAPREDUCE-2327 . MapTask doesn't need to put username information in SpillRecord. Contributed by Todd Lipcon.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt
          against trunk revision 1103921.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt against trunk revision 1103921. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/254//console This message is automatically generated.
          Hide
          Tom White added a comment -

          I've just committed this. Thanks, Todd!

          Show
          Tom White added a comment - I've just committed this. Thanks, Todd!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt
          against trunk revision 1103921.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479210/mapreduce-2327.txt against trunk revision 1103921. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/247//console This message is automatically generated.
          Hide
          Tom White added a comment -

          +1

          Show
          Tom White added a comment - +1
          Hide
          Todd Lipcon added a comment -

          Sure, here's an updated patch that does that.

          Show
          Todd Lipcon added a comment - Sure, here's an updated patch that does that.
          Hide
          Tom White added a comment -

          Rather than passing in null, how about adding a new constructor to SpillRecord that doesn't take an expected owner?

          Show
          Tom White added a comment - Rather than passing in null, how about adding a new constructor to SpillRecord that doesn't take an expected owner?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477614/mapreduce-2327.txt
          against trunk revision 1097315.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477614/mapreduce-2327.txt against trunk revision 1097315. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/191//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12477073/mapreduce-2327.txt
          against trunk revision 1097305.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477073/mapreduce-2327.txt against trunk revision 1097305. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/189//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Good point on the config name. Fixed.

          As for whether this is actually a real bug – now that HADOOP-7172 has gone in, you're right that this won't cause a problem. But, as a matter of principle, we might as well put the right expected owner in the data structure

          Show
          Todd Lipcon added a comment - Good point on the config name. Fixed. As for whether this is actually a real bug – now that HADOOP-7172 has gone in, you're right that this won't cause a problem. But, as a matter of principle, we might as well put the right expected owner in the data structure
          Hide
          Tom White added a comment -

          If security isn't enabled then won't the SecureIOUtils.openForRead() call ignore the username that's passed in?

          The new configuration parameter name should begin "mapreduce", not "mapred", for consistency.

          Show
          Tom White added a comment - If security isn't enabled then won't the SecureIOUtils.openForRead() call ignore the username that's passed in? The new configuration parameter name should begin "mapreduce", not "mapred", for consistency.
          Hide
          Todd Lipcon added a comment -

          Here's a patch on top of MAPREDUCE-2445 including a unit test (which fails without this bug fix)

          Will mark patch avail after 2445 is in.

          Show
          Todd Lipcon added a comment - Here's a patch on top of MAPREDUCE-2445 including a unit test (which fails without this bug fix) Will mark patch avail after 2445 is in.
          Hide
          Todd Lipcon added a comment -

          Marking blocked by MAPREDUCE-2445, since I want to add another test case to that file. Patch forthcoming on top of that one.

          Show
          Todd Lipcon added a comment - Marking blocked by MAPREDUCE-2445 , since I want to add another test case to that file. Patch forthcoming on top of that one.
          Hide
          Nigel Daley added a comment -

          Todd, any update on this for 0.22?

          Show
          Nigel Daley added a comment - Todd, any update on this for 0.22?
          Hide
          Todd Lipcon added a comment -

          Hudson is right, I should write a unit test for this.

          Show
          Todd Lipcon added a comment - Hudson is right, I should write a unit test for this.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12471252/mr-2327.txt
          against trunk revision 1074251.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12471252/mr-2327.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/26//console This message is automatically generated.
          Hide
          Raj Vishwanathan added a comment -

          Todd

          This looks like the bug I hit. Can I work around it by running the job as the user mapred? From the bug report, that should work.

          Raj

          Show
          Raj Vishwanathan added a comment - Todd This looks like the bug I hit. Can I work around it by running the job as the user mapred? From the bug report, that should work. Raj

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development