Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. MAPREDUCE-3141-20111004.txt
      4 kB
      Vinod Kumar Vavilapalli
    2. MAPREDUCE-3141-20111007.txt
      4 kB
      Vinod Kumar Vavilapalli

      Activity

      Vinod Kumar Vavilapalli created issue -
      Vinod Kumar Vavilapalli made changes -
      Field Original Value New Value
      Fix Version/s 0.23.0 [ 12315570 ]
      Vinod Kumar Vavilapalli made changes -
      Component/s security [ 12313041 ]
      Hide
      Vinod Kumar Vavilapalli added a comment -

      After MAPREDUCE-3056 went in, I ran single node secure setup which is breaking with YARN+MR.

      I see that single node secure setup is now identical to a multi-node setup only after MAPREDUCE-3056. Before that, MRAppMaster(run on the same node) used to pick up kerberos tokens from the local token-cache which was filled during job-submit(again on the same node); so in a sense single node-setup wasn't the same as multi-node secure setup.

      I suspect MAPREDUCE-3006 to have broken this code, but surprised that others haven't run into this.

      Show
      Vinod Kumar Vavilapalli added a comment - After MAPREDUCE-3056 went in, I ran single node secure setup which is breaking with YARN+MR. I see that single node secure setup is now identical to a multi-node setup only after MAPREDUCE-3056 . Before that, MRAppMaster(run on the same node) used to pick up kerberos tokens from the local token-cache which was filled during job-submit(again on the same node); so in a sense single node-setup wasn't the same as multi-node secure setup. I suspect MAPREDUCE-3006 to have broken this code, but surprised that others haven't run into this.
      Vinod Kumar Vavilapalli made changes -
      Affects Version/s 0.23.0 [ 12315570 ]
      Priority Major [ 3 ] Blocker [ 1 ]
      Component/s applicationmaster [ 12315625 ]
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Patch that fixes the issue for me on single node secure setup, which is now same as multi-node setup from the POV of security of the communication channels.

      Show
      Vinod Kumar Vavilapalli added a comment - Patch that fixes the issue for me on single node secure setup, which is now same as multi-node setup from the POV of security of the communication channels.
      Vinod Kumar Vavilapalli made changes -
      Attachment MAPREDUCE-3141-20111004.txt [ 12497640 ]
      Vinod Kumar Vavilapalli made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12497640/MAPREDUCE-3141-20111004.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/937//testReport/
      Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/937//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12497640/MAPREDUCE-3141-20111004.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/937//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/937//console This message is automatically generated.
      Vinod Kumar Vavilapalli made changes -
      Component/s mrv2 [ 12314301 ]
      Hide
      Mahadev konar added a comment -

      +1 the patch looks good, just some minor nits:

      The patch has this:

        LOG.info(" --- DEBUG: Token of kind " + tk.getKind()
      +              + "in current ugi in the AppMaster for service "
      +              + tk.getService());
      

      Might want to make it debug.

      also, the javadoc warning needs to be fixed.

      Show
      Mahadev konar added a comment - +1 the patch looks good, just some minor nits: The patch has this: LOG.info( " --- DEBUG: Token of kind " + tk.getKind() + + "in current ugi in the AppMaster for service " + + tk.getService()); Might want to make it debug. also, the javadoc warning needs to be fixed.
      Mahadev konar made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Patch addressing comment and fixing javadoc error.

      Show
      Vinod Kumar Vavilapalli added a comment - Patch addressing comment and fixing javadoc error.
      Vinod Kumar Vavilapalli made changes -
      Attachment MAPREDUCE-3141-20111007.txt [ 12498128 ]
      Vinod Kumar Vavilapalli made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hadoop Flags Reviewed [ 10343 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12498128/MAPREDUCE-3141-20111007.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      +1 core tests. The patch passed unit tests in .

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/963//testReport/
      Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/963//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12498128/MAPREDUCE-3141-20111007.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/963//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/963//console This message is automatically generated.
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Thanks for the review, Mahadev.

      I just committed this to branch-0.23 and trunk.

      Show
      Vinod Kumar Vavilapalli added a comment - Thanks for the review, Mahadev. I just committed this to branch-0.23 and trunk.
      Vinod Kumar Vavilapalli made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Resolution Fixed [ 1 ]
      Hide
      Vinod Kumar Vavilapalli added a comment -

      -1 tests included. The patch doesn't appear to include any new or modified tests.

      Please justify why no new tests are needed for this patch.

      This is a fix for security mode which we don't have test infrastructure for.
      Validated this on a single node secure setup.

      Show
      Vinod Kumar Vavilapalli added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. This is a fix for security mode which we don't have test infrastructure for. Validated this on a single node secure setup.
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk-Commit #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1111/)
      MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv)

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007
      Files :

      • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1111 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1111/ ) MAPREDUCE-3141 . Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Common-trunk-Commit #1033 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1033/)
      MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv)

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007
      Files :

      • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1033 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1033/ ) MAPREDUCE-3141 . Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #1051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1051/)
      MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv)

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007
      Files :

      • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1051 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1051/ ) MAPREDUCE-3141 . Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-0.23-Build #39 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/39/)
      MAPREDUCE-3141. svn merge -c r1180007 --ignore-ancestry ../../trunk/

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180008
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #39 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/39/ ) MAPREDUCE-3141 . svn merge -c r1180007 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180008 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk #853 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/853/)
      MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv)

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007
      Files :

      • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #853 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/853/ ) MAPREDUCE-3141 . Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-trunk #824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/824/)
      MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv)

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007
      Files :

      • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #824 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/824/ ) MAPREDUCE-3141 . Fix the broken MRAppMaster to work over YARN in security mode.(vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180007 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Hdfs-0.23-Build #33 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/33/)
      MAPREDUCE-3141. svn merge -c r1180007 --ignore-ancestry ../../trunk/

      vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180008
      Files :

      • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
      • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Show
      Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #33 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/33/ ) MAPREDUCE-3141 . svn merge -c r1180007 --ignore-ancestry ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1180008 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
      Arun C Murthy made changes -
      Status Resolved [ 5 ] Closed [ 6 ]

        People

        • Assignee:
          Vinod Kumar Vavilapalli
          Reporter:
          Vinod Kumar Vavilapalli
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development