Hadoop YARN
  1. Hadoop YARN
  2. YARN-47 [Umbrella] Security issues in YARN
  3. YARN-49

Improve distributed shell application to work on a secure cluster

    Details

    1. YARN-49-20130923.3.txt
      11 kB
      Vinod Kumar Vavilapalli

      Activity

      Hitesh Shah created issue -
      Hitesh Shah made changes -
      Field Original Value New Value
      Summary Improve distributed shell example to work on a secure cluster Improve distributed shell application to work on a secure cluster
      Description Improve the distributed shell application to be able to work on a secure cluster.
      Vinod Kumar Vavilapalli made changes -
      Parent MAPREDUCE-3101 [ 12524796 ]
      Issue Type Improvement [ 4 ] Sub-task [ 7 ]
      Vinod Kumar Vavilapalli made changes -
      Affects Version/s 0.23.0 [ 12315570 ]
      Component/s security [ 12313041 ]
      Vinod Kumar Vavilapalli made changes -
      Parent MAPREDUCE-3101 [ 12524796 ]
      Issue Type Sub-task [ 7 ] Bug [ 1 ]
      Vinod Kumar Vavilapalli made changes -
      Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
      Key MAPREDUCE-3941 YARN-49
      Affects Version/s 0.23.0 [ 12315570 ]
      Component/s security [ 12313041 ]
      Component/s mrv2 [ 12314301 ]
      Vinod Kumar Vavilapalli made changes -
      Parent YARN-47 [ 12605130 ]
      Issue Type Bug [ 1 ] Sub-task [ 7 ]
      Vinod Kumar Vavilapalli made changes -
      Component/s applications/distributed-shell [ 12319650 ]
      Omkar Vinit Joshi made changes -
      Assignee Hitesh Shah [ hitesh ] Omkar Vinit Joshi [ ojoshi ]
      Hide
      Omkar Vinit Joshi added a comment -

      This is now fixed for secure environment too after YARN-694.

      Show
      Omkar Vinit Joshi added a comment - This is now fixed for secure environment too after YARN-694 .
      Hide
      Arun C Murthy added a comment -

      Can we close this then?

      Show
      Arun C Murthy added a comment - Can we close this then?
      Hide
      Vinod Kumar Vavilapalli added a comment -

      No, the last we tested, it wasn't working yet because of some missing token propagation..

      Show
      Vinod Kumar Vavilapalli added a comment - No, the last we tested, it wasn't working yet because of some missing token propagation..
      Hide
      Omkar Vinit Joshi added a comment -

      yes it is not working because of missing token propagation... I thought it is fixed but it is not..

      Show
      Omkar Vinit Joshi added a comment - yes it is not working because of missing token propagation... I thought it is fixed but it is not..
      Hide
      Mohammad Kamrul Islam added a comment -

      I need it for new Giraph AM with 2.1.x.

      Show
      Mohammad Kamrul Islam added a comment - I need it for new Giraph AM with 2.1.x.
      Hide
      Mohammad Kamrul Islam added a comment -

      Omkar Vinit Joshi do you have WIP patch that i can use for new Giraph AM? It doesn't need to work though.

      Show
      Mohammad Kamrul Islam added a comment - Omkar Vinit Joshi do you have WIP patch that i can use for new Giraph AM? It doesn't need to work though.
      Hide
      Omkar Vinit Joshi added a comment -

      Hi Mohammad Kamrul Islam, I was stuck with some other issues. I don't have any patch as of now.

      Show
      Omkar Vinit Joshi added a comment - Hi Mohammad Kamrul Islam , I was stuck with some other issues. I don't have any patch as of now.
      Hide
      Vinod Kumar Vavilapalli added a comment -

      I started working on this, but I realized it is a little involved as there is zero security work in Dist-shell that's already done.

      Should have a patch in a day or two.

      Show
      Vinod Kumar Vavilapalli added a comment - I started working on this, but I realized it is a little involved as there is zero security work in Dist-shell that's already done. Should have a patch in a day or two.
      Vinod Kumar Vavilapalli made changes -
      Assignee Omkar Vinit Joshi [ ojoshi ] Vinod Kumar Vavilapalli [ vinodkv ]
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Straight forward patch to add security

      • Client obtains delegation token from default file-system (only default FS today, have to extend more) and puts it in AM Container tokens.
      • Because everything else magically happens, AMRMToken, NMToken, ContainerToken etc are already taken care of.
      • One thing that I'm doing in AM is to filter out AMRMToken from sending them across to containers.

      No unit tests. Tested this on a single node secure setup.

      Show
      Vinod Kumar Vavilapalli added a comment - Straight forward patch to add security Client obtains delegation token from default file-system (only default FS today, have to extend more) and puts it in AM Container tokens. Because everything else magically happens, AMRMToken, NMToken, ContainerToken etc are already taken care of. One thing that I'm doing in AM is to filter out AMRMToken from sending them across to containers. No unit tests. Tested this on a single node secure setup.
      Vinod Kumar Vavilapalli made changes -
      Attachment YARN-49-20130923.3.txt [ 12604705 ]
      Vinod Kumar Vavilapalli made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Omkar Vinit Joshi added a comment -

      Thanks vinod..

      Because everything else magically happens, AMRMToken, NMToken, ContainerToken etc are already taken care of.

      This was good and will set as an example for other Yarn app writers to use client libraries.

      One thing that I'm doing in AM is to filter out AMRMToken from sending them across to containers.

      +1

      No unit tests. Tested this on a single node secure setup.

      Tested this on my local secure setup. Also tested AMRMToken removal.

      +1 lgtm

      Show
      Omkar Vinit Joshi added a comment - Thanks vinod.. Because everything else magically happens, AMRMToken, NMToken, ContainerToken etc are already taken care of. This was good and will set as an example for other Yarn app writers to use client libraries. One thing that I'm doing in AM is to filter out AMRMToken from sending them across to containers. +1 No unit tests. Tested this on a single node secure setup. Tested this on my local secure setup. Also tested AMRMToken removal. +1 lgtm
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12604705/YARN-49-20130923.3.txt
      against trunk revision .

      +1 @author. The patch does not contain any @author tags.

      -1 tests included. The patch doesn't appear to include any new or modified tests.
      Please justify why no new tests are needed for this patch.
      Also please list what manual steps were performed to verify this patch.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 eclipse:eclipse. The patch built with eclipse:eclipse.

      +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:

      org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1992//testReport/
      Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1992//console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12604705/YARN-49-20130923.3.txt against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1992//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1992//console This message is automatically generated.
      Hide
      Vinod Kumar Vavilapalli added a comment -

      TestDistributedShell is sometimes running into YARN-1070, a race condition. It passes sometimes and fails sometimes on my box; not related to this patch directly.

      Show
      Vinod Kumar Vavilapalli added a comment - TestDistributedShell is sometimes running into YARN-1070 , a race condition. It passes sometimes and fails sometimes on my box; not related to this patch directly.
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Oh, and I just tested this in a multiple nodes' secure setup, all is well!

      Show
      Vinod Kumar Vavilapalli added a comment - Oh, and I just tested this in a multiple nodes' secure setup, all is well!
      Hide
      Hitesh Shah added a comment -

      +1. Looks good.

      Show
      Hitesh Shah added a comment - +1. Looks good.
      Hide
      Hitesh Shah added a comment -

      Thanks Vinod and Omkar for the patch and reviews. Committed to trunk, branch-2 and branch-2.1-beta.

      Show
      Hitesh Shah added a comment - Thanks Vinod and Omkar for the patch and reviews. Committed to trunk, branch-2 and branch-2.1-beta.
      Hitesh Shah made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Fix Version/s 2.1.2-beta [ 12325051 ]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      SUCCESS: Integrated in Hadoop-trunk-Commit #4468 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4468/)
      YARN-49. Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330)

      • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Show
      Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4468 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4468/ ) YARN-49 . Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330 ) /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hadoop-Yarn-trunk #344 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/344/)
      YARN-49. Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330)

      • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Show
      Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #344 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/344/ ) YARN-49 . Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330 ) /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hadoop-Mapreduce-trunk #1560 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1560/)
      YARN-49. Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330)

      • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Show
      Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1560 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1560/ ) YARN-49 . Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330 ) /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Hide
      Hudson added a comment -

      SUCCESS: Integrated in Hadoop-Hdfs-trunk #1534 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1534/)
      YARN-49. Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330)

      • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
      • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Show
      Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1534 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1534/ ) YARN-49 . Improve distributed shell application to work on a secure cluster. Contributed by Vinod Kumar Vavilapalli. (hitesh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526330 ) /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/ApplicationMaster.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java
      Hide
      Vinod Kumar Vavilapalli added a comment -

      Closing old tickets that are already part of a release.

      Show
      Vinod Kumar Vavilapalli added a comment - Closing old tickets that are already part of a release.
      Vinod Kumar Vavilapalli made changes -
      Status Resolved [ 5 ] Closed [ 6 ]
      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Patch Available Patch Available
      572d 4h 47m 1 Vinod Kumar Vavilapalli 24/Sep/13 01:30
      Patch Available Patch Available Resolved Resolved
      1d 23h 12m 1 Hitesh Shah 26/Sep/13 00:43
      Resolved Resolved Closed Closed
      642d 7h 37m 1 Vinod Kumar Vavilapalli 30/Jun/15 08:20

        People

        • Assignee:
          Vinod Kumar Vavilapalli
          Reporter:
          Hitesh Shah
        • Votes:
          0 Vote for this issue
          Watchers:
          9 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development