Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Jira to track work to secure the ATS

      1. Timeline Security Diagram.pdf
        153 kB
        Zhijie Shen
      2. Timeline_Kerberos_DT_ACLs.patch
        129 kB
        Zhijie Shen
      3. Timeline_Kerberos_DT_ACLs.2.patch
        128 kB
        Zhijie Shen

        Issue Links

          Activity

          Hide
          Zhijie Shen added a comment -

          I'm going to take care of the security issues of the timeline server

          Show
          Zhijie Shen added a comment - I'm going to take care of the security issues of the timeline server
          Hide
          Zhijie Shen added a comment -

          I created an uber patch which integrate the pieces I've done so far. With this patch the timeline server can work in a secure mode (except the generic history service part):

          1. Timeline server can start and login with Kerberors principle and keytab;
          2. The user either passed the Kerberos authentication or obtained the timeline delegation token can get access to the timeline data;
          3. Withe ACLs enabled, only the owner who published the timeline data before can access the data.

          Folks who are interested in the timeline security can play with the patch.

          Show
          Zhijie Shen added a comment - I created an uber patch which integrate the pieces I've done so far. With this patch the timeline server can work in a secure mode (except the generic history service part): 1. Timeline server can start and login with Kerberors principle and keytab; 2. The user either passed the Kerberos authentication or obtained the timeline delegation token can get access to the timeline data; 3. Withe ACLs enabled, only the owner who published the timeline data before can access the data. Folks who are interested in the timeline security can play with the patch.
          Hide
          Zhijie Shen added a comment -

          Kick jenkins to check whether any existing stuff is broken, though the patch passes my local tests.

          Show
          Zhijie Shen added a comment - Kick jenkins to check whether any existing stuff is broken, though the patch passes my local tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12645303/Timeline_Kerberos_DT_ACLs.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:

          org.apache.hadoop.yarn.client.TestRMAdminCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3761//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3761//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645303/Timeline_Kerberos_DT_ACLs.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice: org.apache.hadoop.yarn.client.TestRMAdminCLI +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3761//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3761//console This message is automatically generated.
          Hide
          Zhijie Shen added a comment -

          The test failure should be unrelated: YARN-2075.

          Show
          Zhijie Shen added a comment - The test failure should be unrelated: YARN-2075 .
          Hide
          Zhijie Shen added a comment -

          Hi folks,

          I've just attached a diagram "Timeline Security Diagram.pdf" to demonstrate the rough workflow of the the timeline security. In general, it consists of two parts: authentication and the authorization.

          1. Authentication

          a) When the authentication is enabled, a customized authentication filter will be loaded into the webapp of the timeline server, which prevents unauthorized users to access any timeline web resources. The filter allow users to:

          • negotiate the authentication via HTTP SPNEGO, and login with Kerberos principal and keytab; and
          • request a delegation token after Kerberos login and use it for follow-up secured communication.

          b) TimelineClient is adapted to pass the authentication before putting the timeline data. It can choose append the Kerberos token or delegation token into the HTTP request. The rationale behind supporting delegation token is to allow AM and other containers to use TimelineClient to put the timeline data in a secured manner, where the Kerberos stuff is not available.

          c) TimelineClient also has the API to get the delegation token from the timeline sever (actually from the customized authentication filter). When security is enabled and the timeline service is enabled, and YarnClient is used to submit an application, YarnClient will automatically call TimeClient to get a delegation token and put into the application submission context, such that the AM can used the passed-in delegation token to communicate with the timeline server securely.

          d) Any tool which support SPNEGO/Kerberos, such as Firefox, curl and etc., can access the three GET APIs of the timeline server to inquiry the timeline data.

          2. Authorization

          Once the request from an authenticated user passes the customized authentication filter, it will be processed by the timeline web services. Here we use the ACLs manager to determine whether the user of the request has the access to the requested data. The basic rules are as follows:

          • The access control granularity is entity, which means a user can access all the information of any entity and its events, or he/she can access nothing of it.
          • Currently we only allow the owner of the entity to access it. In the future, we can simply extend the rule to allow Admin and users/groups on the access control list.

          Configuration
          After all, to enable the timeline security, we need to setup Kerberos. In addition, there're a bunch of configurations to do:

          • Make use of the filter initializer to setup the customized authentication filter, and the configuration is much like hadoop-auth style; and
          • ACLs is controlled by YARN ACLs configuration like other YARN daemons.

          I also uploaded my newest uber patch "Timeline_Kerberos_DT_ACLs.2.patch" to demonstrate how the design is implemented

          Show
          Zhijie Shen added a comment - Hi folks, I've just attached a diagram "Timeline Security Diagram.pdf" to demonstrate the rough workflow of the the timeline security. In general, it consists of two parts: authentication and the authorization. 1. Authentication a) When the authentication is enabled, a customized authentication filter will be loaded into the webapp of the timeline server, which prevents unauthorized users to access any timeline web resources. The filter allow users to: negotiate the authentication via HTTP SPNEGO, and login with Kerberos principal and keytab; and request a delegation token after Kerberos login and use it for follow-up secured communication. b) TimelineClient is adapted to pass the authentication before putting the timeline data. It can choose append the Kerberos token or delegation token into the HTTP request. The rationale behind supporting delegation token is to allow AM and other containers to use TimelineClient to put the timeline data in a secured manner, where the Kerberos stuff is not available. c) TimelineClient also has the API to get the delegation token from the timeline sever (actually from the customized authentication filter). When security is enabled and the timeline service is enabled, and YarnClient is used to submit an application, YarnClient will automatically call TimeClient to get a delegation token and put into the application submission context, such that the AM can used the passed-in delegation token to communicate with the timeline server securely. d) Any tool which support SPNEGO/Kerberos, such as Firefox, curl and etc., can access the three GET APIs of the timeline server to inquiry the timeline data. 2. Authorization Once the request from an authenticated user passes the customized authentication filter, it will be processed by the timeline web services. Here we use the ACLs manager to determine whether the user of the request has the access to the requested data. The basic rules are as follows: The access control granularity is entity, which means a user can access all the information of any entity and its events, or he/she can access nothing of it. Currently we only allow the owner of the entity to access it. In the future, we can simply extend the rule to allow Admin and users/groups on the access control list. Configuration After all, to enable the timeline security, we need to setup Kerberos. In addition, there're a bunch of configurations to do: Make use of the filter initializer to setup the customized authentication filter, and the configuration is much like hadoop-auth style; and ACLs is controlled by YARN ACLs configuration like other YARN daemons. I also uploaded my newest uber patch "Timeline_Kerberos_DT_ACLs.2.patch" to demonstrate how the design is implemented
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12645845/Timeline_Kerberos_DT_ACLs.2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:

          org.apache.hadoop.yarn.client.TestRMAdminCLI

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3776//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3776//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645845/Timeline_Kerberos_DT_ACLs.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice: org.apache.hadoop.yarn.client.TestRMAdminCLI +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3776//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3776//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Canceling the uber patch. The individual patches are getting committed separately..

          Show
          Vinod Kumar Vavilapalli added a comment - Canceling the uber patch. The individual patches are getting committed separately..
          Hide
          Hitesh Shah added a comment -

          Vinod Kumar Vavilapalli Zhijie Shen Wasn't most of the secure support for timeline with respect to application data already introduced in 2.5 and 2.6? If yes, does this jira need to be closed out as it confuses users as to whether Timeline is/isn't supported in a secure environment?

          Show
          Hitesh Shah added a comment - Vinod Kumar Vavilapalli Zhijie Shen Wasn't most of the secure support for timeline with respect to application data already introduced in 2.5 and 2.6? If yes, does this jira need to be closed out as it confuses users as to whether Timeline is/isn't supported in a secure environment?
          Hide
          Zhijie Shen added a comment - - edited

          Close the umbrella jira as the security work is almost done during 2.5 and 2.6. The only left issue is to put generic history data in a non-default domain in secure scenario. Since we don't go on to develop new feature for ATS v1, we can leave that jira (YARN-2622) open and see if we have the supporting requirement for it.

          Show
          Zhijie Shen added a comment - - edited Close the umbrella jira as the security work is almost done during 2.5 and 2.6. The only left issue is to put generic history data in a non-default domain in secure scenario. Since we don't go on to develop new feature for ATS v1, we can leave that jira ( YARN-2622 ) open and see if we have the supporting requirement for it.

            People

            • Assignee:
              Zhijie Shen
              Reporter:
              Arun C Murthy
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development