Hadoop Common
  1. Hadoop Common
  2. HADOOP-10671

Unify and simplify common configurations for authentication filters between web console and web hdfs

    Details

    • Type: Improvement Improvement
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: security
    • Labels:

      Description

      Currently it's not able to single sign on between hadoop web console and webhdfs since they don't share common configurations as required to, such as signature secret to sign authenticaton token, and domain cookie etc. This improvement would allow sso between the two, and also simplify the configuration by removing the duplicate effort for the two parts.

      The sso makes sense because in current web console, it integrates webhdfs and we should avoid redundant sign on in different mechanisms. This is necessary when a certain authentication mechanism other than SPNEGO is desired across web console and webhdfs.

      1. HADOOP-10671-v3.patch
        8 kB
        Kai Zheng
      2. hadoop-10671-v2.patch
        3 kB
        Kai Zheng
      3. hadoop-10671.patch
        2 kB
        Kai Zheng

        Issue Links

          Activity

          Hide
          Kai Zheng added a comment -

          Sorry I'm late on this.

          Without this change, the following properties may need be configured for web hdfs, in addition to the similar ones with "hadoop.http" prefix for web UI:

          ### The following properties are for AuthenticationFilter ###
          dfs.web.authentication.type #auth type
          dfs.web.authentication.signature.secret # signature secret string value
          dfs.web.authentication.token.validity
          dfs.web.authentication.cookie.domain
          dfs.web.authentication.cookie.path
          
          #The following properties are for AuthenticationHandlers. It depends on auth type.
          dfs.web.authentication.kerberos.principal
          dfs.web.authentication.kerberos.keytab
          dfs.web.authentication.kerberos.name.rules
          ...
          

          With this change, all the above configuration properties can be avoided if we're using the same auth filter and handler/type with web UI. We only need the ones like the following for both web UI and web hdfs.

          ### The following properties are for AuthenticationFilter ###
          hadoop.http.authentication.type #auth type
          hadoop.http.authentication.signature.secret # signature secret string value
          hadoop.http.authentication.token.validity
          hadoop.http.authentication.cookie.domain
          hadoop.http.authentication.cookie.path
          
          #The following properties are for AuthenticationHandlers. It depends on auth type.
          hadoop.http.authentication.kerberos.principal
          hadoop.http.authentication.kerberos.keytab
          hadoop.http.authentication.kerberos.name.rules
          ...
          

          Makes sense ? Thanks for comments.

          Show
          Kai Zheng added a comment - Sorry I'm late on this. Without this change, the following properties may need be configured for web hdfs, in addition to the similar ones with "hadoop.http" prefix for web UI: ### The following properties are for AuthenticationFilter ### dfs.web.authentication.type #auth type dfs.web.authentication.signature.secret # signature secret string value dfs.web.authentication.token.validity dfs.web.authentication.cookie.domain dfs.web.authentication.cookie.path #The following properties are for AuthenticationHandlers. It depends on auth type. dfs.web.authentication.kerberos.principal dfs.web.authentication.kerberos.keytab dfs.web.authentication.kerberos.name.rules ... With this change, all the above configuration properties can be avoided if we're using the same auth filter and handler/type with web UI. We only need the ones like the following for both web UI and web hdfs. ### The following properties are for AuthenticationFilter ### hadoop.http.authentication.type #auth type hadoop.http.authentication.signature.secret # signature secret string value hadoop.http.authentication.token.validity hadoop.http.authentication.cookie.domain hadoop.http.authentication.cookie.path #The following properties are for AuthenticationHandlers. It depends on auth type. hadoop.http.authentication.kerberos.principal hadoop.http.authentication.kerberos.keytab hadoop.http.authentication.kerberos.name.rules ... Makes sense ? Thanks for comments.
          Hide
          Kai Zheng added a comment -

          Hi Haohui Mai,
          Thanks for your comment and guiding.

          Can you please list all the configurations and then we can discuss what is the best way to move forward ?

          Yes I will list all the affected configuration properties for the discussion in the week.

          To better reflect what's actually done here, I modified the JIRA description. Actually SSO effect between web console and web hdfs is just a result of this change.

          Show
          Kai Zheng added a comment - Hi Haohui Mai , Thanks for your comment and guiding. Can you please list all the configurations and then we can discuss what is the best way to move forward ? Yes I will list all the affected configuration properties for the discussion in the week. To better reflect what's actually done here, I modified the JIRA description. Actually SSO effect between web console and web hdfs is just a result of this change.
          Hide
          Haohui Mai added a comment -

          It is a pain to get multiple auth filters lined up and working today if you have customized authentication mechanism. I think that it is a good direction to go, but I'm concerned about the compatibility issues as it assigns the configuration with new behavior. Can you please list all the configurations and then we can discuss what is the best way to move forward?

          Show
          Haohui Mai added a comment - It is a pain to get multiple auth filters lined up and working today if you have customized authentication mechanism. I think that it is a good direction to go, but I'm concerned about the compatibility issues as it assigns the configuration with new behavior. Can you please list all the configurations and then we can discuss what is the best way to move forward?
          Hide
          Kai Zheng added a comment -

          I have checked the failure test happens random and isn't relevant to this.

          Show
          Kai Zheng added a comment - I have checked the failure test happens random and isn't relevant to this.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702334/HADOOP-10671-v3.patch
          against trunk revision 5af693f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.balancer.TestBalancer

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5838//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5838//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702334/HADOOP-10671-v3.patch against trunk revision 5af693f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.balancer.TestBalancer Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5838//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5838//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Updated the patch, per Alejandro Abdelnur's request adding a test to verify it works as expected.

          Show
          Kai Zheng added a comment - Updated the patch, per Alejandro Abdelnur 's request adding a test to verify it works as expected.
          Hide
          Kai Zheng added a comment -

          Hi Haohui Mai,

          Thanks for your comments. I may need some clarification here. The AuthFilter used by web hdfs is inherited from AuthenticationFilter used by web console, and they both support a set of configurable parameters like signature secret, cookie domain and etc. For users to actually configure such parameters, they need to prepare for two sets of configuration properties, hadoop.http.authentication... and dfs.web.authentication.cookie.domain.... So this patch allows only hadoop.http.authentication... set to be ready for both web console and web hdfs by simply property transforming for web hdfs side, which would not risk and cause incompatible concern. As a good effect of this way, it's possible to enforce the same sign on mechanism with exactly the same configurations for both sides, and the effect is not limited to delegation token mechanism. Please note I'm not solving delegation token specific problem here. I thought it's possible to have more mechanisms as broad web applications do in hadoop web interfaces in future, I'm trying to make the configuration work simplified and unified in a safer way. Hope this clarifying helps. Thanks.

          Show
          Kai Zheng added a comment - Hi Haohui Mai , Thanks for your comments. I may need some clarification here. The AuthFilter used by web hdfs is inherited from AuthenticationFilter used by web console, and they both support a set of configurable parameters like signature secret, cookie domain and etc. For users to actually configure such parameters, they need to prepare for two sets of configuration properties, hadoop.http.authentication... and dfs.web.authentication.cookie.domain... . So this patch allows only hadoop.http.authentication... set to be ready for both web console and web hdfs by simply property transforming for web hdfs side, which would not risk and cause incompatible concern. As a good effect of this way, it's possible to enforce the same sign on mechanism with exactly the same configurations for both sides, and the effect is not limited to delegation token mechanism. Please note I'm not solving delegation token specific problem here. I thought it's possible to have more mechanisms as broad web applications do in hadoop web interfaces in future, I'm trying to make the configuration work simplified and unified in a safer way. Hope this clarifying helps. Thanks.
          Hide
          Haohui Mai added a comment -

          I'm not sure whether this is the right approach to take given the fact that the information of the web console comes from JMX and webhdfs, there are few points to protect all the static files here. Maybe we need to look into (1) unifying the auth filter for JMX and the GET_DELEGATION_TOKEN call in webhdfs, and (2) update the web UI to issue GET_DELEGATION_TOKEN call if required.

          Show
          Haohui Mai added a comment - I'm not sure whether this is the right approach to take given the fact that the information of the web console comes from JMX and webhdfs, there are few points to protect all the static files here. Maybe we need to look into (1) unifying the auth filter for JMX and the GET_DELEGATION_TOKEN call in webhdfs, and (2) update the web UI to issue GET_DELEGATION_TOKEN call if required.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12650768/hadoop-10671-v2.patch
          against trunk revision b18d383.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5823//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5823//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650768/hadoop-10671-v2.patch against trunk revision b18d383. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5823//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5823//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Would anyone help look at this and give more comment ? Thanks.

          Show
          Kai Zheng added a comment - Would anyone help look at this and give more comment ? Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12650768/hadoop-10671-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4081//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4081//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650768/hadoop-10671-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4081//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4081//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Hi Alejandro,

          Thanks for your review and comment for the initial patch. I refined the patch and tested it as follows per your request.
          In core-site.xml, added the following properties:
          <pre>
          <property>
          <name>hadoop.http.authentication.cookie.domain</name>
          <value>hadoop-auth.com</value>
          </property>

          <property>
          <name>dfs.web.authentication.cookie.domain</name>
          <value>dfs-web.com</value>
          </property>
          </pre>

          And noticed that for web console, it picked up and used the value of hadoop-auth.com, for web hdfs, it got the value of dfs-web.com as expected.

          Show
          Kai Zheng added a comment - Hi Alejandro, Thanks for your review and comment for the initial patch. I refined the patch and tested it as follows per your request. In core-site.xml, added the following properties: <pre> <property> <name>hadoop.http.authentication.cookie.domain</name> <value>hadoop-auth.com</value> </property> <property> <name>dfs.web.authentication.cookie.domain</name> <value>dfs-web.com</value> </property> </pre> And noticed that for web console, it picked up and used the value of hadoop-auth.com, for web hdfs, it got the value of dfs-web.com as expected.
          Hide
          Kai Zheng added a comment -

          Updated the patch with some fixes and comments.

          Show
          Kai Zheng added a comment - Updated the patch with some fixes and comments.
          Hide
          Alejandro Abdelnur added a comment -

          would be possible to have a simple tests that show that the properties with hadoop.* properties are picked up as df.web.* and that the precedence works as you indicate?

          Show
          Alejandro Abdelnur added a comment - would be possible to have a simple tests that show that the properties with hadoop.* properties are picked up as df.web.* and that the precedence works as you indicate?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12648920/hadoop-10671.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4029//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4029//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648920/hadoop-10671.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4029//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4029//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Attached a simple patch:

          It allows webhdfs to access configuration properties prefixed with 'hadoop.http.authentication.' for web console, by simply transforming the properties with new prefix of 'dfs.web.authentication.' that can be picked up by webhdfs. Any manually configured properties for webhdfs with 'dfs.web.authentication.' are of higher priority since they can override the ones from web console.

          Show
          Kai Zheng added a comment - Attached a simple patch: It allows webhdfs to access configuration properties prefixed with 'hadoop.http.authentication.' for web console, by simply transforming the properties with new prefix of 'dfs.web.authentication.' that can be picked up by webhdfs. Any manually configured properties for webhdfs with 'dfs.web.authentication.' are of higher priority since they can override the ones from web console.

            People

            • Assignee:
              Kai Zheng
              Reporter:
              Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:

                Development