Hadoop Common
  1. Hadoop Common
  2. HADOOP-10670

Allow AuthenticationFilters to load secret from signature secret files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In Hadoop web console, by using AuthenticationFilterInitializer, it's allowed to configure AuthenticationFilter for the required signature secret by specifying signature.secret.file property. This improvement would also allow this when AuthenticationFilterInitializer isn't used in situations like webhdfs.

      1. HADOOP-10670-v6.patch
        21 kB
        Kai Zheng
      2. HADOOP-10670-v5.patch
        18 kB
        Kai Zheng
      3. HADOOP-10670-v4.patch
        18 kB
        Kai Zheng
      4. hadoop-10670-v3.patch
        12 kB
        Kai Zheng
      5. hadoop-10670-v2.patch
        9 kB
        Kai Zheng
      6. hadoop-10670.patch
        4 kB
        Kai Zheng

        Issue Links

          Activity

          Kai Zheng created issue -
          Kai Zheng made changes -
          Field Original Value New Value
          Attachment hadoop-10670.patch [ 12648916 ]
          Hide
          Kai Zheng added a comment -

          Attached a patch. Changes summary:
          1. Moving signature file reading from AuthenticationFilterInitializer to AuthenticationFilter:

          2. And in AuthenticationFilter, if SIGNATURE_SECRET is configured, then use it; otherwise if SIGNATURE_SECRET_FILE is configured, then use it; otherwise generate a secret as before.

          Show
          Kai Zheng added a comment - Attached a patch. Changes summary: 1. Moving signature file reading from AuthenticationFilterInitializer to AuthenticationFilter: 2. And in AuthenticationFilter, if SIGNATURE_SECRET is configured, then use it; otherwise if SIGNATURE_SECRET_FILE is configured, then use it; otherwise generate a secret as before.
          Kai Zheng made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12648916/hadoop-10670.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4028//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12648916/hadoop-10670.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4028//console This message is automatically generated.
          Kai Zheng made changes -
          Component/s security [ 12312526 ]
          Hide
          Alejandro Abdelnur added a comment -
          • String signatureSecretFile = config.getProperty(SIGNATURE_SECRET_FILE);, the property name should be prefixed with configPrefix +.
          • Using a secret file is more secure than having the secret inline in the configuration. The secret file should have precedence over the inline secret. The inline secret should be deprecated, we should print a warning on that.

          Do we have a testcase for this in the AuthenticationFilterInitializer tests? if so, we should move them to the AuthenticationFilter tests.

          Other than that, looks good.

          Show
          Alejandro Abdelnur added a comment - String signatureSecretFile = config.getProperty(SIGNATURE_SECRET_FILE); , the property name should be prefixed with configPrefix + . Using a secret file is more secure than having the secret inline in the configuration. The secret file should have precedence over the inline secret. The inline secret should be deprecated, we should print a warning on that. Do we have a testcase for this in the AuthenticationFilterInitializer tests? if so, we should move them to the AuthenticationFilter tests. Other than that, looks good.
          Hide
          Kai Zheng added a comment -

          Updated the patch according to review comment with unit tests.

          Show
          Kai Zheng added a comment - Updated the patch according to review comment with unit tests.
          Kai Zheng made changes -
          Attachment hadoop-10670-v2.patch [ 12650776 ]
          Hide
          Kai Zheng added a comment -

          Hi Alejandro,

          All the issues you commented for the initial patch are fixed, and moved the test case accordingly. Would you help review again? Thanks.

          Show
          Kai Zheng added a comment - Hi Alejandro, All the issues you commented for the initial patch are fixed, and moved the test case accordingly. Would you help review again? Thanks.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12650776/hadoop-10670-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4082//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4082//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650776/hadoop-10670-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4082//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4082//console This message is automatically generated.
          Hide
          Zhijie Shen added a comment -

          1. One question: Is configPrefix still necessary here? Other properties seem not to require this prefix.

          +    String signatureSecretFile = config.getProperty(configPrefix + SIGNATURE_SECRET_FILE);
          
          +      signatureSecret = config.getProperty(configPrefix + SIGNATURE_SECRET);
          

          And when we get the config, we already filter properties by the prefix, don't we?

              configPrefix = (configPrefix != null) ? configPrefix + "." : "";
              Properties config = getConfiguration(configPrefix, filterConfig);
          

          2. Close it in the finally block? And use IOUtils?

          +        reader.close();
          

          3. This patch may affect TimelineAuthenticationFilterInitializer as well.

          Show
          Zhijie Shen added a comment - 1. One question: Is configPrefix still necessary here? Other properties seem not to require this prefix. + String signatureSecretFile = config.getProperty(configPrefix + SIGNATURE_SECRET_FILE); + signatureSecret = config.getProperty(configPrefix + SIGNATURE_SECRET); And when we get the config, we already filter properties by the prefix, don't we? configPrefix = (configPrefix != null ) ? configPrefix + "." : ""; Properties config = getConfiguration(configPrefix, filterConfig); 2. Close it in the finally block? And use IOUtils? + reader.close(); 3. This patch may affect TimelineAuthenticationFilterInitializer as well.
          Hide
          Kai Zheng added a comment -

          Zhijie, thanks for your review.

          1. I would agree configPrefix might not be needed here since the prefix has already been processed and removed. But I'm not very sure about this. Would you think could it be better to clean this up in a separate JIRA, since this JIRA would focus on the signature configuration stuff.

          2. Yes I agree we can use IOUtils to improve and fix the moved codes.

          3. What do you mean by 'affect'? Good or bad? Could you help clarify since I'm not familiar with it. Thanks.

          Show
          Kai Zheng added a comment - Zhijie, thanks for your review. 1. I would agree configPrefix might not be needed here since the prefix has already been processed and removed. But I'm not very sure about this. Would you think could it be better to clean this up in a separate JIRA, since this JIRA would focus on the signature configuration stuff. 2. Yes I agree we can use IOUtils to improve and fix the moved codes. 3. What do you mean by 'affect'? Good or bad? Could you help clarify since I'm not familiar with it. Thanks.
          Hide
          Zhijie Shen added a comment -

          But I'm not very sure about this

          Neither do I. Alejandro Abdelnur, would you mind commenting on this?

          What do you mean by 'affect'? Good or bad?

          Basically I duplicate TimelineAuthenticationFilterInitializer with AuthenticationFilterInitializer, and make minor touch to change the prefix and accept empty secret. By moving the logic of loading the secret file into AuthenticationFilter, you may also want to remove the analog part in TimelineAuthenticationFilterInitializer as well.

          BTW, can we close HADOOP-10600 as duplicate of this jira?

          Show
          Zhijie Shen added a comment - But I'm not very sure about this Neither do I. Alejandro Abdelnur , would you mind commenting on this? What do you mean by 'affect'? Good or bad? Basically I duplicate TimelineAuthenticationFilterInitializer with AuthenticationFilterInitializer, and make minor touch to change the prefix and accept empty secret. By moving the logic of loading the secret file into AuthenticationFilter, you may also want to remove the analog part in TimelineAuthenticationFilterInitializer as well. BTW, can we close HADOOP-10600 as duplicate of this jira?
          Hide
          Kai Zheng added a comment -

          Zhijie, I resolved HADOOP-10600 as a duplicate of this per your request.

          Will update the patch according to your comments.

          Show
          Kai Zheng added a comment - Zhijie, I resolved HADOOP-10600 as a duplicate of this per your request. Will update the patch according to your comments.
          Kai Zheng made changes -
          Attachment hadoop-10670-v3.patch [ 12659867 ]
          Hide
          Kai Zheng added a comment -

          Updated the patch with the following changes per the review comments:
          1. Close the reader in finally block. I didn't use IOUtils due to inappropriate deps.
          2. Also changed TimelineAuthenticationFilterInitializer similarly.

          Show
          Kai Zheng added a comment - Updated the patch with the following changes per the review comments: 1. Close the reader in finally block. I didn't use IOUtils due to inappropriate deps. 2. Also changed TimelineAuthenticationFilterInitializer similarly.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12659867/hadoop-10670-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice:

          org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler
          org.apache.hadoop.security.authentication.util.TestKerberosUtil
          org.apache.hadoop.security.authentication.server.TestKerberosAuthenticationHandler
          org.apache.hadoop.ipc.TestRPCCallBenchmark
          org.apache.hadoop.net.TestNetUtils
          org.apache.hadoop.security.TestSecurityUtil
          org.apache.hadoop.crypto.key.TestKeyProviderFactory
          org.apache.hadoop.ipc.TestRPC
          org.apache.hadoop.security.TestDoAsEffectiveUser
          org.apache.hadoop.conf.TestConfiguration
          org.apache.hadoop.ipc.TestIPC

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4428//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4428//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659867/hadoop-10670-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice: org.apache.hadoop.security.authentication.server.TestAltKerberosAuthenticationHandler org.apache.hadoop.security.authentication.util.TestKerberosUtil org.apache.hadoop.security.authentication.server.TestKerberosAuthenticationHandler org.apache.hadoop.ipc.TestRPCCallBenchmark org.apache.hadoop.net.TestNetUtils org.apache.hadoop.security.TestSecurityUtil org.apache.hadoop.crypto.key.TestKeyProviderFactory org.apache.hadoop.ipc.TestRPC org.apache.hadoop.security.TestDoAsEffectiveUser org.apache.hadoop.conf.TestConfiguration org.apache.hadoop.ipc.TestIPC +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4428//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4428//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          I double checked the reported test failures. They can't be recreated in my environment and think they're not relevant to the patch.

          Zhijie, would you help review it once more?
          Thanks.

          Show
          Kai Zheng added a comment - I double checked the reported test failures. They can't be recreated in my environment and think they're not relevant to the patch. Zhijie, would you help review it once more? Thanks.
          Hide
          Kai Zheng added a comment -

          Would anyone give it a more review ? I thought it makes sense to get it in the new release. Thanks.

          Show
          Kai Zheng added a comment - Would anyone give it a more review ? I thought it makes sense to get it in the new release. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12659867/hadoop-10670-v3.patch
          against trunk revision b18d383.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5822//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12659867/hadoop-10670-v3.patch against trunk revision b18d383. -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5822//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          The patch looks like obsolete and I will update it with latest codes. Thanks for triggering this !

          Show
          Kai Zheng added a comment - The patch looks like obsolete and I will update it with latest codes. Thanks for triggering this !
          Hide
          Kai Zheng added a comment -

          Updated the patch according to latest codes.

          Show
          Kai Zheng added a comment - Updated the patch according to latest codes.
          Kai Zheng made changes -
          Attachment HADOOP-10670-v4.patch [ 12702103 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702103/HADOOP-10670-v4.patch
          against trunk revision 742f9d9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//artifact/patchprocess/newPatchFindbugsWarningshadoop-auth.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702103/HADOOP-10670-v4.patch against trunk revision 742f9d9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//artifact/patchprocess/newPatchFindbugsWarningshadoop-auth.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5827//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Checking the Findbugs warnings, looks like I can resolve one, the other two are inherited from existing codes, and intended.

          Show
          Kai Zheng added a comment - Checking the Findbugs warnings, looks like I can resolve one, the other two are inherited from existing codes, and intended.
          Hide
          Kai Zheng added a comment -

          Updated the patch resolving one Findbug warning.

          Show
          Kai Zheng added a comment - Updated the patch resolving one Findbug warning.
          Kai Zheng made changes -
          Attachment HADOOP-10670-v5.patch [ 12702125 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702125/HADOOP-10670-v5.patch
          against trunk revision 9ae7f9e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//artifact/patchprocess/newPatchFindbugsWarningshadoop-auth.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702125/HADOOP-10670-v5.patch against trunk revision 9ae7f9e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-auth hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//artifact/patchprocess/newPatchFindbugsWarningshadoop-auth.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5828//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          As I mentioned above, the new Findbug warnings are inherited from existing codes, and intended.

          Show
          Kai Zheng added a comment - As I mentioned above, the new Findbug warnings are inherited from existing codes, and intended.
          Hide
          Haohui Mai added a comment -

          The approach looks good.

          +      // The precedence from high to low : file, inline string, random
          +      if (signatureSecretFile != null) {
          +        providerClassName = FileSignerSecretProvider.class.getName();
          

          I think the way the code works is a side effect on HADOOP-10868. We do not support inlining the secret in the configuration. Anyone can read the configuration can forge the authentication cookie. This is a security vulnerability since the Hadoop configuration is readable by both servers and clients. We have similar issues in NFS / LDAP and we store the secret / credentials in a separate file and guard them by setting the permissions properly.

          We should remove StringSecretProvider once we have FileSecretProvider. Robert Kanter, can you comment on this?

          I think the patch also needs to remove the duplicated code RMAuthenticationFilterInitializer as well.

          Show
          Haohui Mai added a comment - The approach looks good. + // The precedence from high to low : file, inline string, random + if (signatureSecretFile != null ) { + providerClassName = FileSignerSecretProvider.class.getName(); I think the way the code works is a side effect on HADOOP-10868 . We do not support inlining the secret in the configuration. Anyone can read the configuration can forge the authentication cookie. This is a security vulnerability since the Hadoop configuration is readable by both servers and clients. We have similar issues in NFS / LDAP and we store the secret / credentials in a separate file and guard them by setting the permissions properly. We should remove StringSecretProvider once we have FileSecretProvider . Robert Kanter , can you comment on this? I think the patch also needs to remove the duplicated code RMAuthenticationFilterInitializer as well.
          Hide
          Robert Kanter added a comment -

          It's not really a side effect of HADOOP-10868; it's a side effect of the original implementation, which simply loaded the secret from a config property, or used a random one if not set. HADOOP-10791 added support for pluggable providers (to allow HADOOP-1868 to work), and included StringSignerSecretProvider to be backwards compatible with that setting.

          While I agree that FileSignerSecretProvider is more secure, I'm not sure we can simply remove StringSignerSecretProvider without breaking compatibility. What if we instead deprecate it, log a warning about it not being recommended, and add a note to the docs?

          Show
          Robert Kanter added a comment - It's not really a side effect of HADOOP-10868 ; it's a side effect of the original implementation, which simply loaded the secret from a config property, or used a random one if not set. HADOOP-10791 added support for pluggable providers (to allow HADOOP-1868 to work), and included StringSignerSecretProvider to be backwards compatible with that setting. While I agree that FileSignerSecretProvider is more secure, I'm not sure we can simply remove StringSignerSecretProvider without breaking compatibility. What if we instead deprecate it, log a warning about it not being recommended, and add a note to the docs?
          Hide
          Kai Zheng added a comment -

          Thanks Haohui Mai and Robert Kanter for the review and discussion. I will update accordingly.

          Show
          Kai Zheng added a comment - Thanks Haohui Mai and Robert Kanter for the review and discussion. I will update accordingly.
          Hide
          Kai Zheng added a comment -

          Updated the patch according to review comments.

          Show
          Kai Zheng added a comment - Updated the patch according to review comments.
          Kai Zheng made changes -
          Attachment HADOOP-10670-v6.patch [ 12707196 ]
          Hide
          Kai Zheng added a comment -

          The new patch addressed Haohui Mai's comment about RMAuthenticationFilterInitializer.

          What if we instead deprecate it, log a warning about it not being recommended, and add a note to the docs?

          Good idea. I forgot to incorporate this in the patch, so let me do it tomorrow in another version. Thanks.

          Show
          Kai Zheng added a comment - The new patch addressed Haohui Mai 's comment about RMAuthenticationFilterInitializer. What if we instead deprecate it, log a warning about it not being recommended, and add a note to the docs? Good idea. I forgot to incorporate this in the patch, so let me do it tomorrow in another version. Thanks.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707196/HADOOP-10670-v6.patch
          against trunk revision e556198.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-archives.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5997//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5997//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707196/HADOOP-10670-v6.patch against trunk revision e556198. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-archives. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/5997//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/5997//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          it's a side effect of the original implementation, which simply loaded the secret from a config property, or used a random one if not set.

          My understanding is that the use case of inlining the secret is never supported. The property is used to pass the secret internally. The way it works before HADOOP-10868 is the following:

          • Users specify the initializer of the authentication filter in the configuration.
          • AuthenticationFilterInitializer reads the secret file. The server will not start if the secret file does not exists. The initializer will set the property if it read the file correctly.
          • There is no way to specify the secret in the configuration out-of-the-box – the secret is always overwritten by AuthenticationFilterInitializer.

          It looks like that there might be some misunderstandings in the above work flow in HADOOP-10868. We can remove StringSecretProvider in a separate jira. Robert Kanter what do you think?

          Show
          Haohui Mai added a comment - it's a side effect of the original implementation, which simply loaded the secret from a config property, or used a random one if not set. My understanding is that the use case of inlining the secret is never supported. The property is used to pass the secret internally. The way it works before HADOOP-10868 is the following: Users specify the initializer of the authentication filter in the configuration. AuthenticationFilterInitializer reads the secret file. The server will not start if the secret file does not exists. The initializer will set the property if it read the file correctly. There is no way to specify the secret in the configuration out-of-the-box – the secret is always overwritten by AuthenticationFilterInitializer . It looks like that there might be some misunderstandings in the above work flow in HADOOP-10868 . We can remove StringSecretProvider in a separate jira. Robert Kanter what do you think?
          Hide
          Haohui Mai added a comment -

          +1. I'll commit it shortly.

          Show
          Haohui Mai added a comment - +1. I'll commit it shortly.
          Haohui Mai made changes -
          Summary Allow AuthenticationFilter to respect signature secret file even without AuthenticationFilterInitializer Allow all AuthenticationFilter to load secret from signature secret files
          Haohui Mai made changes -
          Summary Allow all AuthenticationFilter to load secret from signature secret files Allow AuthenticationFilters to load secret from signature secret files
          Hide
          Haohui Mai added a comment -

          I've committed the patch to trunk and branch-2. Thanks for the reviews and the contribution.

          Show
          Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks for the reviews and the contribution.
          Haohui Mai made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.7.0 [ 12327583 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7434 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7434/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7434 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7434/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          Hide
          Charles Lamb added a comment -

          Haohui Mai,Kai Zheng,

          Is it possible that something didn't get committed with this patch?

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-auth: Compilation failure: Compilation failure:
          [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java:[287,29] cannot find symbol
          [ERROR] symbol: class FileSignerSecretProvider
          [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter
          [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java:[301,29] cannot find symbol
          [ERROR] symbol: class FileSignerSecretProvider
          [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter

          Show
          Charles Lamb added a comment - Haohui Mai , Kai Zheng , Is it possible that something didn't get committed with this patch? [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-auth: Compilation failure: Compilation failure: [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java: [287,29] cannot find symbol [ERROR] symbol: class FileSignerSecretProvider [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter [ERROR] /disk2/spare8/hadoop/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java: [301,29] cannot find symbol [ERROR] symbol: class FileSignerSecretProvider [ERROR] location: class org.apache.hadoop.security.authentication.server.AuthenticationFilter
          Hide
          Haohui Mai added a comment -

          Thanks for the heads up. It should be fixed by now. I also pushed the changes to 2.7.

          Show
          Haohui Mai added a comment - Thanks for the heads up. It should be fixed by now. I also pushed the changes to 2.7.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #7435 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7435/)
          Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)

          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7435 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7435/ ) Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Hide
          Robert Kanter added a comment -

          I took a look at AuthenticationFilterInitializer and you're right; that appears to be how it worked before. When working on HADOOP-10868 and HADOOP-10791, I didn't realize that. The motivation for that work had been Oozie HA, and we didn't have the code in AuthenticationFilterInitializer, so it had previously been reading a secret string directly from the configuration (which I know is bad). It's good that we have that sorted out now though

          Show
          Robert Kanter added a comment - I took a look at AuthenticationFilterInitializer and you're right; that appears to be how it worked before. When working on HADOOP-10868 and HADOOP-10791 , I didn't realize that. The motivation for that work had been Oozie HA, and we didn't have the code in AuthenticationFilterInitializer , so it had previously been reading a secret string directly from the configuration (which I know is bad). It's good that we have that sorted out now though
          Haohui Mai made changes -
          Link This issue is related to HADOOP-11748 [ HADOOP-11748 ]
          Hide
          Kai Zheng added a comment -

          Thanks Haohui Mai a lot for the help !

          Show
          Kai Zheng added a comment - Thanks Haohui Mai a lot for the help !
          Hide
          Sangjin Lee added a comment -

          I believe this breaks the RM. Prior to this JIRA, RMAuthenticationFilterInitializer threw an exception only if security was enabled (see l.99):

          95	      } catch (IOException ex) {		
          96	        // if running in non-secure mode, this filter only gets added		
          97	        // because the user has not setup his own filter so just generate		
          98	        // a random secret. in secure mode, the user needs to setup security		
          99	        if (UserGroupInformation.isSecurityEnabled()) {		
          100	          throw new RuntimeException(		
          101	            "Could not read HTTP signature secret file: " + signatureSecretFile);		
          102	        }		
          103	      } finally {		
          104	        IOUtils.closeQuietly(reader);		
          105	      }		
          

          Now it appears that this check has been removed.

          Show
          Sangjin Lee added a comment - I believe this breaks the RM. Prior to this JIRA, RMAuthenticationFilterInitializer threw an exception only if security was enabled (see l.99): 95 } catch (IOException ex) { 96 // if running in non-secure mode, this filter only gets added 97 // because the user has not setup his own filter so just generate 98 // a random secret. in secure mode, the user needs to setup security 99 if (UserGroupInformation.isSecurityEnabled()) { 100 throw new RuntimeException( 101 "Could not read HTTP signature secret file: " + signatureSecretFile); 102 } 103 } finally { 104 IOUtils.closeQuietly(reader); 105 } Now it appears that this check has been removed.
          Hide
          Kai Zheng added a comment -

          Hi Sangjin Lee,

          Thanks for your finding. My question is, why we have an IOException there ? In current codes (after the checkin), it may not run into this situation since the signature file is checked before reading it. Do we have any test failure for this ? Thanks.

          Show
          Kai Zheng added a comment - Hi Sangjin Lee , Thanks for your finding. My question is, why we have an IOException there ? In current codes (after the checkin), it may not run into this situation since the signature file is checked before reading it. Do we have any test failure for this ? Thanks.
          Hide
          Sangjin Lee added a comment -

          I stumbled upon this while running unit tests that start the mini YARN cluster. But I just verified the real issue by doing a build off of the trunk and starting a pseudo-distributed cluster. Basically RM fails to start in the non-secure mode with the following exception:

          2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: javax.servlet.ServletException: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
          2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster}
          javax.servlet.ServletException: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
          	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266)
          	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225)
          	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161)
          	at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)
          	at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
          	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
          	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
          	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
          	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
          	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
          	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
          	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
          	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
          	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
          	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
          	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
          	at org.mortbay.jetty.Server.doStart(Server.java:224)
          	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
          	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773)
          	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
          	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
          Caused by: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret
          	at org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59)
          	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264)
          	... 23 more
          ...
          2015-03-25 22:02:42,538 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
          org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server
          	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074)
          	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          	at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208)
          Caused by: java.io.IOException: Problem in starting http server. Server handlers failed
          	at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:785)
          	at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274)
          	... 4 more
          

          I suspect the same failure on branch-2.

          Note that core-default.xml has the property defined:

          <property>
            <name>hadoop.http.authentication.signature.secret.file</name>
            <value>${user.home}/hadoop-http-auth-signature-secret</value>
            <description>
              The signature secret for signing the authentication tokens.
              The same secret should be used for JT/NN/DN/TT configurations.
            </description>
          </property>
          
          Show
          Sangjin Lee added a comment - I stumbled upon this while running unit tests that start the mini YARN cluster. But I just verified the real issue by doing a build off of the trunk and starting a pseudo-distributed cluster. Basically RM fails to start in the non-secure mode with the following exception: 2015-03-25 22:02:42,526 WARN org.mortbay.log: failed RMAuthenticationFilter: javax.servlet.ServletException: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret 2015-03-25 22:02:42,526 WARN org.mortbay.log: Failed startup of context org.mortbay.jetty.webapp.WebAppContext@6de50b08{/,jar:file:/Users/sjlee/hadoop-3.0.0-SNAPSHOT/share/hadoop/yarn/hadoop-yarn-common-3.0.0-SNAPSHOT.jar!/webapps/cluster} javax.servlet.ServletException: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:266) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:225) at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:161) at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53) at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713) at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282) at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518) at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152) at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) at org.mortbay.jetty.Server.doStart(Server.java:224) at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:773) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208) Caused by: java.lang.RuntimeException: Could not read signature secret file: /Users/sjlee/hadoop-http-auth-signature-secret at org.apache.hadoop.security.authentication.util.FileSignerSecretProvider.init(FileSignerSecretProvider.java:59) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeSecretProvider(AuthenticationFilter.java:264) ... 23 more ... 2015-03-25 22:02:42,538 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager org.apache.hadoop.yarn.webapp.WebAppException: Error starting http server at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:279) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startWepApp(ResourceManager.java:974) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1074) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1208) Caused by: java.io.IOException: Problem in starting http server. Server handlers failed at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:785) at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:274) ... 4 more I suspect the same failure on branch-2. Note that core-default.xml has the property defined: <property> <name>hadoop.http.authentication.signature.secret.file</name> <value>${user.home}/hadoop-http-auth-signature-secret</value> <description> The signature secret for signing the authentication tokens. The same secret should be used for JT/NN/DN/TT configurations. </description> </property>
          Sangjin Lee made changes -
          Link This issue breaks HADOOP-11754 [ HADOOP-11754 ]
          Hide
          Kai Zheng added a comment -

          Thanks for your helpful information. You're right there can be problem introduced by this work. The work here assumes if the property hadoop.http.authentication.signature.secret.file is set, then the file should be there, and it will attempt to read the file. If the file isn't there, it will report the exception as you attached. Maybe we can have more check, not only checking the property, but also checking the file should be there available for reading ? Kinds of little awkward, but should fix the problem. Another clean approach is we could avoid having the default property value ?

          Show
          Kai Zheng added a comment - Thanks for your helpful information. You're right there can be problem introduced by this work. The work here assumes if the property hadoop.http.authentication.signature.secret.file is set, then the file should be there, and it will attempt to read the file. If the file isn't there, it will report the exception as you attached. Maybe we can have more check, not only checking the property, but also checking the file should be there available for reading ? Kinds of little awkward, but should fix the problem. Another clean approach is we could avoid having the default property value ?
          Hide
          Haohui Mai added a comment -

          The current behavior of the filter initializer is expected – it should bail out if the secret file is unavailable.

          I believe the right thing to do is to modify the RM to avoid binding the filter when it is not in the secure mode.

          Show
          Haohui Mai added a comment - The current behavior of the filter initializer is expected – it should bail out if the secret file is unavailable. I believe the right thing to do is to modify the RM to avoid binding the filter when it is not in the secure mode.
          Hide
          Kai Zheng added a comment -

          I totally agree. It would be great if RM can change in that way. Not sure how easy it would be.

          Show
          Kai Zheng added a comment - I totally agree. It would be great if RM can change in that way. Not sure how easy it would be.
          Hide
          Kai Zheng added a comment -

          Not binding the filter when not in secure mode in RM may be a little risk. How about this, remove the signature file property instead ? I thought it's easy done in much safer.

          Show
          Kai Zheng added a comment - Not binding the filter when not in secure mode in RM may be a little risk. How about this, remove the signature file property instead ? I thought it's easy done in much safer.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #144 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/144/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #144 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/144/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #878 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/878/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #878 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/878/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/144/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #144 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/144/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2094 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2094/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2094 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2094/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2076 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2076/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2076 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2076/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/CHANGES.txt Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/135/)
          HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
            Addendum for HADOOP-10670. (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157)
          • hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java
          • hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/135/ ) HADOOP-10670 . Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. (wheat9: rev e4b8d9e72d54d4725bf2a902452459b6b243b2e9) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/AuthenticationFilterInitializer.java hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/server/TestAuthenticationFilter.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/server/AuthenticationFilter.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestAuthenticationFilter.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/security/http/RMAuthenticationFilterInitializer.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java Addendum for HADOOP-10670 . (wheat9: rev 3807884263f859f0aaf6a7cbf0009ffc6543c157) hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/util/TestFileSignerSecretProvider.java hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/FileSignerSecretProvider.java
          Hide
          Junping Du added a comment -

          Gentlemen, just tracing from YARN test failures (TestDistributedShell) and found that this patch break RM get started in insecure model which is very risky to 2.7. I just filed HADOOP-11763 and deliver a quick patch to fix it (comment out the default value of "hadoop.http.authentication.signature.secret.file").
          I am not sure if we can find some better way (like comments above - "modify the RM to avoid binding the filter when it is not in the secure mode") quickly. If not, let's go with the easy way like HADOOP-11763, or we should revert the change here for 2.7 release.
          CC to Vinod Kumar Vavilapalli.

          Show
          Junping Du added a comment - Gentlemen, just tracing from YARN test failures (TestDistributedShell) and found that this patch break RM get started in insecure model which is very risky to 2.7. I just filed HADOOP-11763 and deliver a quick patch to fix it (comment out the default value of "hadoop.http.authentication.signature.secret.file"). I am not sure if we can find some better way (like comments above - "modify the RM to avoid binding the filter when it is not in the secure mode") quickly. If not, let's go with the easy way like HADOOP-11763 , or we should revert the change here for 2.7 release. CC to Vinod Kumar Vavilapalli .
          Junping Du made changes -
          Link This issue is related to HADOOP-11763 [ HADOOP-11763 ]
          Hide
          Junping Du added a comment -

          Just found that HADOOP-11754 already there. Mark HADOOP-11763 as duplicated.

          Show
          Junping Du added a comment - Just found that HADOOP-11754 already there. Mark HADOOP-11763 as duplicated.
          Vinod Kumar Vavilapalli made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Benoy Antony made changes -
          Link This issue relates to HADOOP-11567 [ HADOOP-11567 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          15m 36s 1 Kai Zheng 09/Jun/14 06:44
          Patch Available Patch Available Resolved Resolved
          289d 12h 29m 1 Haohui Mai 25/Mar/15 18:13
          Resolved Resolved Closed Closed
          30d 4h 35m 1 Vinod Kumar Vavilapalli 24/Apr/15 23:49

            People

            • Assignee:
              Kai Zheng
              Reporter:
              Kai Zheng
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development