Hi Jian He,
Correct me if i am wrong , Existing ClientRMService.isAllowedDelegationTokenOp() doing the UserGroupInformation.isSecurityEnabled() is not sufficient ?

HI Jian He/Naganarasimha G R

Seems like check is wrong in isAllowedDelegationTokenOp

    if (UserGroupInformation.isSecurityEnabled()) {
      return EnumSet.of(AuthenticationMethod.KERBEROS,
                        AuthenticationMethod.KERBEROS_SSL,
                        AuthenticationMethod.CERTIFICATE)
          .contains(UserGroupInformation.getCurrentUser()
                  .getRealAuthenticationMethod());
    } else {
      return true;
    }

Else case should have returned false rt?

Yes Bibin A Chundatt you are right, check is there but else case should return false, Missed to see this !.

Uploading patch for the same if you have started working on it please do reassign.

-1 overall

This message was automatically generated.

Hi Jian He
Is this change as expected by you?
Any change required other than the above is required?
Looks like testcases needs lot of correction.

Is there any actual harm in returning a useless delegation token? I know on the HDFS side of the house, returning null tokens has been extremely beneficial in streamlining the code.

Yes, there is. For example, oozie grabs this token in insecure mode and pass the token around in insecure mode which actually breaks in some places.

That sounds like a bigger set of bugs than not issuing delegation tokens....

Hi Jian He

Attaching patch after testcase updation.
TestRMWebServicesDelegationTokens havnt corrected yet.
In nonsecure mode what should be the behaviour for RMWebServicesDelegationTokens.

Currently it will be 500 Internal Error

-1 overall

This message was automatically generated.

yes, oozie has fixed its own. This is just YARN side fix.

-1 overall

This message was automatically generated.

+1 overall

This message was automatically generated.

Bibin A Chundatt, thanks for working on this !
calling initializeUserGroupSecureMode everywhere in all related test cases does not seem like an elegant solution. why is this call needed?
Could you do it in a more clean way?

Hi Jian He

The test classes required updation were as below

TestClientRMService
TestRMDelegation
TokensTestRMWebServicesDelegationTokens

All testcases related to token were written in non secure mode.After the change made in fix for disabling token in non secure mode all token related test cases started failing.Because as per the new condition only in secure mode & AuthenticationMethod=kerberos will get the delegation token.

Also TestClientRMService contains testcases with non secure mode(12) and secure(8) mode. For testcases related to token had to create state Kerberos+Secured . Method initializeUserGroupSecureMode was created for the same in which authentication Method as kerberos is set for UserGroupInformation.

UserGroupInformation state had to be set only for few test cases so initialization in before class was not a choice .
Any other suggestion if you are having please do share, will try my best .

You can try using below changes ?

-      UserGroupInformation.createRemoteUser("owner");
+      UserGroupInformation.createRemoteUser("owner", AuthMethod.KERBEROS);
   private static final UserGroupInformation other =
-      UserGroupInformation.createRemoteUser("other");
+      UserGroupInformation.createRemoteUser("other", AuthMethod.KERBEROS);
   private static final UserGroupInformation tester =
-      UserGroupInformation.createRemoteUser("tester");
+      UserGroupInformation.createRemoteUser("tester", AuthMethod.KERBEROS);

Hi Jian He

Thanks for the suggestion

Tested with your suggestion too still its failing. In UserGroupConfiguration will be loaded with non secure mode conf. Tried setting in BeforeClass too. Then all other nonsecure will try to load keytab file will fail.
Can refactor the testclass by spliting token related testcases to different test class.That should be fine rt?

Have tried out separating out i find this the best way to do it

Hi Jian He

Attaching patch after refactoring testcases.Please do review
All testcases are passing fine.

+1 overall

This message was automatically generated.

Committed to trunk and branch-2, thanks Bibin A Chundatt !

FAILURE: Integrated in Hadoop-trunk-Commit #8447 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8447/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java

FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #389 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/389/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk #2331 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2331/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java

FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #383 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/383/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java

SUCCESS: Integrated in Hadoop-Yarn-trunk #1121 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1121/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #368 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/368/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java

FAILURE: Integrated in Hadoop-Hdfs-trunk #2308 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2308/)
YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt (jianhe: rev e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7)

• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesDelegationTokens.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
• hadoop-yarn-project/CHANGES.txt
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestTokenClientRMService.java
• hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestRMDelegationTokens.java

Note that this change has broken Oozie, see YARN-5750. As such I think this minimally needs to be reverted from branch-2.8 and branch-2.

Jason Lowe, I think this jira's intention is not wrong, deletion token is not required for unsecure cluster. why does oozie require the delegation token in unsecure cluster ? This jira was actually opened because of this, previous comment said so. But I don't remember what the exact issue is...

I'm just going off of Robert Kanter's post in the yarn-dev mailing list that says this change breaks Oozie in non-secure cluster deployments. If Oozie was (mostly?) working in a non-secure setup before and now can't submit any kind of workflow successfully after this change then it seems like we need to revert this or risk breaking a major piece of functionality in the Hadoop stack when people upgrade to 2.8.

Oozie had previously been always getting a delegation token, regardless of security. The only difference was that in a non-secure cluster, it would use a hardcoded dummy renewer, and the correct renewer in a secure cluster. It's been that way for many years so I'm not sure why it was done that way, but it has been working with both Hadoop 1.x and 2.x.

The delegationToken retrieved from RM in non-secure cluster is useless from RM's point of view, it's not being used for authentication, and not being renewed by RM.

I was trying to find the original issue that makes me open this issue. couldn't find it.
Anyway, if the consensus is to revert it. we can revert it.

While myself, Jian He and Vinod Kumar Vavilapalli were debugging some Yarn issues with RM delegation tokens, we thought it would be good to stop issuing RM delegation tokens in a non-secure cluster because If the token is there, it is used and the expiration of the token is not handled correctly with no renewers.

We can revert it probably from branch-2 for existing applications like Oozie

Right. It's not really used. The incompatibility is more that a client used to be able to always call JobClient#getDelegationToken regardless of security, but now that will fail on a non-secure cluster. I'd vote for reverting this from 2.8 and branch-2 as it breaks this API, and leaving it for 3.0. Alternatively, we could try looking into having JobClient return a dummy token without talking to Hadoop, but I don't know what side effects that might have.

ok，I'll revert it from 2.8, branch-2, and leave it in trunk

it is now reverted from 2.8 and branch-2

Removed 2.8.0 from the fixed version

Thanks!

I've also marked this as incompatible and put something in the Release Note field. I'll also close YARN-5750.

The general contract for servers is to return null when tokens are not applicable. This violates that contract and throws an exception. How is a generalized client supposed to pre-meditate fetching a token? And how to handle a generic IOE?

I'd rather see this reverted from trunk and never integrated. We've historically had lots of problem with all the security enabled conditionals, which is why one of my multi-year old endeavors is to have tokens always enabled and gut the security conditionals. I've always admired the fact that yarn unconditionally used them... This is a step backwards.

That's a good point Daryn Sharp.
Jian He, what was the motivation for removing the delegation tokens and throwing the exception? The JIRA description doesn't actually say.

One little request, if we do revert this, let's please do so under a new JIRA since this was released in 3.0.0-alpha1. This way the changelog for alpha1 -> alpha2 will show the revert.

what was the motivation for removing the delegation tokens and throwing the exception

I had seen an issue that RM issues the delegation token in unsecure cluster and that caused some issues. However, I couldn't find the exact issue in my database. Venkat Ranganathan and I decided to not issue the token. And the original code itself is to throw the exception if not allowed. There was no intention to change that part of the logic.

Daryn Sharp,

The general contract for servers is to return null when tokens are not applicable. I'd rather see this reverted from trunk and never integrated

The original code also does not return null. It returns a real token. Are you suggesting we should change the code to return null in 3.x ?

Jian He IIRC, the issue is that since the delegation tokens, if present, are used and will cause jobs to fail after the token expiry as there are no renewers. We had a customer complaining about it and the solution was to either break the job into multiple steps or use a secure cluster. Honestly I don't know if any other app relies on getting a delegation token from RM in a unsecure cluster and may be this is isolated to fixing Oozie alone to not request a RM delegation token in a unsecure cluster. And Oozie has several conditionals for handling secure/unsecure).

If we do issue tokens in an unsecure cluster, we should make sure it just is nothing but a dummy token and unused.

FYI, I've filed YARN-5882 as an alpha2 blocker so we have an open JIRA for tracking. We can use it to log the revert if we decide to do that.

Daryn Sharp, what's your suggestion to my last question ? Reverting this patch is not making the API return null.

The original code also does not return null. It returns a real token. Are you suggesting we should change the code to return null in 3.x ?

Ping on this JIRA.

Should we revert while discussion continues? I'd prefer to stick with the 2.x behavior if we're not in agreement, and Daryn's previous comment seemed like a strong -0 if not a -1.

Below is the original code, the logic is also bizarre: what's the point of having this condition guarding against whether security is enabled or not, given that it is allowed in unsecure cluster anyway ? doesn't this look like a bug ?

  private boolean isAllowedDelegationTokenOp() throws IOException {
    if (UserGroupInformation.isSecurityEnabled()) {
      return EnumSet.of(AuthenticationMethod.KERBEROS,
                        AuthenticationMethod.KERBEROS_SSL,
                        AuthenticationMethod.CERTIFICATE)
          .contains(UserGroupInformation.getCurrentUser()
                  .getRealAuthenticationMethod());
    } else {
      return true;
    }
  }

Venkat Ranganathan mentioned there was a jira to fix this in Ozzie itself, do you have the jira number ?

I'd like to revert this change while discussion continues (using YARN-5882 as a release notes vehicle). Assuming that Oozie wants to keep supporting 2.x releases there must be an Oozie side fix, and keeping the 3.x behavior inline with 2.x is a conservative choice.

Assuming that Oozie wants to keep supporting 2.x releases there must be an Oozie side fix,

IIUC, since this is already reverted from branch-2, Oozie no longer requires a fix to support 2.x release.

To support 3.x which has this change. Oozie needs to have a fix.

My point is that the old behavior is a bug – ( the logic is not matching its exception). Exception says delegationToken can only be done in kerberos env, but the logic returns true if it is unsecure env. – Isn't this self-conflicting ?

      if (!isAllowedDelegationTokenOp()) {
        throw new IOException(
            "Delegation Token can be cancelled only with kerberos authentication");
      }

My preference is to keep this in 3.x so that future apps on YARN are not repeating the same mistake as Ozzie, and Ozzie should fix this to support 3.x line
On the other hand, if other folks think it's more important to keep it compatible and with minimal surprise for 3.x, I'm also ok to revert it.

Oozie hasn't officially supported Hadoop 3 yet, so I think we'd okay leaving this in Hadoop 3 and having Oozie fix it on it's side of things. Peter Cseh, you've been looking at this issue more than I have; what do you think?
It should be a pretty simple if statement check, but my only concern is that if Hadoop 3 requires Oozie to check if Kerberos is enabled, and Hadoop 2 doesn't, Oozie will need to know if it's using Hadoop 2 or Hadoop 3, which it currently has no idea. Unless there's some clean and easy way to do that?

I know that Andrew Wang really wants to get alpha2 out very soon, so perhaps we're best off reverting it for now and continuing this discussion for the beta1 release?

ok, let's revert it from trunk.

I've committed a patch to revert this logic to return true if kerberos is not enabled.

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10994/)
Revert YARN-4126. RM should not issue delegation tokens in unsecure (jianhe: rev ada876cd1d22b61f237603cf339bbed65285dab8)

• (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java

Jian He, the commit you did to revert this simply changes the boolean. Shouldn't we do a revert of the whole patch (i.e. the tests)? This is going to be confusing for others to follow what happened.

trunk: https://github.com/apache/hadoop/commit/ada876cd1d22b61f237603cf339bbed65285dab8
branch-2: https://github.com/apache/hadoop/commit/bb8214cab8a239385c2b06ad689d87b82abf6964

The rest are test refactorings which are good to have, IMO

Thanks for reverting Jian He, though in the future anything that's not a full revert should go through code review, and not be committed with a message that says "Revert ...".

I'll use YARN-5882 to log the fact that essentially test-only changes that were made, to trunk/branch-2/branch-2.8. Otherwise the test changes aren't logged by any JIRA.