While looking at HADOOP-10895, we discovered that the KerberosAuthenticator can authenticate with a non-secure cluster, even without falling back.
The problematic code is here:
Sometimes the JVM does the SPNEGO for us, and path A is used. However, if the KerberosAuthenticator tries to talk to a non-secure cluster, path A also succeeds in this case.
More details can be found in this comment:
We've actually dealt with this before.
HADOOP-8883 tried to fix a related problem by adding another condition to path A that would look for a header. However, the JVM hides this header, making path A never occur. We reverted this change in HADOOP-10078, and didn't realize that there was still a problem until now.