Bug 54076 - SPNEGO authenticator's stateless-ness incompatible with stateful clients
Summary: SPNEGO authenticator's stateless-ness incompatible with stateful clients
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Catalina (show other bugs)
Version: trunk
Hardware: All All
: P2 major (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-30 13:49 UTC by Michael Osipov
Modified: 2012-11-05 09:37 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Osipov 2012-10-30 13:49:27 UTC
At the moment, Tomcat is not able to store connection-level information to support stateful authentication mechanisms. This has been confirmed on the mailinglist: http://www.mail-archive.com/users@tomcat.apache.org/msg102169.html

Unfortunately, a client which observes that, like Apache HTTPComponents or libserf (used in Apache Subversion) with authenticate only the first the request and expect the server to cache that for the connection. Subsequent comm will end in an endless loop. See here http://serf.googlecode.com/issues/attachment?aid=770006001&name=tomcat-serf-spnego-response.png&token=GDWvY5f7eMDzDGwtg1tD5N_MUXY%3A1351604707042&inline=1 and here http://serf.googlecode.com/issues/attachment?aid=770006002&name=serf-endless-loop.png&token=cHvvfubJuAHDuTMjG_OHOaps5hQ%3A1351604707042&inline=1

A simple hint can tell the client that the server does not support stateful auth on a connection-level. Add here http://svn.apache.org/viewvc/tomcat/tc7.0.x/trunk/java/org/apache/catalina/authenticator/SpnegoAuthenticator.java?view=markup#l272 before line 272 "response.addHeader("Connection", "close");" and the client will open a new connection and reauth further requests.

Not doing so makes the Tomcat server vulnerable to DoS as you can see in the second screenshot.
Comment 1 Mark Thomas 2012-10-30 16:34:59 UTC
Is that really a DoS (in that the server is unavailable to other clients) or do you mean that the client gets stuck in an infinite loop?

For a DoS, the resources used by Tomcat need to be out of proportion to the number of requests. i.e. If a client just sending a request again and again uses roughly the same server resources as this case then it is not a DoS. The report does not (at this point) sound like a DoS.
Comment 2 Michael Osipov 2012-10-30 16:59:36 UTC
Well, I guess the latter applies but I have tried only one client at the same time. I was able to loop client and server within seconds for tens of thousands of requests (as you have seen in the screenshot). I guess if you scale to 100 parallel connections or more what would happen to a Tomcat instance?

May this lead to a DoS?
Comment 3 Mark Thomas 2012-11-03 20:59:57 UTC
There is no evidence of a DoS in this report.
Comment 4 Mark Thomas 2012-11-03 22:38:13 UTC
There is one work-around already available. Set alwaysUseSession on the Authenticator Valve.

I have added support for a second work-around to trunk and 7.0.x. This work-around enables HTTP keep-alive to be disabled for specified user-agents if they attempt to use SPNEGO. This will be included in 7.0.33 onwards.

I'm not a huge fan of adding the ability to cache information per connection as that goes against the stateless nature of HTTP. That said, I'd be prepared to look at a patch that did this and, depending on how invasive it was, would consider such a patch for 8.0.x.
Comment 5 Michael Osipov 2012-11-04 16:52:23 UTC
(In reply to comment #4)
> There is one work-around already available. Set alwaysUseSession on the
> Authenticator Valve.

This isn't even a workaround for me. You cannot guarantee that the client will respond with the JSESSIONID cookie. You could end up with generating a huge amount of empty sessions.
 
> I have added support for a second work-around to trunk and 7.0.x. This
> work-around enables HTTP keep-alive to be disabled for specified user-agents
> if they attempt to use SPNEGO. This will be included in 7.0.33 onwards.

Well, the server admin needs to know the client's UA preemptively. Is this really feasable?
The client cannot know that the server is incapable of performing stateful auth.
I'd rather always write "Connection: close" for general safety.
 
> I'm not a huge fan of adding the ability to cache information per connection
> as that goes against the stateless nature of HTTP. That said, I'd be
> prepared to look at a patch that did this and, depending on how invasive it
> was, would consider such a patch for 8.0.x.

We have discussed this already on the mailing list. Yes, HTTP is stateless but some auth mechs are stateful. This means that HTTP has to be stateful somehow. Since this is done on the connection-level, you already have the statefulness w/o tampering of the HTTP model. Consider that SSL is stateful too and simply wraps HTTP messages.
Comment 6 Mark Thomas 2012-11-04 19:24:34 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > There is one work-around already available. Set alwaysUseSession on the
> > Authenticator Valve.
> 
> This isn't even a workaround for me. You cannot guarantee that the client
> will respond with the JSESSIONID cookie. You could end up with generating a
> huge amount of empty sessions.

While it might not be a valid work-around for you it may well work for others. One of the purposes of Bugzilla is to provide useful information to others that stumble across an issue, not just to fix the issue for the original reporter.

> > I have added support for a second work-around to trunk and 7.0.x. This
> > work-around enables HTTP keep-alive to be disabled for specified user-agents
> > if they attempt to use SPNEGO. This will be included in 7.0.33 onwards.
> 
> Well, the server admin needs to know the client's UA preemptively. Is this
> really feasable?

Yes, in some circumstances.
1. In many environments where SPNEGO is used (I am thinking corporate environments) the user agents are fixed, known and controlled.
2. You can always write the regular expression to not match known 'safe' UAs (an inverse match is harder but not impossible to write).
3. Analysis of your access logs will tell you which user agents you are seeing.

> The client cannot know that the server is incapable of performing stateful
> auth.
> I'd rather always write "Connection: close" for general safety.

If you prefer to significantly reduce performance for all UAs that wish to use SPNEGO then you are free to use ".*" as your regular expression and send "Connection: close" to all UAs.

> > I'm not a huge fan of adding the ability to cache information per connection
> > as that goes against the stateless nature of HTTP. That said, I'd be
> > prepared to look at a patch that did this and, depending on how invasive it
> > was, would consider such a patch for 8.0.x.
> 
> We have discussed this already on the mailing list. Yes, HTTP is stateless
> but some auth mechs are stateful. This means that HTTP has to be stateful
> somehow. Since this is done on the connection-level, you already have the
> statefulness w/o tampering of the HTTP model. Consider that SSL is stateful
> too and simply wraps HTTP messages.

SSL is not SPNEGO. That is comparing apples and oranges. The complication factor with SPNEGO is that the handshake occurs at layer 7 but the caching needs to be at layer 6. With SSL/TLS everything happens at layer 6. I'm not saying it is impossible, nor am I saying Tomcat would never implement such a scheme. I am saying I don't particularly like the idea but am prepared to consider any patch proposed.
Comment 7 Michael Osipov 2012-11-05 09:37:27 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > There is one work-around already available. Set alwaysUseSession on the
> > > Authenticator Valve.
> > 
> > This isn't even a workaround for me. You cannot guarantee that the client
> > will respond with the JSESSIONID cookie. You could end up with generating a
> > huge amount of empty sessions.
> 
> While it might not be a valid work-around for you it may well work for
> others. One of the purposes of Bugzilla is to provide useful information to
> others that stumble across an issue, not just to fix the issue for the
> original reporter.
> 
> > > I have added support for a second work-around to trunk and 7.0.x. This
> > > work-around enables HTTP keep-alive to be disabled for specified user-agents
> > > if they attempt to use SPNEGO. This will be included in 7.0.33 onwards.
> > 
> > Well, the server admin needs to know the client's UA preemptively. Is this
> > really feasable?
> 
> Yes, in some circumstances.
> 1. In many environments where SPNEGO is used (I am thinking corporate
> environments) the user agents are fixed, known and controlled.

I would object at least this one. Given a realistic example: We have more than 50 domains in our forest with around 1000 DCs or more. Try too find someone who is responsible for a buggy server who will alter the config for you. Good luck.