So. Many. Questions. And stop editing comments hours later!
I'm going to address the ordering issue, first. You are correct that, according to the spec, the order that SCIs are executed in is unspecified. This is an oversight that I intend to lobby hard to have corrected in Servlet 3.2 (and judging by the Tomcat mailing list, I'll have some backup). However, in practice, this is not actually the case. If you read the responses to the message you link to, you will see that Tomcat orders SCIs according to the web-fragment ordering. This is legal, because the spec doesn't prohibit it, it just doesn't require it. Furthermore, five of the eight major containers (Tomcat, TomEE, JBoss, WebLogic, and WebSphere) order SCIs according to the web-fragment ordering. After further inspection, it appears that GlassFish, Jetty, and Resin do execute them in "random" order (ugh). However, with that said, this "random" order is actually alphabetic by JAR name, and log4j*.jar comes before spring*.jar. So, most users will never see a problem with ordering.
(Yes, I'm operating under the assumption that Spring's SCI is the one we're most concerned about. I've yet to see any other SCIs in any other third-party libraries or in containers that could result in inadvertent initialization of Log4j. I am, nevertheless, filing improvement requests with GlassFish, Jetty, and Resin asking them to follow what other containers do.)
With that out of the way, I'm not saying it's "invalid to specify a servlet context listener in the web.xml" in Servlet 3.0+. I'm saying it's invalid (but harmless) to re-define Log4j's listener in web.xml in Servlet 3.0+. I'm not completely opposed to changing the exception to a warning, but I don't agree with the idea, either. Added in the SCI, it ensures that the filter runs before any filters in web.xml. It also ensures that it runs before any filters in other SCIs (with the ordering issue kept in mind, of course). The exception lets the user know in no uncertain terms that Log4j might not work correctly because the filter wasn't defined early enough, and that they can very simply fix the issue by taking the filter out of their web.xml. That's my stance.
Web applications cannot define their own ServletContainerInitializer implementations without putting them in a JAR file within WEB-INF/lib, so we can't reasonably expect a user to do this. An annotation is a potential idea, but I just don't see the necessity. The existing code is capable of initializing correctly. Just because there was a bug doesn't mean the whole concept is fatally flawed. I strongly stand behind the idea of the user not having to do anything other than add log4j2.xml to their application unless they need to customize behavior (which they can do using the context parameters; they don't need to override the existing initializer to do this). If they really need to stop Log4j from auto-initializing in Servlet 3.0+, we don't have to provide a mechanism to support this. They can use <absolute-ordering> in web.xml to exclude the Log4j web-fragment. I'm okay with adding an example of doing this to the documentation.
I do agree that the initializer should not do anything if the Log4j JARs come from the container and the application isn't actually using Log4j. What is the correct way to detect this? I believe this is sufficient, but I could be wrong:
- If the Log4j context parameters are specified, the application obviously intends to use Log4j, so initialize it.
- If the Log4j context parameters are not specified but one of the standard configuration files is present, the application obviously intends to use Log4j, so initialize it.
- Otherwise, don't initialize Log4j.
If this is correct, this leads me to a question. How do I look for the "standard configuration" without Log4j initializing the null or default configuration if it doesn't find one?