[FELIX-3846] Felix + Felix HTTP Jetty causes IllegalStateException during start up

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: framework-4.0.3, http-2.2.0
• Fix Version/s: http-2.2.1
• Component/s: Framework, HTTP Service
• Environment: Java version: 1.6.0_37, vendor: Apple Inc.; ; All operating systems
• Assignee: Unassigned
• Reporter: Ronald Chen
• Created: 15/Jan/13 02:28
• Updated: 28/Sep/13 17:40
• Resolved: 15/Aug/13 12:47

Description

There is a race condition during the activation of the felix http jetty bundle when used within a felix container.

During the activation of the felix http jetty bundle you will intermittently see:
java.lang.IllegalStateException: Can only register services while bundle is active or activating.
at org.apache.felix.framework.Felix.registerService(Felix.java:3209) ~[na:na]
at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346) ~[na:na]
at org.apache.felix.http.base.internal.HttpServiceController.register(HttpServiceController.java:135) ~[org.apache.felix.http.jetty_2.2.0.jar:na]
at org.apache.felix.http.base.internal.DispatcherServlet.init(DispatcherServlet.java:48) ~[org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.servlet.Context.startContext(Context.java:140) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.jetty.Server.doStart(Server.java:224) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.apache.felix.http.jetty.internal.JettyService.initializeJetty(JettyService.java:164) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.apache.felix.http.jetty.internal.JettyService.startJetty(JettyService.java:115) [org.apache.felix.http.jetty_2.2.0.jar:na]
at org.apache.felix.http.jetty.internal.JettyService.run(JettyService.java:290) [org.apache.felix.http.jetty_2.2.0.jar:na]
at java.lang.Thread.run(Thread.java:680) [na:1.6.0_37]

I debugged into line Felix.java:3209 and discovered the cause by is thrown by Felix.java:4875 which is:
throw new IllegalStateException("Unable to acquire bundle lock, thread interrupted.");

I dug around and discovered the thread which acquired the lock was the one created on JettyService.java:75

Looking at usages of the thread field, I think I discovered the source of the interrupt on line JettyService.java:75.

What I think is happening is this:
1. felix container starts jetty bundle
2. jetty activator is run which creates a new jetty service
3. jetty service creates a new thread and initializes jetty
4. jetty attempts to register the http service to the felix container and obtains the felix bundle lock (Felix.java:4871)
5. at the same time something calls JettyService.update(Dictionary) which interrupts the thread in attempts to restart the jetty server
6. this causes the felix bundle lock to be interrupted and IllegalStateException is thrown

At which point the http service is not registered and we sob quietly.

To fix this issue I think the thread jetty service creates should never leave the jetty space. When it needs to register the http service a new thread should be created so it doesn't allow the possibility of the jetty service thread to obtain a lock it shouldn't.

Alternatively in stop using interrupt() to signal change as it is dangerous! Use a java 5 concurrent lock object.

Repo case:
I can repo this case within my own system by restarting bundle "Apache Felix Http Jetty (2.2.0)" again and again, but I don't know if there is a smaller repo case.

Issue Links

• relates to: ACE-348 Ace server unavailable after (re)start