[AMBARI-2024] Ambari Server becomes unresponsive after crashing on http reads on jersey.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.3
• Fix Version/s: 1.2.3
• Component/s: ambari-server
• Assignee: Siddharth Wagle
• Reporter: Siddharth Wagle
• Created: 25/Apr/13 18:14
• Updated: 08/May/13 05:50
• Resolved: 25/Apr/13 20:09

Description

The api's are being handled by a queuedthreadpool. The queuedthread pool size is 25.

Somehow the http connections are being torn down from the UI side but the server still is hanging onto that socket and reading (most likely UI will also need to close http connections if its not using them - which might be an issue as well but doesnt have to addressed as urgent). The server has a read timeout of 0 which means it will just hang on to that socket for read. This causes all the threads to block at one time or the other. Simple solution is add read timeouts to all the SelectChannelConnector and SslSelectChannelConnector we use.

Exception trace:

SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
org.eclipse.jetty.io.EofException: early EOF
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:65)
at org.codehaus.jackson.impl.ByteSourceBootstrapper.ensureLoaded(ByteSourceBootstrapper.java:507)
at org.codehaus.jackson.impl.ByteSourceBootstrapper.detectEncoding(ByteSourceBootstrapper.java:129)
at org.codehaus.jackson.impl.ByteSourceBootstrapper.constructParser(ByteSourceBootstrapper.java:224)
at org.codehaus.jackson.JsonFactory._createJsonParser(JsonFactory.java:785)
at org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:561)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.readFrom(JacksonJsonProvider.java:414)
at com.sun.jersey.json.impl.provider.entity.JacksonProviderProxy.readFrom(JacksonProviderProxy.java:139)
at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)

Notice the API's is being called all the time - meaning they probalby had a browser up and running for a long time.

There might be a possibilility that the browser might have some issues after running for a long time. Something to keep in mind when this happens again. Easy way to check that is to call Ambari server API's and also bring up a new browser window (new instance) and try hitting the browser UI.