Uploaded image for project: 'CXF'
  1. CXF
  2. CXF-5742

Http Request Payload missing from Apache HTTP Server to Embedded Jetty server

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Cannot Reproduce
    • 2.7.10
    • Invalid
    • JAX-RS
    • None

    • CXF 2.7.10
      Unix platform and JDK 7
      Container being used - Jetty 8

    • Unknown

    Description

      Hello,

      We have a REST web service application based on Spring and CXF.

      The client can send a XML request and receive the response, the interceptors are all fine, and authentication works fine too.

      But We intermittently observing the Request Payload being missed on the Jetty 8 server when tested under a considerable load.

      The Apache HTTP Server(two Apache HTTP servers) is under a load Balancer and the load is further distributed to 4 Jetty Servers.

      We observed "Payload:" header printed in the server logs for the requests that go through successfully. For the requests that failed no Payload Header was observed to be printed in the jetty server Logs.

      Below are the details of the Request and Response.

      Request:
      --------------------------------------
      May 13, 2014 9:34:09 PM org.apache.cxf.interceptor.LoggingInInterceptor
      INFO: Inbound Message
      ----------------------------
      ID: 5234
      Address: https://qa.p.****.com/rest/1/07/Account/urn:****:accountid:org:dece:F94FCC3565718056E0401F0A3E997368/User
      Encoding: UTF-8
      Http-Method: POST
      Content-Type: application/xml
      Headers:

      {Accept=[*/*], Cache-Control=[no-cache], connection=[Keep-Alive], Content-Length=[1252], content-type=[application/xml], cookie=[JSESSIONID=oovyjh0hjty6l16xgbbo8eaq.api4], ****_header=[ t=1400016849110245 U3KP0QofmJEAAA2eCmgAAAEt urn:dece:org:org:dece:700700 (null)], ****_request=[Unique_ID=U3KP0QofmJEAAA2eCmgAAAEt], dynaTrace=[NA=POST_UserCreate;PC=UserCreate;VU=431;ID=1400016848618], host=[qa.p.****.com:7003], SSL_CLIENT_NODEID=[urn:dece:org:org:dece:700700], user-agent=[Jakarta Commons-HttpClient/3.1], VHOST=[provision]}

      --------------------------------------

      Exception:

      May 13, 2014 9:34:09 PM org.apache.cxf.jaxrs.provider.AbstractJAXBProvider handleExceptionStart
      WARNING: javax.xml.bind.UnmarshalException

      • with linked exception:
        [com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
        at [row,col {unknown-source}]: [1,0]]
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.handleStreamException(UnmarshallerImpl.java:436)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:372)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:342)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:272)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:225)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:184)
        at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1280)
        at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBody(JAXRSUtils.java:1231)
        at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameter(JAXRSUtils.java:782)
        at org.apache.cxf.jaxrs.utils.JAXRSUtils.processParameters(JAXRSUtils.java:741)
        at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.processRequest(JAXRSInInterceptor.java:254)
        at org.apache.cxf.jaxrs.interceptor.JAXRSInInterceptor.handleMessage(JAXRSInInterceptor.java:90)
        at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
        at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
        at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
        at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
        at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
        at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
        at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1496)
        at biz.neustar.dece.log.DeceLogRequestInterceptor.doFilter(DeceLogRequestInterceptor.java:352)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1484)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:108)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1476)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
        Caused by: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
        at [row,col {unknown-source}

        ]: [1,0]
        at com.ctc.wstx.sr.StreamScanner.throwUnexpectedEOF(StreamScanner.java:677)
        at com.ctc.wstx.sr.BasicStreamReader.handleEOF(BasicStreamReader.java:2139)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromProlog(BasicStreamReader.java:2045)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1134)
        at com.sun.xml.bind.v2.runtime.unmarshaller.StAXStreamConnector.bridge(StAXStreamConnector.java:164)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:370)
        at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:342)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.unmarshalFromInputStream(JAXBElementProvider.java:272)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.doUnmarshal(JAXBElementProvider.java:225)
        at org.apache.cxf.jaxrs.provider.JAXBElementProvider.readFrom(JAXBElementProvider.java:184)
        ... 47 more

      Response:

      May 13, 2014 9:34:09 PM org.apache.cxf.interceptor.LoggingOutInterceptor
      INFO: Outbound Message
      ---------------------------
      ID: 5234
      Response-Code: 500
      Content-Type: application/xml
      Headers:

      {Connection=[close], x-Transaction-Info=[t=1400016849 U3KP0QofmJEAAA2eCmgAAAEt urn:****:org:org:dece:700700 10.31.153.241], Date=[Tue, 13 May 2014 21:34:09 GMT]}

      Payload: <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
      <ErrorList xmlns="http://www.****.org/schema/**//***" xmlns:ns3="http://www.w3.org/2000/09/xmldsig#" xmlns:ns4="http://www.w3.org/2001/04/xmlenc#">
      <Error ErrorID="urn:****:errorid:org:dece:InternalServerError">
      <Reason Language="en">An internal server error occurred.</Reason>
      <OriginalRequest>https://qa.p.****.com/rest/1/07/Account/urn:****:accountid:org:dece:F94FCC3565718056E0401F0A3E997368/User</OriginalRequest>
      <ErrorLink>https://qa.q.****.com:7003/rest/1/07/error/en/ErrorList.html#InternalServerError</ErrorLink>
      </Error>
      </ErrorList>

      --------------------------------------

      As we observe this issue intermittently, at this moment we don't have a consistent way to reproduce and debug this issue. Please help us resolve this issue.

      Attachments

        Activity

          People

            dkulp Daniel Kulp
            harsha_gudladona sai sri harsha gudladona
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: