You need to upgrade both Tomcat and your version of tc-native to the latest versions and re-test. There have been multiple fixes in this area since the versions you are using.

Please tell me the tomcat and tc-native version it had been fixed.
The system run in production environment, so if I upgrade
the tomcat and tc-native, I have to prove to my customer 
this problem never appear.

https://issues.apache.org/bugzilla/show_bug.cgi?id=51813

In the above ticket, it has been commented that "Please re-open if it appears".

(In reply to Shunsuke Tanaka from comment #2)
> Please tell me the tomcat and tc-native version it had been fixed.
> The system run in production environment, so if I upgrade
> the tomcat and tc-native, I have to prove to my customer 
> this problem never appear.

If you need to prove to your customer that the problem [will] never appear, then that's your job, since they are paying you to get the job done.

> https://issues.apache.org/bugzilla/show_bug.cgi?id=51813
> 
> In the above ticket, it has been commented that "Please re-open if it
> appears".

So why didn't you re-open that ticket if you are having the same problem?

Let me save you some time: the patch went into 1.1.28 but it only tries to prevent the JVM from crashing when the application is doing something it should not be doing (like keeping references to request and response objects). That patch fixes nothing other than the symptom (the JVM crash)... the core problem is misuse of resources.

In reviewing the code (again) for sendbb, there are very few opportunities for a null-pointer de-reference. Are you able to re-build tcnative with debugging symbols and do some back-tracing for me? "sendbb+0x75" can mean different things depending upon the environment.

Seriously considering marking this as a duplicate of bug #51813 and re-opening that one, just because it's got better information and a longer history.

Sorry. You're right.
I looked for fixed report of this problem in Tomcat Changelog expectantly, but I can't find that.
However I got the suggestion Tomcat/Tomcat Native upgrade in comment#1.
I think I overlooked the fixed report, so I asked what version the problem is fixed.
Can I ask you  whether Bug #51813 is the same as the problem I have?
I can't judge that, because there are some differences of Java,OS,Tomcat version.
If same problem, I will mark this as a duplicate of bug #51813.

>Are you able to re-build tcnative with debugging symbols and do some back-tracing for me?

Do you need back-trace of the time when the problem occured?
If yes, it is difficult for me that I change the production environment.
Now I try reproducing the problem in test environment.

(In reply to Shunsuke Tanaka from comment #5)
> Can I ask you  whether Bug #51813 is the same as the problem I have?

The "problem" is that tcnative crashes in the "sendbb" function. The cause is elsewhere, this is just the symptom. So yes, you have the same symptom as reported in bug #51813.

> I can't judge that, because there are some differences of Java,OS,Tomcat
> version.
> If same problem, I will mark this as a duplicate of bug #51813.
> 
> >Are you able to re-build tcnative with debugging symbols and do some back-tracing for me?
> 
> Do you need back-trace of the time when the problem occured?

If I could get that, it would be best. Also please tell me exactly which package you downloaded. You have 1.1.30 shown, but 1.1.32 is the latest. It would be best to build the latest (1.1.30) with debug symbols but if you have to stick with 1.1.30, that's okay too. Just enough to see which of the possibilities this could be.

I'm surprised that the checks already in there aren't protecting you... something deeper must be going on.

> If yes, it is difficult for me that I change the production environment.

If you built tcnative yourself, re-building it with debug symbols should not be a problem... if Tomcat crashes with some regularity, replacing the native library shouldn't represent too much of a burden...

> Now I try reproducing the problem in test environment.

Fair enough. Get me whatever data you can.

(In reply to Mark Thomas from comment #1)
> You need to upgrade both Tomcat and your version of tc-native to the latest
> versions and re-test. There have been multiple fixes in this area since the
> versions you are using.

From Tomcat side there have been the following fixes in the "Coyote" (connector) category:

7.0.46:
55602: Ensure that sockets removed from the Poller and then closed in the APR/native connector are removed and then closed in a thread-safe manner.

7.0.54:
56399: Assert that both Coyote and Catalina request objects have been properly recycled.

Maybe there are other changes as well.

It would be better to test with an up-to-date version rather than with 7.0.40. 
(7.0.59 is currently being voted on and is likely to be released in a few days).

But with comet the cause may actually be in your own web application. Are you sure that you aren't trying to write to a response that have already been completed and recycled?

> Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
> J  org.apache.tomcat.jni.Socket.sendbb(JII)I
> J  org.apache.coyote.http11.InternalAprOutputBuffer.flushBuffer()V
> J 
> org.apache.coyote.http11.AbstractHttp11Processor.action(Lorg/apache/coyote/
> ActionCode;Ljava/lang/Object;)V
> J  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V
> J  org.apache.catalina.connector.Response.flushBuffer()V
> J 
> jp.co.xxxxxxx.raise.app.fw.socketio.transport.XHRStreaming.flushResponse()Z
> J  scala.concurrent.impl.Future$PromiseCompletingRunnable.run()V
> J  akka.dispatch.TaskInvocation.run()V
> J 
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/
> ThreadPoolExecutor$Worker;)V
> j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
> j  java.lang.Thread.run()V+11
> v  ~StubRoutines::call_stub

In the above stack trace - how does it happen that you are directly calling org.apache.catalina.connector.Response method flushBuffer()?

A web application shall not access that connector Response class directly. A application shall use CometEvent.getHttpServletResponse() which returns a facade, not the internal object.


If you set the following system property, the facades are recycled when request processing cycle finishes, protecting Tomcat internals. Using this property will not help you if you are accessing the internals directly.

org.apache.catalina.connector.RECYCLE_FACADES=true

System properties reference:
http://tomcat.apache.org/tomcat-7.0-doc/config/systemprops.html#Security

Christopher, Konstantin
Thank you for your comment.


(In reply to Christopher Schultz from comment #6)
> please tell me exactly which package you downloaded.

I downloaded tomcat-native-1.1.30-src.tar.gz.
I builded and upgraded tcnative from 1.1.27 to 1.1.30 in 2014/04/26.


> I'm surprised that the checks already in there aren't protecting you... something deeper must be going on.

If this problem would be caused by tomcat code(if the tcnative receives the wrong value from Tomcat),
can this symptom occur?
What do you think the following double check?

if(!s->net || s->net != &apr_socket_layer){
  tcn_ThrowAPRException(e, APR_EINVALSOCK);
  return -(jint)APR_EINVALSOCK;
}

If the tcnative receives the wrong value from Tomcat, I think this check works effectively.
And if condition is false and this symptom occur, it seems to me that either of "s->opaque, s->jsbbuff" value is broken.
I'm sorry, if I have to say irrelevant things.


> If you built tcnative yourself, re-building it with debug symbols should not be a problem... 

I have to care performance problem to change debug level of tcnative.


> if Tomcat crashes with some regularity, replacing the native library shouldn't represent too much of a burden...

The symptom occured twice a year in production environment.



(In reply to Konstantin Kolinko from comment #7)
> From Tomcat side there have been the following fixes in the "Coyote" (connector) category:
> 7.0.46:
> 7.0.54:

If this problem would be caused by tomcat thread-safe or recycle problem, as you say,
I have a feeling that this symptom never occur by upgrading Tomcat.


> Are you sure that you aren't trying to write to a response that have already been completed and recycled?

How can I check that you say?


> how does it happen that you are directly calling org.apache.catalina.connector.Response method flushBuffer()?

I have no idea.
The stack trace has been writen into the core dump file when the symptom occured.
When I set breakpoint and step-run, I confirmed that org.apache.catalina.connector.CoyoteWriter is called.


> org.apache.catalina.connector.RECYCLE_FACADES=true

Thank you for your information.
Also in this case, I have to care performance problem to change this parameter.


I will consider that upgrade Tomcat and/or change RECYCLE_FACADES value to true.

We have the same issue but with newer Tomcat and Native library. JVM randomly crashes with the following dump. Unfortunately it's hard for us to track this issue down in our environment. So, it's just a comment that the issue is reproducible with the newer versions.

Tomcat 7.0.59
Websockets and HTTP streaming over Atmopshere Framework
Tomcat Native library 1.1.32
APR version 1.3.9
Java version 1.7.0_21
OS version Red Hat Enterprise Linux 6.6 （64bit）

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f2376b61b8b, pid=14828, tid=139778958763776
#
# JRE version: 7.0_21-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libtcnative-1.so+0x13b8b]  Java_org_apache_tomcat_jni_Socket_send+0x7b
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00007f1fd8020000):  JavaThread "Atmosphere-Shared-AsyncOp-13" daemon [_thread_in_native, id=31127, stack(0x00007f20d31f2000,0x00007f20d32f3000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x0000000000000040

Registers:
RAX=0x0000000000000000, RBX=0x00007f20a82a0c28, RCX=0x0000000000000081, RDX=0x00007f20d32ef188
RSP=0x00007f20d32ef160, RBP=0x00007f1fd80201d8, RSI=0x00007f20d32ef190, RDI=0x0000000000000000
R8 =0x00007f20d32ef190, R9 =0x0000000000000001, R10=0x00007f239d9c7cfb, R11=0x0000000000000001
R12=0x00007f20d32ef190, R13=0x000000000000004c, R14=0x00007f20d32f11f0, R15=0x00007f1fd8020000
RIP=0x00007f2376b61b8b, EFLAGS=0x0000000000010246, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f20d32ef160)
0x00007f20d32ef160:   2c4e4f4954415249 69646e6f00000000
0x00007f20d32ef170:   4952503d00000000 00007f2000000000
0x00007f20d32ef180:   6f6d612c44455341 0000000000000001
0x00007f20d32ef190:   707974227b7c3681 56524553223a2265
0x00007f20d32ef1a0:   224f464e00000000 3a2279646f62222c
0x00007f20d32ef1b0:   726576726573227b 34313a22656d6954
0x00007f20d32ef1c0:   3936393736343133 2274222c7d333235
0x00007f20d32ef1d0:   373634313334313a 7b7d363630313739
0x00007f20d32ef1e0:   3a22644974656222 3438303733313532
0x00007f20d32ef1f0:   74756f797562222c 363a2265756c6156
0x00007f20d32ef200:   343730333339392e 313a2274222c5d7d
0x00007f20d32ef210:   3739373634313334 2e393a7d34363830
0x00007f20d32ef220:   746562222c323634 323135323a226449
0x00007f20d32ef230:   222c5d7d38373936 34313334313a2274
0x00007f20d32ef240:   3436383037393736 3a226449747d7d7d
0x00007f20d32ef250:   3238363734313532 313a2274222c5d7d
0x00007f20d32ef260:   3639373634313334 3a22647d39393637
0x00007f20d32ef270:   3630343535313532 313a2274222c5d7d
0x00007f20d32ef280:   3639373634313334 7d33327d35363337
0x00007f20d32ef290:   6552746e656d6574 7672655300000000
0x00007f20d32ef2a0:   737365733f74656c 317b3d64496e6f69
0x00007f20d32ef2b0:   6e756f636361267d 267d327b3d644974
0x00007f20d32ef2c0:   7b3d646f69726570 6c61636f6c267d33
0x00007f20d32ef2d0:   222c227d347b3d65 22746e756f636361
0x00007f20d32ef2e0:   756f636361227b3a 3a2265646f43746e
0x00007f20d32ef2f0:   305f615f73617122 62222c2230333932
0x00007f20d32ef300:   6572727543657361 4247223a2279636e
0x00007f20d32ef310:   6f636361222c2250 2265707954746e75
0x00007f20d32ef320:   544e45494c43223a 756f636361222c22
0x00007f20d32ef330:   697461657243746e 3a22656d69546e6f
0x00007f20d32ef340:   3537393334393331 62222c3432313435
0x00007f20d32ef350:   6572727543657361 696365725079636e 

Instructions: (pc=0x00007f2376b61b8b)
0x00007f2376b61b6b:   30 48 89 ef 44 89 c9 4d 89 e0 ff 90 40 06 00 00
0x00007f2376b61b7b:   48 8b 43 30 48 8d 54 24 28 48 8b 7b 18 4c 89 e6
0x00007f2376b61b8b:   ff 50 40 85 c0 0f 84 8a 00 00 00 83 f8 0b 0f 94
0x00007f2376b61b9b:   c2 3d c2 d4 01 00 75 65 48 8b 4c 24 28 48 85 c9 

Register to memory mapping:

RAX=0x0000000000000000 is an unknown value
RBX=0x00007f20a82a0c28 is an unknown value
RCX=0x0000000000000081 is an unknown value
RDX=0x00007f20d32ef188 is pointing into the stack for thread: 0x00007f1fd8020000
RSP=0x00007f20d32ef160 is pointing into the stack for thread: 0x00007f1fd8020000
RBP=0x00007f1fd80201d8 is an unknown value
RSI=0x00007f20d32ef190 is pointing into the stack for thread: 0x00007f1fd8020000
RDI=0x0000000000000000 is an unknown value
R8 =0x00007f20d32ef190 is pointing into the stack for thread: 0x00007f1fd8020000
R9 =0x0000000000000001 is an unknown value
R10=0x00007f239d9c7b10 [CodeBlob (0x00007f239d9c7b10)]
Framesize: 10
R11=0x0000000000000001 is an unknown value
R12=0x00007f20d32ef190 is pointing into the stack for thread: 0x00007f1fd8020000
R13=0x000000000000004c is an unknown value
R14=0x00007f20d32f11f0 is pointing into the stack for thread: 0x00007f1fd8020000
R15=0x00007f1fd8020000 is a thread


Stack: [0x00007f20d31f2000,0x00007f20d32f3000],  sp=0x00007f20d32ef160,  free space=1012k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtcnative-1.so+0x13b8b]  Java_org_apache_tomcat_jni_Socket_send+0x7b

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J  org.apache.tomcat.jni.Socket.$$YJP$$send(J[BII)I
J  org.apache.catalina.websocket.WsOutbound.doWriteBytes(Ljava/nio/ByteBuffer;Z)V
J  org.atmosphere.container.version.TomcatWebSocket.write(Ljava/lang/String;)Lorg/atmosphere/websocket/WebSocket;
J  org.atmosphere.websocket.WebSocket.write(Lorg/atmosphere/cpr/AtmosphereResponse;[BII)Lorg/atmosphere/websocket/WebSocket;
J  org.atmosphere.websocket.WebSocket.write(Lorg/atmosphere/cpr/AtmosphereResponse;[B)Lorg/atmosphere/websocket/WebSocket;
J  org.atmosphere.cpr.AtmosphereResponse$2.write([B)V
J  ****.MessageSerializerInterceptor$1.write(Ljava/io/OutputStream;Ljava/lang/Object;)V
J  org.atmosphere.handler.AbstractReflectorAtmosphereHandler.onStateChange(Lorg/atmosphere/cpr/AtmosphereResourceEvent;)V
J  org.atmosphere.config.managed.ManagedAtmosphereHandler.onStateChange(Lorg/atmosphere/cpr/AtmosphereResourceEvent;)V
J  org.atmosphere.cpr.DefaultBroadcaster$WriteOperation.call()Ljava/lang/Object;
J  org.atmosphere.cpr.DefaultBroadcaster.prepareInvokeOnStateChange(Lorg/atmosphere/cpr/AtmosphereResource;Lorg/atmosphere/cpr/AtmosphereResourceEvent;)V
J  org.atmosphere.cpr.DefaultBroadcaster.executeAsyncWrite(Lorg/atmosphere/cpr/DefaultBroadcaster$AsyncWriteToken;)V
J  org.atmosphere.cpr.DefaultBroadcaster$3.run()V
J  java.util.concurrent.FutureTask.run()V
J  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+14
j  java.lang.Thread.run()V+20
v  ~StubRoutines::call_stub

I'm marking this as a duplicate as that looks to be the most likely cause based on the information in this thread.

Regarding comment #9, that may be the same issue or it could be unrelated. If you still the the issue with the latest Tomcat Native 1.2.x, latest stable release of a supported Tomcat version and the latest Atmosphere release then please open a new issue and provide the simplest possible test case that reproduces the issue. Please also make sure you are using the JSR 356 WebSocket API, not the deprecated Tomcat WebSocket API.

*** This bug has been marked as a duplicate of bug 51813 ***