I also tried running guacd in docker images. Even seeing same result in docker container instance.
guacd: ERROR: Error reading data
guacd: INFO: Client disconnected
guacd: INFO: SSH connection ended.
This looks different, actually. There's no segfault. It looks like the actual underlying TCP connection to the SSH server somehow dies, and the Guacamole side of things logs the failure and cleans things up in an orderly fashion.
Also attaching gcore dump of both guacd and ssh process
Unfortunately, that core dump will only be readable in the context of the specific binary built on your system. It's unlikely that any other build will match things sufficiently.
Below is the gdb output from guacd ...
The next time this happens, and you get the prompt from gdb that execution has stopped, please use the "bt" command to produce the stacktrace. Lacking that, gdb only prints the line where the signal was received, which in this case is deep within a system call. The context relevant to debugging Guacamole is further up the chain.
... I got nothing from sshd.
There is no need to run gdb against sshd. Only the guard process which is created by guacd for the SSH connection needs to be debugged. For example, in one of your earlier comments, the segfault in the system logs was:
[Fri Aug 26 07:32:16 2016] guacd: segfault at 10 ip 00007f378f3faba1 sp 00007f37887d3bb8 error 4 in libpthread-2.17.so[7f378f3f1000+16000]
The PID of the guacd process that died here was 4388. If you look earlier in the logs, you should find something like:
guacd: INFO: Protocol "ssh" selected
guacd: INFO: Connection ID is "... randomly-generated identifier ..."
guacd: INFO: Starting client
guacd: INFO: SSH connection successful
That would have been the PID to attach gdb to for the sake of debugging. When trying to reproduce this, you should:
- Connect to SSH via Guacamole as normal.
- Once connected, look at the logs to find the messages above.
- Determine the PID from those messages, and attach gdb.
- Immediately run gdb's "continue" command to prevent the connection from timing out.
- Reproduce the issue.
- Once gdb pauses execution due to the segfault, run gdb's "bt" command to produce a stacktrace.
As there will be multiple threads within the same process handling that SSH connection, it is always possible that gdb will not be within the relevant thread when the segfault signal is received, and that the stacktrace will be bogus ... In that case, the gdb command to use would be "thread apply all bt", which will produce stacktraces for all threads.
... That should hopefully be enough information for us to get an idea of where things are going wrong.