[DISPATCH-895] qpid-dispatch crashes with a SEGFAULT in libqpid-proton

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0
• Fix Version/s: 1.0.1
• Component/s: Routing Engine

Environment

OS: Red Hat Enterprise Linux 7.4
Arch: x86_64
Installed Packages
Name : qpid-cpp-client
Arch : x86_64
Version : 1.36.0
Release : 9.el7
Size : 4.4 M
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : Libraries for Qpid C++ client applications
URL : http://qpid.apache.org
License : ASL 2.0
Description : Run-time libraries for AMQP client applications developed using Qpid
: C++. Clients exchange messages with an AMQP message broker using
: the AMQP protocol.

Name : qpid-cpp-server
Arch : x86_64
Version : 1.36.0
Release : 9.el7
Size : 5.2 M
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : An AMQP message broker daemon
URL : http://qpid.apache.org
License : ASL 2.0
Description : A message broker daemon that receives stores and routes messages using
: the open AMQP messaging protocol.

Name : qpid-cpp-server-linearstore
Arch : x86_64
Version : 1.36.0
Release : 9.el7
Size : 797 k
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : Red Hat persistence extension to the Qpid messaging sytem
URL : http://qpid.apache.org
License : ASL 2.0
Description : Red Hat persistence extension to the Qpid AMQP broker: persistent message
: storage using a libaio-based asynchronous journal.

Name : qpid-dispatch-debuginfo
Arch : x86_64
Version : 0.8.0
Release : 10.el7
Size : 2.1 M
Repo : installed
Summary : Debug information for package qpid-dispatch
URL : http://qpid.apache.org/
License : ASL 2.0
Description : This package provides debug information for package qpid-dispatch.
: Debug information is useful when developing applications that use this
: package or when debugging this package.

Name : qpid-dispatch-router
Arch : x86_64
Version : 0.8.0
Release : 10.el7
Size : 1.0 M
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : The Qpid Dispatch Router executable
URL : http://qpid.apache.org/
License : ASL 2.0
Description : The Qpid Dispatch Router executable.

Name : qpid-proton-c
Arch : x86_64
Version : 0.16.0
Release : 7.el7
Size : 334 k
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : C libraries for Qpid Proton
URL : http://qpid.apache.org/proton/
License : ASL 2.0
Description : C libraries for Qpid Proton.

Name : qpid-proton-debuginfo
Arch : x86_64
Version : 0.16.0
Release : 7.el7
Size : 6.8 M
Repo : installed
Summary : Debug information for package qpid-proton
URL : http://qpid.apache.org/proton/
License : ASL 2.0
Description : This package provides debug information for package qpid-proton.
: Debug information is useful when developing applications that use this
: package or when debugging this package.

Name : qpid-qmf
Arch : x86_64
Version : 1.36.0
Release : 9.el7
Size : 476 k
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : The QPID Management Framework
URL : http://qpid.apache.org
License : ASL 2.0
Description : The Qpid Management Framework is a general-purpose management bus built on Qpid
: messaging. It takes advantage of the scalability, security, and rich
: capabilities of Qpid to provide flexible and easy-to-use manageability to a
: large set of applications.

Name : qpid-tools
Arch : noarch
Version : 1.36.0
Release : 9.el7
Size : 384 k
Repo : installed
From repo : rhel-server-7-satellite-6-beta-rpms
Summary : Management and diagostic tools for Apache Qpid
URL : http://qpid.apache.org
License : ASL 2.0
Description : Management and diagnostic tools for Apache Qpid brokers and clients.

• Flags: Important
• External issue URL: https://bugzilla.redhat.com/show_bug.cgi?id=1519140
• External issue ID: 1519140
• Assignee: Ganesh Murthy
• Reporter: Saurabh Badhwar
• Created: 07/Dec/17 03:56
• Updated: 02/Oct/19 17:55
• Resolved: 07/Dec/17 14:57

Description

During the work process associated with Red Hat Satellite, we found that Qpid-dispatch router started to crash abruptly without any changes being made to it. On checking the logs, it seems like qpid-dispatch-router experienced a SEGFAULT and crashed.
Once this crash starts to happen there is no way we are able to restart qpid-dispatch since it continues to fail.

A trace from the /var/log/messages

[root@c10-h19-r730xd-vm1 qdrouterd]# cat /var/log/messages | grep qdrouterd
Nov 28 00:59:50 c10-h19-r730xd-vm1 kernel: qdrouterd[28352]: segfault at 180 ip 00007fdd86d37291 sp 00007fdd7951c2b8 error 4 in libqpid-proton.so.10.0.0[7fdd86d19000+4b000]
Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process exited, code=killed, status=11/SEGV
Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered failed state.
Nov 28 00:59:50 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
Nov 30 03:44:04 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***
Nov 30 03:45:58 c10-h19-r730xd-vm1 kernel: qdrouterd[26607]: segfault at 180 ip 00007f9f4d230291 sp 00007f9f3fa152b8 error 4 in libqpid-proton.so.10.0.0[7f9f4d212000+4b000]
Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process exited, code=killed, status=11/SEGV
Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered failed state.
Nov 30 03:45:58 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
Nov 30 03:50:12 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***
Nov 30 04:06:07 c10-h19-r730xd-vm1 ansible-file: Invoked with directory_mode=None force=False remote_src=None path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=0755 attributes=None backup=None
Nov 30 04:06:10 c10-h19-r730xd-vm1 ansible-stat: Invoked with checksum_algorithm=sha1 get_checksum=True path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1 follow=False get_md5=False get_mime=True get_attributes=True
Nov 30 04:06:13 c10-h19-r730xd-vm1 ansible-copy: Invoked with directory_mode=None force=True remote_src=None owner=root follow=False local_follow=None group=root unsafe_writes=None setype=None content=NOT_LOGGING_PARAMETER serole=None dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None original_basename=limits-qdrouterd.conf regexp=None validate=None src=/root/.ansible/tmp/ansible-tmp-1512032767.73-18163995376563/source seuser=None delimiter=None mode=0644 attributes=None backup=False
Nov 30 04:06:15 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True executable=None _uses_shell=False _raw_params=restorecon -vR /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None stdin=None
Nov 30 04:06:42 c10-h19-r730xd-vm1 kernel: qdrouterd[29818]: segfault at 180 ip 00007f0863215291 sp 00007ffd3666b678 error 4 in libqpid-proton.so.10.0.0[7f08631f7000+4b000]
Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process exited, code=killed, status=11/SEGV
Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered failed state.
Nov 30 04:06:42 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
Nov 30 04:06:58 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***
Nov 30 04:08:02 c10-h19-r730xd-vm1 ansible-file: Invoked with directory_mode=None force=False remote_src=None path=/etc/systemd/system/qdrouterd.service.d owner=root follow=False group=root unsafe_writes=None state=directory content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None selevel=None original_basename=None regexp=None validate=None src=None seuser=None recurse=False delimiter=None mode=0755 attributes=None backup=None
Nov 30 04:08:05 c10-h19-r730xd-vm1 ansible-stat: Invoked with checksum_algorithm=sha1 get_checksum=True path=/etc/systemd/system/qdrouterd.service.d/limits.conf checksum_algo=sha1 follow=False get_md5=False get_mime=True get_attributes=True
Nov 30 04:08:07 c10-h19-r730xd-vm1 ansible-file: Invoked with directory_mode=None force=False remote_src=None path=/etc/systemd/system/qdrouterd.service.d/limits.conf owner=root follow=False group=root unsafe_writes=None state=file content=NOT_LOGGING_PARAMETER serole=None diff_peek=None setype=None dest=/etc/systemd/system/qdrouterd.service.d/limits.conf selevel=None original_basename=limits-qdrouterd.conf regexp=None validate=None src=limits-qdrouterd.conf seuser=None recurse=False delimiter=None mode=0644 attributes=None backup=None
Nov 30 04:08:09 c10-h19-r730xd-vm1 ansible-command: Invoked with warn=True executable=None _uses_shell=False _raw_params=restorecon -vR /etc/systemd/system/qdrouterd.service.d removes=None creates=None chdir=None stdin=None
Nov 30 04:08:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***
Nov 30 04:08:47 c10-h19-r730xd-vm1 kernel: qdrouterd[31775]: segfault at 180 ip 00007f56880e4291 sp 00007f56799852b8 error 4 in libqpid-proton.so.10.0.0[7f56880c6000+4b000]
Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process exited, code=killed, status=11/SEGV
Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered failed state.
Nov 30 04:08:47 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
Nov 30 04:09:06 c10-h19-r730xd-vm1 kernel: qdrouterd[32150]: segfault at 180 ip 00007fb455eea291 sp 00007fb446f8a2b8 error 4 in libqpid-proton.so.10.0.0[7fb455ecc000+4b000]
Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service: main process exited, code=killed, status=11/SEGV
Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: Unit qdrouterd.service entered failed state.
Nov 30 04:09:06 c10-h19-r730xd-vm1 systemd: qdrouterd.service failed.
Nov 30 04:09:21 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***
Nov 30 04:11:20 c10-h19-r730xd-vm1 /usr/sbin/katello-service: *** status failed: qdrouterd ***

From the messages, it seems like, the SEGFAULT happen in libqpid-proton.so