[IMPALA-3544] crash in DiskIoMgr::GetFreeBuffer

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.5.0
• Component/s: Backend
• Labels: crash hang
• Environment: Ubuntu Trusty, CDH 5.7.0
• Target Version: Impala 2.7.0
• Assignee: Sailesh Mukil
• Reporter: Scott Wallace
• Created: 16/May/16 15:56
• Updated: 22/May/18 16:00
• Resolved: 19/Aug/16 23:50

Description

Impala becomes unresponsive, causing a full outage for our reporting system. Impalad logs show various errors as follows:

Time	Log Level	Source	Log Message
May 16, 3:20:59.953 PM	ERROR	logging.cc:120	
stderr will be logged to this file.
May 16, 3:27:38.652 PM	ERROR	data-stream-sender.cc:261	
channel send status: 
Sender timed out waiting for receiver fragment instance: a045d3abcd1bed4f:a116563837c8ebda
May 16, 3:27:39.738 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:44.784 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:46.055 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:46.371 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:46.906 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:51.003 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:52.479 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:54.227 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:54.298 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:54.305 PM	ERROR	data-stream-sender.cc:261	
channel send status: 
Sender timed out waiting for receiver fragment instance: 7446d34d9eefb9f:436f61bcd3c78eb3
May 16, 3:27:55.022 PM	ERROR	data-stream-sender.cc:261	
channel send status: 
Sender timed out waiting for receiver fragment instance: b44445bcb782b5a4:5cef13b26af20aa1
May 16, 3:27:55.646 PM	ERROR	data-stream-sender.cc:261	
channel send status: 
Sender timed out waiting for receiver fragment instance: 3249c6494703ecc7:3b9186fbed00dc
May 16, 3:27:56.042 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:56.277 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:56.749 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:57.126 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:27:59.030 PM	ERROR	data-stream-sender.cc:261	
channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
May 16, 3:28:03.573 PM	ERROR	data-stream-sender.cc:261	
channel send status: 
Sender timed out waiting for receiver fragment instance: c64b22033608d609:10e070d34aa299dd

Issue Links

• duplicates: IMPALA-3764 Fuzz-test HDFS scanners
• is duplicated by: IMPALA-7057 crash when DiskIoMgr::GetFreeBuffer(long*)+0xa9
• relates to: IMPALA-2877 Unexpected exits with impala::DiskIoMgr::GetFreeBuffer in dump