Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3544

crash in DiskIoMgr::GetFreeBuffer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: None
    • Component/s: Backend
    • Labels:
    • Environment:
      Ubuntu Trusty, CDH 5.7.0

      Description

      Impala becomes unresponsive, causing a full outage for our reporting system. Impalad logs show various errors as follows:

      Time	Log Level	Source	Log Message
      May 16, 3:20:59.953 PM	ERROR	logging.cc:120	
      stderr will be logged to this file.
      May 16, 3:27:38.652 PM	ERROR	data-stream-sender.cc:261	
      channel send status: 
      Sender timed out waiting for receiver fragment instance: a045d3abcd1bed4f:a116563837c8ebda
      May 16, 3:27:39.738 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:44.784 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:46.055 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:46.371 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:46.906 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:51.003 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:52.479 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:54.227 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:54.298 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:54.305 PM	ERROR	data-stream-sender.cc:261	
      channel send status: 
      Sender timed out waiting for receiver fragment instance: 7446d34d9eefb9f:436f61bcd3c78eb3
      May 16, 3:27:55.022 PM	ERROR	data-stream-sender.cc:261	
      channel send status: 
      Sender timed out waiting for receiver fragment instance: b44445bcb782b5a4:5cef13b26af20aa1
      May 16, 3:27:55.646 PM	ERROR	data-stream-sender.cc:261	
      channel send status: 
      Sender timed out waiting for receiver fragment instance: 3249c6494703ecc7:3b9186fbed00dc
      May 16, 3:27:56.042 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:56.277 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:56.749 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-worker-6-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:57.126 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:27:59.030 PM	ERROR	data-stream-sender.cc:261	
      channel send status: Couldn't open transport for ux-reporting-engine-master-1-prod-us-east-1a:22000 (connect() failed: Connection timed out)
      May 16, 3:28:03.573 PM	ERROR	data-stream-sender.cc:261	
      channel send status: 
      Sender timed out waiting for receiver fragment instance: c64b22033608d609:10e070d34aa299dd
      

        Attachments

        1. Screen Shot 2016-05-27 at 12.43.23 PM.png
          227 kB
          Scott Wallace
        2. 2016-05-26_1438.png
          56 kB
          Scott Wallace
        3. 2016-05-26_1440.png
          68 kB
          Scott Wallace
        4. hs_err_pid7016.log
          386 kB
          Scott Wallace
        5. impalad.ux-reporting-engine-worker-18-prod-us-east-1a.impala.log.ERROR.20160519-140954.7016
          2 kB
          Scott Wallace
        6. 2016-05-18_1140.png
          234 kB
          Scott Wallace
        7. 2016-05-18_1109.png
          235 kB
          Scott Wallace
        8. impalad.ux-reporting-engine-worker-13-prod-us-east-1a.impala.log.ERROR.20160516-152100.89097
          31 kB
          Scott Wallace
        9. 2016-05-17_1441.png
          97 kB
          Scott Wallace
        10. impalad.ux-reporting-engine-worker-6-prod-us-east-1a.impala.log.INFO.20160516-152100.44312
          16 kB
          Scott Wallace

          Issue Links

            Activity

              People

              • Assignee:
                sailesh Sailesh Mukil
                Reporter:
                swallace Scott Wallace
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: