[IMPALA-1863] q21 hangs when run with mem_limit=780MB

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.2
• Fix Version/s: Impala 2.2
• Environment: commit 45897c41cd43b8061e019480acfc35306d26d4c5
• Assignee: Daniel Hecht
• Reporter: Daniel Hecht
• Created: 09/Mar/15 23:19
• Updated: 25/Mar/15 01:41
• Resolved: 25/Mar/15 01:41

Description

When running in an EC2 instance against S3, I'm seeing q21 hang when run by test_mem_usage_scaling.py. The minimum steps to repro appears to be:

./run-tests.py query_test/test_mem_usage_scaling.py

I haven't been able to repro on my physical host, but I think that's just an environmental difference. A couple of possible differences with the EC2 host is physical memsize is slightly less (MemTotal: 30822708 kB), and when running against S3, number of disk-io threads is 16.

The hung query in the log files is 5c4ea905ef3e8e81:c65a0b777b980983. node1 and node2 each have two instances similar to:

I0309 15:54:39.176250 30049 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
Buffered block mgr
  Num writes outstanding: 0
  Num free io buffers: 0
  Num unpinned blocks: 0
  Num available buffers: -189
  Total pinned buffers: 350
  Unfullfilled reserved buffers: 189
  Remaining memory: 8126448 (#blocks=0)
  Block write threshold: 2
Client 0x8ec5c50
  num_reserved_buffers=34
  num_tmp_reserved_buffers=0
  num_pinned_buffers=13
    @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
    @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
    @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
    @          0x148d3bf  impala::BufferedTupleStream::SwitchToIoBuffers()
    @          0x1581253  impala::PartitionedHashJoinNode::ReserveTupleStreamBlocks()
    @          0x1579724  impala::PartitionedHashJoinNode::AppendRowStreamFull()
    @     0x7f464b5d3ac1  (unknown)

I0309 15:54:44.609871 30202 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
Buffered block mgr
  Num writes outstanding: 0
  Num free io buffers: 0
  Num unpinned blocks: 0
  Num available buffers: -204
  Total pinned buffers: 398
  Unfullfilled reserved buffers: 204
  Remaining memory: 7864304 (#blocks=0)
  Block write threshold: 2
Client 0xd62dc50
  num_reserved_buffers=34
  num_tmp_reserved_buffers=0
  num_pinned_buffers=32
    @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
    @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
    @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
    @     0x7f464b5d1028  (unknown)

Ippo, I'm assigning to you since you're looking at other similar problems, but let me know if it looks unrelated.