Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1863

q21 hangs when run with mem_limit=780MB

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.2
    • Fix Version/s: Impala 2.2
    • Component/s: None
    • Labels:
      None
    • Environment:
      commit 45897c41cd43b8061e019480acfc35306d26d4c5

      Description

      When running in an EC2 instance against S3, I'm seeing q21 hang when run by test_mem_usage_scaling.py. The minimum steps to repro appears to be:

      ./run-tests.py query_test/test_mem_usage_scaling.py
      

      I haven't been able to repro on my physical host, but I think that's just an environmental difference. A couple of possible differences with the EC2 host is physical memsize is slightly less (MemTotal: 30822708 kB), and when running against S3, number of disk-io threads is 16.

      The hung query in the log files is 5c4ea905ef3e8e81:c65a0b777b980983. node1 and node2 each have two instances similar to:

      I0309 15:54:39.176250 30049 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -189
        Total pinned buffers: 350
        Unfullfilled reserved buffers: 189
        Remaining memory: 8126448 (#blocks=0)
        Block write threshold: 2
      Client 0x8ec5c50
        num_reserved_buffers=34
        num_tmp_reserved_buffers=0
        num_pinned_buffers=13
          @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
          @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
          @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
          @          0x148d3bf  impala::BufferedTupleStream::SwitchToIoBuffers()
          @          0x1581253  impala::PartitionedHashJoinNode::ReserveTupleStreamBlocks()
          @          0x1579724  impala::PartitionedHashJoinNode::AppendRowStreamFull()
          @     0x7f464b5d3ac1  (unknown)
      
      I0309 15:54:44.609871 30202 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -204
        Total pinned buffers: 398
        Unfullfilled reserved buffers: 204
        Remaining memory: 7864304 (#blocks=0)
        Block write threshold: 2
      Client 0xd62dc50
        num_reserved_buffers=34
        num_tmp_reserved_buffers=0
        num_pinned_buffers=32
          @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
          @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
          @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
          @     0x7f464b5d1028  (unknown)
      

      Ippo, I'm assigning to you since you're looking at other similar problems, but let me know if it looks unrelated.

        Attachments

        1. impalad_node1.INFO.gz
          176 kB
          Dan Hecht
        2. impalad_node2.INFO.gz
          174 kB
          Dan Hecht
        3. impalad.INFO.gz
          2.58 MB
          Dan Hecht
        4. profile.txt
          119 kB
          Dan Hecht

          Activity

            People

            • Assignee:
              dhecht Dan Hecht
              Reporter:
              dhecht Dan Hecht
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: