Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1863

q21 hangs when run with mem_limit=780MB

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.2
    • Fix Version/s: Impala 2.2
    • Component/s: None
    • Labels:
      None
    • Environment:
      commit 45897c41cd43b8061e019480acfc35306d26d4c5

      Description

      When running in an EC2 instance against S3, I'm seeing q21 hang when run by test_mem_usage_scaling.py. The minimum steps to repro appears to be:

      ./run-tests.py query_test/test_mem_usage_scaling.py
      

      I haven't been able to repro on my physical host, but I think that's just an environmental difference. A couple of possible differences with the EC2 host is physical memsize is slightly less (MemTotal: 30822708 kB), and when running against S3, number of disk-io threads is 16.

      The hung query in the log files is 5c4ea905ef3e8e81:c65a0b777b980983. node1 and node2 each have two instances similar to:

      I0309 15:54:39.176250 30049 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -189
        Total pinned buffers: 350
        Unfullfilled reserved buffers: 189
        Remaining memory: 8126448 (#blocks=0)
        Block write threshold: 2
      Client 0x8ec5c50
        num_reserved_buffers=34
        num_tmp_reserved_buffers=0
        num_pinned_buffers=13
          @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
          @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
          @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
          @          0x148d3bf  impala::BufferedTupleStream::SwitchToIoBuffers()
          @          0x1581253  impala::PartitionedHashJoinNode::ReserveTupleStreamBlocks()
          @          0x1579724  impala::PartitionedHashJoinNode::AppendRowStreamFull()
          @     0x7f464b5d3ac1  (unknown)
      
      I0309 15:54:44.609871 30202 buffered-block-mgr.cc:859] Query id=5c4ea905ef3e8e81:c65a0b777b980983 was unable to get minimum required buffers.
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -204
        Total pinned buffers: 398
        Unfullfilled reserved buffers: 204
        Remaining memory: 7864304 (#blocks=0)
        Block write threshold: 2
      Client 0xd62dc50
        num_reserved_buffers=34
        num_tmp_reserved_buffers=0
        num_pinned_buffers=32
          @           0xf8c7d0  impala::BufferedBlockMgr::FindBufferForBlock()
          @           0xf86fd1  impala::BufferedBlockMgr::GetNewBlock()
          @          0x148dde5  impala::BufferedTupleStream::NewBlockForWrite()
          @     0x7f464b5d1028  (unknown)
      

      Ippo, I'm assigning to you since you're looking at other similar problems, but let me know if it looks unrelated.

        Attachments

        1. profile.txt
          119 kB
          Daniel Hecht
        2. impalad.INFO.gz
          2.58 MB
          Daniel Hecht
        3. impalad_node2.INFO.gz
          174 kB
          Daniel Hecht
        4. impalad_node1.INFO.gz
          176 kB
          Daniel Hecht

          Activity

            People

            • Assignee:
              dhecht Daniel Hecht
              Reporter:
              dhecht Daniel Hecht
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: