Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2517

buffered-block-mgr.cc:891] Check failed: Validate()

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: Impala 2.3.0
    • Component/s: None
    • Labels:
    • Environment:
      impalad version 2.3.0-cdh5-INTERNAL DEBUG (build c21a2032f6b9b6a0861993372da9d749a3e905d1)
      Built on Thu, 08 Oct 2015 01:58:44 PST

      Description

      After ~4 hrs and ~10k queries a stress run crashed saying

      impala-stress-5.vpc.cloudera.com crashed:
      F1008 18:38:58.361202 10472 buffered-block-mgr.cc:891] Check failed: Validate() 
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 5
        Num unpinned blocks: 35
        Num available buffers: -6
        Total pinned buffers: 454
        Unfullfilled reserved buffers: 334
        Remaining memory: 2422471680 (#blocks=288)
        Block write threshold: 4
      [...skipped...]
      #6  0x000000000205f4ed in google::LogMessageFatal::~LogMessageFatal (this=0x7f8fdcb03280, __in_chrg=<value optimized out>) at src/logging.cc:1836
      #7  0x0000000001231ba7 in impala::BufferedBlockMgr::DeleteBlock (this=0x7cb7c00, block=0x568c06c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:891
      #8  0x000000000122993e in impala::BufferedBlockMgr::Block::Delete (this=0x568c06c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-block-mgr.cc:133
      #9  0x0000000001574135 in impala::BufferedTupleStream::Close (this=0xc773b560) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/buffered-tuple-stream.cc:159
      #10 0x000000000168d012 in impala::PartitionedHashJoinNode::BuildHashTables (this=0x6cc0fb00, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:1237
      #11 0x000000000168757d in impala::PartitionedHashJoinNode::ProcessBuildInput (this=0x6cc0fb00, state=0x1b3346d00, level=1) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:666
      #12 0x000000000168891d in impala::PartitionedHashJoinNode::PrepareNextPartition (this=0x6cc0fb00, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:781
      #13 0x000000000168a253 in impala::PartitionedHashJoinNode::GetNext (this=0x6cc0fb00, state=0x1b3346d00, out_batch=0x1d8dd5860, eos=0x6cc11be1) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:941
      #14 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x6cc11a80, state=0x1b3346d00, out_batch=0xec5932c0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692
      #15 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x6cc11a80, state=0x1b3346d00, out_batch=0xec5932c0, eos=0x3d0165361) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916
      #16 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x3d0165200, state=0x1b3346d00, out_batch=0x6c4574a0) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692
      #17 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x3d0165200, state=0x1b3346d00, out_batch=0x6c4574a0, eos=0x3d0167761) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916
      #18 0x000000000168787f in impala::PartitionedHashJoinNode::NextProbeRowBatch (this=0x3d0167600, state=0x1b3346d00, out_batch=0x7f8fdcb04b20) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:692
      #19 0x0000000001689f47 in impala::PartitionedHashJoinNode::GetNext (this=0x3d0167600, state=0x1b3346d00, out_batch=0x7f8fdcb04b20, eos=0x7f8fdcb04d3f) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-hash-join-node.cc:916
      #20 0x000000000166eba7 in impala::PartitionedAggregationNode::Open (this=0x28844c600, state=0x1b3346d00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/exec/partitioned-aggregation-node.cc:240
      #21 0x000000000155fc4b in impala::PlanFragmentExecutor::OpenInternal (this=0x1e2d7dd28) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/plan-fragment-executor.cc:334
      #22 0x000000000155faec in impala::PlanFragmentExecutor::Open (this=0x1e2d7dd28) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/runtime/plan-fragment-executor.cc:320
      #23 0x0000000001333060 in impala::FragmentMgr::FragmentExecState::Exec (this=0x1e2d7db00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-exec-state.cc:50
      #24 0x000000000132b67e in impala::FragmentMgr::FragmentExecThread (this=0x5b92ea0, exec_state=0x1e2d7db00) at /usr/src/debug/impala-2.3.0-cdh5.5.0-SNAPSHOT/be/src/service/fragment-mgr.cc:70
      

      I'll post the location of the logs/core dump when it is available. The cluster will be up until tomorrow if anyone wants to look now. The crashed node was impala-stress-5.vpc.cloudera.com.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              caseyc casey
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: