[IMPALA-5093] Rare failure to decode LZ4 batch

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.10.0
• Fix Version/s: Impala 2.10.0
• Component/s: Distributed Exec
• Epic Color: ghx-label-4
• Assignee: Sailesh Mukil
• Reporter: Henry Robinson
• Created: 17/Mar/17 19:07
• Updated: 14/Jul/17 21:46
• Resolved: 14/Jul/17 21:46

Description

KRPC sometimes hits this DCHECK

https://github.com/henryr/Impala/blob/krpc/be/src/runtime/row-batch.cc#L108

which indicates that Lz4Compress::ProcessBlock has failed to decompress the incoming row batch. Not much clarity about how this happens yet.

Stack trace:

6  0x0000000002c7598e in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x00000000017914ba in impala::RowBatch::RowBatch (this=0x3d8af3c0, row_desc=..., input_batch=..., mem_tracker=0x13ad1c80) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/row-batch.cc:108
#8  0x000000000174c655 in impala::DataStreamRecvr::SenderQueue::AddBatch (this=0xc962800, payload=...) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/data-stream-recvr.cc:210
#9  0x000000000174e13a in impala::DataStreamRecvr::AddBatch (this=0xcdda580, payload=...) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/data-stream-recvr.cc:352
#10 0x000000000173f076 in impala::DataStreamMgr::AddData (this=0xe4a0b20, fragment_instance_id=..., payload=...) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/data-stream-mgr.cc:190
#11 0x00000000018e8c63 in impala::DataStreamService::TransmitData (this=0xdb357c0, request=0x4338c3f0, response=0xd802c00, context=0x11d27b60) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/service/impala-internal-service.cc:77
#12 0x00000000018ed74e in _ZZN6impala19DataStreamServiceIfC4ERK13scoped_refptrIN4kudu12MetricEntityEERKS1_INS2_3rpc13ResultTrackerEEENKUlPKN6google8protobuf7MessageEPSE_PNS7_10RpcContextEE0_clESG_SH_SJ_ ()
    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/service/impala_internal_service.service.cc:157
#13 0x00000000018eff3b in std::_Function_handler<void(const google::protobuf::Message*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::DataStreamServiceIf::DataStreamServiceIf(const scoped_refptr<kudu::MetricEntity>&, const scoped_refptr<kudu::rpc::ResultTracker>&)::<lambda(const google::protobuf::Message*, google::protobuf::Message*, kudu::rpc::RpcContext*)> >::_M_invoke(const std::_Any_data &, const google::protobuf::Message *, google::protobuf::Message *, kudu::rpc::RpcContext *) (__functor=..., __args#0=0x4338c3f0,
    __args#1=0xd802c00, __args#2=0x11d27b60) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/gcc-4.9.2/include/c++/4.9.2/functional:2039
#14 0x0000000001d9fcb4 in std::function<void(const google::protobuf::Message*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(const google::protobuf::Message *, google::protobuf::Message *, kudu::rpc::RpcContext *) const (this=0xeb320b8,
    __args#0=0x4338c3f0, __args#1=0xd802c00, __args#2=0x11d27b60) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/gcc-4.9.2/include/c++/4.9.2/functional:2439
#15 0x0000000001d9f6b7 in kudu::rpc::GeneratedServiceIf::Handle (this=0xdb357c0, call=0xcf37480) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/service_if.cc:134
#16 0x00000000016abfb8 in impala::ImpalaServicePool::RunThread (this=0xe85ac80) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/rpc/impala-service-pool.cc:130
#17 0x00000000016ab5db in impala::ImpalaServicePool::<lambda()>::operator()(void) const (__closure=0x7f5e11a86be8) at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/rpc/impala-service-pool.cc:68