Log file created at: 2015/08/08 03:21:32 Running on machine: vd0212.halxg.cloudera.com Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0808 03:21:32.191912 18727 logging.cc:120] stderr will be logged to this file. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/fe/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/testdata/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] ================================================================= ==18727==ERROR: AddressSanitizer: heap-use-after-free on address 0x631009346820 at pc 0xfd43c9 bp 0x7fa15a3de580 sp 0x7fa15a3de578 READ of size 1 at 0x631009346820 thread T29996 #0 0xfd43c8 in impala::Tuple::IsNull(impala::NullIndicatorOffset const&) const /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/tuple.h:115 #1 0xfd247d in impala::SlotRef::GetBigIntVal(impala::ExprContext*, impala::TupleRow*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exprs/slot-ref.cc:402 #2 0xf78fa8 in impala::ExprContext::GetValue(impala::Expr*, impala::TupleRow*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exprs/expr-context.cc:232 #3 0x17ba054 in void impala::Tuple::MaterializeExprs(impala::TupleRow*, impala::TupleDescriptor const&, std::vector > const&, impala::MemPool*, std::vector >*, int*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/tuple.cc:96 #4 0x19b65ed in impala::Status impala::Sorter::Run::AddBatch(impala::RowBatch*, int, int*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/sorter.cc:414 #5 0x19ae93a in impala::Sorter::AddBatch(impala::RowBatch*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/sorter.cc:966 #6 0x1932543 in impala::SortNode::SortInput(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/sort-node.cc:156 #7 0x19320d3 in impala::SortNode::Open(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/sort-node.cc:76 #8 0x17a41b3 in impala::PlanFragmentExecutor::OpenInternal() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/plan-fragment-executor.cc:334 #9 0x17a361c in impala::PlanFragmentExecutor::Open() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/plan-fragment-executor.cc:320 #10 0x1497b53 in impala::FragmentMgr::FragmentExecState::Exec() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-exec-state.cc:50 #11 0x148b9fc in impala::FragmentMgr::FragmentExecThread(impala::FragmentMgr::FragmentExecState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-mgr.cc:70 #12 0x148e2d9 in boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 #13 0x13010d5 in boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 #14 0x15df408 in impala::Thread::SuperviseThread(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.cc:314 #15 0x15e3fcd in void boost::_bi::list4, std::allocator > >, boost::_bi::value, std::allocator > >, boost::_bi::value >, boost::_bi::value*> >::operator(), std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list0>(boost::_bi::type, void (*&)(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:457 #16 0x15e3ec9 in boost::_bi::bind_t, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list4, std::allocator > >, boost::_bi::value, std::allocator > >, boost::_bi::value >, boost::_bi::value*> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 #17 0x19e6433 in thread_proxy ??:0 #18 0xee3f13 in __asan::AsanThread::ThreadStart(unsigned long) ??:0 #19 0x30bba079d0 in ?? ??:0 #20 0x30bb2e89dc in ?? ??:0 0x631009346820 is located 8224 bytes inside of 65536-byte region [0x631009344800,0x631009354800) freed by thread T29996 here: #0 0xeddf84 in operator delete[](void*) ??:0 #1 0x133bc02 in impala::BufferedBlockMgr::DeleteBlock(impala::BufferedBlockMgr::Block*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-block-mgr.cc:822 #2 0x133b56e in impala::BufferedBlockMgr::Block::Delete() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-block-mgr.cc:131 #3 0x17bee56 in impala::BufferedTupleStream::Close() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:148 #4 0x18f8344 in impala::PartitionedAggregationNode::Partition::Close(bool) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:591 #5 0x18f781c in impala::PartitionedAggregationNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:316 #6 0x18ee020 in impala::NestedLoopJoinNode::NextProbeRow(impala::RuntimeState*, impala::RowBatch*, bool) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/nested-loop-join-node.cc:542 #7 0x18eb00e in impala::NestedLoopJoinNode::GetNextInnerJoin(impala::RuntimeState*, impala::RowBatch*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/nested-loop-join-node.cc:194 #8 0x18ea7b2 in impala::NestedLoopJoinNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/nested-loop-join-node.cc:145 #9 0x19324ff in impala::SortNode::SortInput(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/sort-node.cc:155 #10 0x19320d3 in impala::SortNode::Open(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/sort-node.cc:76 #11 0x17a41b3 in impala::PlanFragmentExecutor::OpenInternal() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/plan-fragment-executor.cc:334 #12 0x17a361c in impala::PlanFragmentExecutor::Open() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/plan-fragment-executor.cc:320 #13 0x1497b53 in impala::FragmentMgr::FragmentExecState::Exec() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-exec-state.cc:50 #14 0x148b9fc in impala::FragmentMgr::FragmentExecThread(impala::FragmentMgr::FragmentExecState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-mgr.cc:70 #15 0x148e2d9 in boost::_bi::bind_t, boost::_bi::list2, boost::_bi::value > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 #16 0x13010d5 in boost::function0::operator()() const /usr/include/boost/function/function_template.hpp:1012 #17 0x15df408 in impala::Thread::SuperviseThread(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.cc:314 #18 0x15e3fcd in void boost::_bi::list4, std::allocator > >, boost::_bi::value, std::allocator > >, boost::_bi::value >, boost::_bi::value*> >::operator(), std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list0>(boost::_bi::type, void (*&)(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:457 #19 0x15e3ec9 in boost::_bi::bind_t, std::allocator > const&, std::basic_string, std::allocator > const&, boost::function, impala::Promise*), boost::_bi::list4, std::allocator > >, boost::_bi::value, std::allocator > >, boost::_bi::value >, boost::_bi::value*> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20 #20 0x19e6433 in thread_proxy ??:0 previously allocated by thread T2847 here: #0 0xedddc4 in operator new[](unsigned long) ??:0 #1 0x1341ced in impala::BufferedBlockMgr::GetNewBlock(impala::BufferedBlockMgr::Client*, impala::BufferedBlockMgr::Block*, impala::BufferedBlockMgr::Block**, long) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-block-mgr.cc:407 #2 0x17bd175 in impala::BufferedTupleStream::NewBlockForWrite(int, bool*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:211 #3 0x17bc909 in impala::BufferedTupleStream::Init(impala::RuntimeProfile*, bool) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/buffered-tuple-stream.cc:128 #4 0x18fa15a in impala::PartitionedAggregationNode::Partition::InitStreams() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:450 #5 0x18f4dd5 in impala::PartitionedAggregationNode::CreateHashPartitions(int) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:753 #6 0x18f358d in impala::PartitionedAggregationNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:195 #7 0x17ef488 in impala::ExecNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/exec-node.cc:153 #8 0x1960b1a in impala::BlockingJoinNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/blocking-join-node.cc:60 #9 0x18e9780 in impala::NestedLoopJoinNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/nested-loop-join-node.cc:69 #10 0x17ef488 in impala::ExecNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/exec-node.cc:153 #11 0x1931bbd in impala::SortNode::Prepare(impala::RuntimeState*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/exec/sort-node.cc:45 #12 0x17a1970 in impala::PlanFragmentExecutor::Prepare(impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/runtime/plan-fragment-executor.cc:236 #13 0x1497a50 in impala::FragmentMgr::FragmentExecState::Prepare(impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-exec-state.cc:44 #14 0x148b416 in impala::FragmentMgr::ExecPlanFragment(impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-mgr.cc:51 #15 0x1407b4d in impala::ImpalaInternalService::ExecPlanFragment(impala::TExecPlanFragmentResult&, impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/impala-internal-service.h:37 #16 0x1674a83 in impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:949 #17 0x1674657 in impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::basic_string, std::allocator > const&, int, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:922 #18 0x1405f1f in apache::thrift::TDispatchProcessor::process(boost::shared_ptr, boost::shared_ptr, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/thirdparty/thrift-0.9.0/build/include/thrift/TDispatchProcessor.h:121 #19 0x22ba4f8 in apache::thrift::server::TThreadedServer::Task::run() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/thirdparty/thrift-0.9.0/lib/cpp/src/thrift/server/TThreadedServer.cpp:70 Thread T29996 created by T2847 here: #0 0xed98e0 in pthread_create ??:0 #1 0x19e560d in boost::thread::start_thread() ??:0 #2 0x15dece8 in impala::Thread::StartThread(boost::function const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.cc:276 #3 0x148d92f in impala::Thread::Thread(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, void (impala::FragmentMgr::* const&)(impala::FragmentMgr::FragmentExecState*), impala::FragmentMgr* const&, impala::FragmentMgr::FragmentExecState* const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.h:76 #4 0x148b61d in impala::FragmentMgr::ExecPlanFragment(impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/fragment-mgr.cc:63 #5 0x1407b4d in impala::ImpalaInternalService::ExecPlanFragment(impala::TExecPlanFragmentResult&, impala::TExecPlanFragmentParams const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/impala-internal-service.h:37 #6 0x1674a83 in impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:949 #7 0x1674657 in impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::basic_string, std::allocator > const&, int, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:922 #8 0x1405f1f in apache::thrift::TDispatchProcessor::process(boost::shared_ptr, boost::shared_ptr, void*) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/thirdparty/thrift-0.9.0/build/include/thrift/TDispatchProcessor.h:121 #9 0x22ba4f8 in apache::thrift::server::TThreadedServer::Task::run() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/thirdparty/thrift-0.9.0/lib/cpp/src/thrift/server/TThreadedServer.cpp:70 Thread T2847 created by T60 here: #0 0xed98e0 in pthread_create ??:0 #1 0x19e560d in boost::thread::start_thread() ??:0 #2 0x15dece8 in impala::Thread::StartThread(boost::function const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.cc:276 #3 0x12bc76b in impala::Thread::Thread, impala::Promise*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value*> > > >(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, boost::_bi::bind_t, impala::Promise*>, boost::_bi::list3, boost::_bi::value >, boost::_bi::value*> > > const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.h:63 #4 0x12bb315 in impala::ThriftThread::start() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/rpc/thrift-thread.cc:30 #5 0x22b8580 in apache::thrift::server::TThreadedServer::serve() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/thirdparty/thrift-0.9.0/lib/cpp/src/thrift/server/TThreadedServer.cpp:191 #6 0x603004b250bf in Thread T60 created by T0 here: #0 0xed98e0 in pthread_create ??:0 #1 0x19e560d in boost::thread::start_thread() ??:0 #2 0x15dece8 in impala::Thread::StartThread(boost::function const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.cc:276 #3 0x12b8e18 in impala::Thread::Thread(std::basic_string, std::allocator > const&, std::basic_string, std::allocator > const&, void (impala::ThriftServer::ThriftServerEventProcessor::* const&)(), impala::ThriftServer::ThriftServerEventProcessor* const&) /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/util/thread.h:69 #4 0x12acecb in impala::ThriftServer::ThriftServerEventProcessor::StartAndWaitForServer() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/rpc/thrift-server.cc:123 #5 0x12af994 in impala::ThriftServer::Start() /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/rpc/thrift-server.cc:388 #6 0xeebede in main /data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/src/service/impalad-main.cc:76 #7 0x30bb21ed5c in ?? ??:0 Shadow bytes around the buggy address: 0x0c6281260cb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260cc0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260cd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260ce0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260cf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd =>0x0c6281260d00: fd fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260d10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260d20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260d30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260d40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x0c6281260d50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 ASan internal: fe ==18727==ABORTING