Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.9.0
-
ghx-label-1
Description
ASAN build failed with the following messages. Attila Jeges, would you mind taking a look ? Looks like it's a long standing issue and recent changes to sequence table writer may have added test coverages which expose it.
==29329==ERROR: AddressSanitizer: heap-use-after-free on address 0x608000c1b238 at pc 0x000000fbb9eb bp 0x7f5dc50324a0 sp 0x7f5dc5031c50 READ of size 65 at 0x608000c1b238 thread T22312 #0 0xfbb9ea in memcpy /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:438 #1 0x7f63678e351a in std::char_traits<char>::copy(char*, char const*, unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/char_traits.h:275 #2 0x7f63678e351a in std::basic_streambuf<char, std::char_traits<char> >::xsputn(char const*, long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/streambuf.tcc:90 #3 0x7f63678da6a2 in std::basic_streambuf<char, std::char_traits<char> >::sputn(char const*, long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/streambuf:451 #4 0x7f63678da6a2 in std::ostream::_M_write(char const*, long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/ostream:313 #5 0x7f63678da6a2 in std::ostream::write(char const*, long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/ostream.tcc:196 #6 0x19bfa25 in impala::WriteStream::WriteBytes(int, char const*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/write-stream.inline.h:73:3 #7 0x19c4ac0 in impala::HdfsSequenceTableWriter::ConsumeRow(impala::TupleRow*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:331:3 #8 0x19c2a1e in impala::HdfsSequenceTableWriter::AppendRows(impala::RowBatch*, std::vector<int, std::allocator<int> > const&, bool*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:115:35 #9 0x19b1d97 in impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*, impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition, std::default_delete<impala::OutputPartition> >, std::vector<int, std::allocator<int> > >*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/ Impala/be/src/exec/hdfs-table-sink.cc:281:31 #10 0x19b57a7 in impala::HdfsTableSink::Send(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-table-sink.cc:628:31 #11 0x1d1688a in impala::PlanFragmentExecutor::ExecInternal() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:359:31 #12 0x1d16155 in impala::PlanFragmentExecutor::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:337:14 #13 0x1d0dd3a in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:68:7 #14 0x1d1c1cf in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110:3 #15 0x1d1da57 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Im pala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #16 0x12f30a2 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14 #17 0x16c5ab5 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:325:3 #18 0x16cfb8a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457:9 #19 0x16cfa17 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<l ong>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #20 0x1da7249 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da7249) #21 0x3ddda07850 in start_thread (/lib64/libpthread.so.0+0x3ddda07850) #22 0x3ddd6e894c in clone (/lib64/libc.so.6+0x3ddd6e894c) 0x608000c1b238 is located 24 bytes inside of 90-byte region [0x608000c1b220,0x608000c1b27a) freed by thread T22312 here: #0 0x10392a0 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94 #1 0x19c48d8 in impala::HdfsSequenceTableWriter::ConsumeRow(impala::TupleRow*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:307:5 #2 0x19c2a1e in impala::HdfsSequenceTableWriter::AppendRows(impala::RowBatch*, std::vector<int, std::allocator<int> > const&, bool*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:115:35 #3 0x19b1d97 in impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*, impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition, std::default_delete<impala::OutputPartition> >, std::vector<int, std::allocator<int> > >*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/ Impala/be/src/exec/hdfs-table-sink.cc:281:31 #4 0x19b57a7 in impala::HdfsTableSink::Send(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-table-sink.cc:628:31 #5 0x1d1688a in impala::PlanFragmentExecutor::ExecInternal() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:359:31 #6 0x1d16155 in impala::PlanFragmentExecutor::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:337:14 #7 0x1d0dd3a in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:68:7 #8 0x1d1c1cf in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110:3 #9 0x1d1da57 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #10 0x12f30a2 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14 #11 0x16c5ab5 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:325:3 #12 0x16cfb8a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457:9 #13 0x16cfa17 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #14 0x1da7249 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da7249) previously allocated by thread T22312 here: #0 0x1038ca0 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62 #1 0x7f6367902c48 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104 #2 0x7f6367902c48 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:607 #3 0x7f63678e1225 in char* std::string::_S_construct_aux<char*>(char*, char*, std::allocator<char> const&, std::__false_type) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1743 #4 0x7f63678e1225 in char* std::string::_S_construct<char*>(char*, char*, std::allocator<char> const&) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1764 #5 0x7f63678e1225 in basic_string<char*> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:229 #6 0x7f63678e1225 in std::basic_stringbuf<char, std::char_traits<char>, std::allocator<char> >::str() const /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/sstream:133 #7 0x7f63678e1225 in std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >::str() const /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/gcc/build/x86_64-unknown-linux-gnu/libstdc++-v3/include/sstream:584 #8 0x19bfa4d in impala::WriteStream::String() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/write-stream.inline.h:94:10 #9 0x19c48c5 in impala::HdfsSequenceTableWriter::ConsumeRow(impala::TupleRow*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:307:52 #10 0x19c2a1e in impala::HdfsSequenceTableWriter::AppendRows(impala::RowBatch*, std::vector<int, std::allocator<int> > const&, bool*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-sequence-table-writer.cc:115:35 #11 0x19b1d97 in impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*, impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition, std::default_delete<impala::OutputPartition> >, std::vector<int, std::allocator<int> > >*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-table-sink.cc:281:31 #12 0x19b57a7 in impala::HdfsTableSink::Send(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-table-sink.cc:628:31 #13 0x1d1688a in impala::PlanFragmentExecutor::ExecInternal() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:359:31 #14 0x1d16155 in impala::PlanFragmentExecutor::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:337:14 #15 0x1d0dd3a in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/fragment-instance-state.cc:68:7 #16 0x1d1c1cf in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:110:3 #17 0x1d1da57 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #18 0x12f30a2 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:766:14 #19 0x16c5ab5 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:325:3 #20 0x16cfb8a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457:9 #21 0x16cfa17 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20:16 #22 0x1da7249 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da7249) Thread T22312 created by T9927 here: #0 0xf72df9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238 #1 0x1da6629 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da6629) Thread T9927 created by T75 here: #0 0xf72df9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238 #1 0x1da6629 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da6629) Thread T75 created by T74 here: #0 0xf72df9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238 #1 0x1da6629 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da6629) Thread T74 created by T0 here: #0 0xf72df9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238 #1 0x1da6629 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1da6629) SUMMARY: AddressSanitizer: heap-use-after-free /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:438 in memcpy Shadow bytes around the buggy address: 0x0c108017b5f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b610: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b620: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b630: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa =>0x0c108017b640: fa fa fa fa fd fd fd[fd]fd fd fd fd fd fd fd fd 0x0c108017b650: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b660: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b670: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x0c108017b680: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fa 0x0c108017b690: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Heap right redzone: fb Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack partial redzone: f4 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==29329==ABORTING
Attachments
Attachments
Issue Links
- breaks
-
IMPALA-3079 Fix Sequence file writer (crashes or produces invalid files)
- Resolved
- relates to
-
IMPALA-3079 Fix Sequence file writer (crashes or produces invalid files)
- Resolved