Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 3.1.0
-
ghx-label-7
Description
ASAN tests have been failing for the last few runs. Here is the output:
==117268==ERROR: AddressSanitizer: use-after-poison on address 0x7ef9312e0ec0 at pc 0x0000016a0c98 bp 0x7ef91e502540 sp 0x7ef91e501cf0 READ of size 32681 at 0x7ef9312e0ec0 thread T82364 #0 0x16a0c97 in __interceptor_memcpy /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738 #1 0x7f02e43c2dca in jni_SetByteArrayRegion (/usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so+0x6d3dca) #2 0x4ae29b7 in hdfsWrite /container.redhat6/build/cdh/hadoop/3.0.0-cdh6.x-SNAPSHOT/rpm/BUILD/hadoop-3.0.0-cdh6.x-SNAPSHOT/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:1626 #3 0x332ca67 in impala::HdfsTableWriter::Write(unsigned char const*, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-writer.cc:46:13 #4 0x25ac037 in impala::HdfsParquetTableWriter::BaseColumnWriter::Flush(long*, long*, long*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:736:5 #5 0x25b0e57 in impala::HdfsParquetTableWriter::FlushCurrentRowGroup() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1195:5 #6 0x25b4147 in impala::HdfsParquetTableWriter::Finalize() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1161:3 #7 0x251e424 in impala::HdfsTableSink::FinalizePartitionFile(impala::RuntimeState*, impala::OutputPartition*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:620:5 #8 0x2523d1b in impala::HdfsTableSink::FlushFinal(impala::RuntimeState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:660:5 #9 0x1fc09f0 in impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:324:3 #10 0x1fbbd5c in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14 #11 0x1fd5f94 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24 #12 0x1cdef96 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14 #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3 #14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9 #15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27 #16 0x39a11e9 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9) #17 0x7f02e19f4e24 in start_thread (/lib64/libpthread.so.0+0x7e24) #18 0x7f02e150b34c in __clone (/lib64/libc.so.6+0xf834c) 0x7ef9312e0ec0 is located 161472 bytes inside of 524288-byte region [0x7ef9312b9800,0x7ef931339800) allocated by thread T82364 here: #0 0x16fcd88 in malloc /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:67 #1 0x1f2a5ec in impala::MemPool::FindChunk(long, bool) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.cc:140:45 #2 0x1f44c37 in unsigned char* impala::MemPool::Allocate<false>(long, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:284:9 #3 0x1f44c37 in impala::MemPool::Allocate(long) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/mem-pool.h:107 #4 0x25ad32c in impala::HdfsParquetTableWriter::BaseColumnWriter::FinalizeCurrentPage() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:797:61 #5 0x25bbdea in impala::HdfsParquetTableWriter::BaseColumnWriter::AppendRow(impala::TupleRow*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:590:5 #6 0x25b36cc in impala::HdfsParquetTableWriter::AppendRows(impala::RowBatch*, std::vector<int, std::allocator<int> > const&, bool*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-parquet-table-writer.cc:1124:7 #7 0x251ddec in impala::HdfsTableSink::WriteRowsToPartition(impala::RuntimeState*, impala::RowBatch*, std::pair<std::unique_ptr<impala::OutputPartition, std::default_delete<impala::OutputPartition> >, std::vector<int, std::allocator<int> > >*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:256:5 #8 0x2522f01 in impala::HdfsTableSink::Send(impala::RuntimeState*, impala::RowBatch*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/exec/hdfs-table-sink.cc:590:5 #9 0x1fc07ad in impala::FragmentInstanceState::ExecInternal() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:316:5 #10 0x1fbbd5c in impala::FragmentInstanceState::Exec() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/fragment-instance-state.cc:95:14 #11 0x1fd5f94 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/query-state.cc:478:24 #12 0x1cdef96 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:766:14 #13 0x23ac27e in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:359:3 #14 0x23b7708 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:525:9 #15 0x23b755b in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20:27 #16 0x39a11e9 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a11e9) Thread T82364 created by T82363 here: #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317 #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9) #2 0x45e0360d (<unknown module>) Thread T82363 created by T207 here: #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317 #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9) #2 0x45e0360d (<unknown module>) Thread T207 created by T199 here: #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317 #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9) #2 0x45e0360d (<unknown module>) Thread T199 created by T198 here: #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317 #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9) #2 0x45e0360d (<unknown module>) Thread T198 created by T0 here: #0 0x164420d in __interceptor_pthread_create /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:317 #1 0x39a05c9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x39a05c9) #2 0x45e0360d (<unknown module>) SUMMARY: AddressSanitizer: use-after-poison /data/jenkins/workspace/impala-toolchain-package-build/label/impala-toolchnbld-cent70-ec2-c3-4xl-ondem/toolchain/source/llvm/llvm-5.0.1.src-p1/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:738 in __interceptor_memcpy Shadow bytes around the buggy address: 0x0fdfa6254180: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa6254190: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa62541a0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa62541b0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa62541c0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 =>0x0fdfa62541d0: f7 f7 f7 f7 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 0x0fdfa62541e0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa62541f0: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa6254200: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa6254210: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 0x0fdfa6254220: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==117268==ABORTING