Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.7.0, Impala 2.8.0
Description
I can reproduce this by running a local stress test under ASAN:
./buildall.sh -asan -noclean -notests -skiptests -ninja -start_impala_cluster ./tests/stress/concurrent_select.py --cancel-probability=0.5 --max-queries 10000 --tpch-db=tpch_20_parquet --fail-upon-successive-errors=100
The full log is attached, but the condensed version is:
==8612==ERROR: AddressSanitizer: heap-use-after-free on address 0x63100b00480b at pc 0x000000fcc9c5 bp 0x7fb48bb23520 sp 0x7fb48bb22cd0 READ of size 18 at 0x63100b00480b thread T3933 #0 0xfcc9c4 in __asan_memcpy /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:393 #1 0x1bf87e5 in impala::BufferedTupleStream::CopyStrings(impala::Tuple const*, std::vector<impala::SlotDescriptor*, std::allocator<impala::SlotDescriptor*> > const&) /tmp/be/src/runtime/buffered-tuple-stream.cc:840:7 #2 0x1bfb429 in bool impala::BufferedTupleStream::DeepCopyInternal<false>(impala::TupleRow*) /tmp/be/src/runtime/buffered-tuple-stream.cc:815:30 #3 0x1bf86ac in impala::BufferedTupleStream::DeepCopy(impala::TupleRow*) /tmp/be/src/runtime/buffered-tuple-stream.cc:752:12 #4 0x7fb56e118464 (<unknown module>) #5 0x18921f5 in impala::PartitionedHashJoinNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) /tmp/be/src/exec/partitioned-hash-join-node.cc:538:24 #6 0x19001c7 in impala::BlockingJoinNode::GetFirstProbeRow(impala::RuntimeState*) /tmp/be/src/exec/blocking-join-node.cc:240:31 #7 0x188ba03 in impala::PartitionedHashJoinNode::Open(impala::RuntimeState*) /tmp/be/src/exec/partitioned-hash-join-node.cc:187:29 #8 0x186732b in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) /tmp/be/src/exec/partitioned-aggregation-node.cc:317:29 #9 0x1c60ece in impala::PlanFragmentExecutor::OpenInternal() /tmp/be/src/runtime/plan-fragment-executor.cc:322:31 #10 0x1c60a05 in impala::PlanFragmentExecutor::Open() /tmp/be/src/runtime/plan-fragment-executor.cc:295:19 0x63100b00480b is located 11 bytes inside of 65528-byte region [0x63100b004800,0x63100b0147f8) freed by thread T3960 here: #0 0xfe2620 in __interceptor_free /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:38 #1 0x13a1162 in impala::MemPool::FreeAll() /tmp/be/src/runtime/mem-pool.cc:89:5 #2 0x17cbede in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) /tmp/be/src/exec/hdfs-parquet-scanner.cc:540:9 #3 0x17c86f4 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) /tmp/be/src/exec/hdfs-parquet-scanner.cc:402:19 #4 0x17c7714 in impala::HdfsParquetScanner::ProcessSplit() /tmp/be/src/exec/hdfs-parquet-scanner.cc:334:31 #5 0x17592a5 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) /tmp/be/src/exec/hdfs-scan-node.cc:527:12 #6 0x17583df in impala::HdfsScanNode::ScannerThread() /tmp/be/src/exec/hdfs- previously allocated by thread T3960 here: #0 0xfe2928 in __interceptor_malloc /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:52 #1 0x139f526 in impala::MemPool::FindChunk(long, bool) /tmp/be/src/runtime/mem-pool.cc:149:45 #2 0x13042b9 in unsigned char* impala::MemPool::Allocate<true>(long) /tmp/be/src/runtime/mem-pool.h:236:32 #3 0x180a950 in impala::BaseScalarColumnReader::ReadDataPage() /tmp/be/src/exec/parquet-column-readers.cc:907:11 #4 0x180e495 in impala::BaseScalarColumnReader::NextPage() /tmp/be/src/exec/parquet-column-readers.cc:989:28 #5 0x18432d8 in bool impala::ScalarColumnReader<impala::StringValue, true>::ReadValueBatch<false>(impala::MemPool*, int, int, unsigned char*, int*) /tmp/be/src/exec/parquet-column-readers.cc:299:14 #6 0x17cba79 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) /tmp/be/src/exec/hdfs-parquet-scanner.cc:532:30 #7 0x17c86f4 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) /tmp/be/src/exec/hdfs-parquet-scanner.cc:402:19 #8 0x17c7714 in impala::HdfsParquetScanner::ProcessSplit() /tmp/be/src/exec/hdfs-parquet-scanner.cc:334:31 #9 0x17592a5 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) /tmp/be/src/exec/hdfs-scan-node.cc:527:12 #10 0x17583df in impala::HdfsScanNode::ScannerThread() /tmp/be/src/exec/hdfs-scan-node.cc:418:16