[IMPALA-6946] Hit DCHECK in impala::RleBatchDecoder<unsigned int>::GetRepeatedValue

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.11.0, Impala 3.0, Impala 2.12.0, Impala 2.13.0, Impala 3.1.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: crash
• Target Version: Impala 2.13.0, Impala 3.1.0
• Epic Color: ghx-label-9
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 27/Apr/18 16:57
• Updated: 22/May/18 13:23
• Resolved: 07/May/18 22:01

Description

The bug comes from conversion between signed and unsigned.

 DCHECK_GT(num_repeats_to_consume, 0);

(gdb) p num_repeats_to_consume
$2 = -1003251240

#2  0x0000000001f422f0 in impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x6f547d50, pool=0x7fb37bdbb330, max_values=0, 
    tuple_size=90, tuple_mem=0x0, num_values=0x7fb37bdbb330) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#3  0x0000000003fceea7 in google::LogMessage::Flush() ()
#4  0x0000000003fd246e in google::LogMessageFatal::~LogMessageFatal() ()
#5  0x0000000001f97197 in impala::RleBatchDecoder<unsigned int>::GetRepeatedValue (this=0x1803fc6a0, num_repeats_to_consume=-1003251240)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/rle-encoding.h:493
#6  0x0000000001f96d7b in impala::DictDecoder<impala::StringValue>::DecodeNextValue (this=0x1803fc698, value=0x16b7e9000)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:455
#7  0x0000000001f94528 in impala::DictDecoder<impala::StringValue>::GetNextValue (value=0x16b7e9000, this=0x1803fc698)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/dict-encoding.h:446
#8  impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::ReadSlot<true, false> (pool=0x6f547d98, tuple=0x16b7e9000, this=0x1803fc400)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:499
#9  impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::MaterializeValueBatch<false, true, false> (this=0x1803fc400, pool=0x6f547d98, max_values=1, tuple_size=90, 
    tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:413
#10 0x0000000001f67b4a in impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::MaterializeValueBatch<false, true> (this=0x1803fc400, pool=0x6f547d98, max_values=1, 
    tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x7fb37bdbb5f0) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:436
#11 0x0000000001f657c2 in impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::ReadValueBatch<false> (this=0x1803fc400, pool=0x6f547d98, max_values=1, tuple_size=90, 
    tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:354
#12 0x0000000001f422f0 in impala::ScalarColumnReader<impala::StringValue, (parquet::Type::type)6, true>::ReadNonRepeatedValueBatch (this=0x1803fc400, pool=0x6f547d98, max_values=1, 
    tuple_size=90, tuple_mem=0x16b7e9000 "", num_values=0x6f547d50) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/parquet-column-readers.cc:252
#13 0x0000000001d70a1c in impala::HdfsParquetScanner::AssembleRows (this=0x1a926000, column_readers=..., row_batch=0xdfc0780, skip_row_group=0x1a9261b8)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:936
#14 0x0000000001d6d4f7 in impala::HdfsParquetScanner::GetNextInternal (this=0x1a926000, row_batch=0xdfc0780)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:434
#15 0x0000000001d6b6c6 in impala::HdfsParquetScanner::ProcessSplit (this=0x1a926000) at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:332
#16 0x0000000001cf30e4 in impala::HdfsScanNode::ProcessSplit (this=0x10c21000, filter_ctxs=..., expr_results_pool=0x7fb37bdbc480, scan_range=0xf3276c0, scanner_thread_reservation=0x7fb37bdbc400)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:482
#17 0x0000000001cf2476 in impala::HdfsScanNode::ScannerThread (this=0x10c21000, scanner_thread_reservation=90112)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:385
#18 0x0000000001cf18a8 in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7fb37bdbcbc8)
    at /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:300

I don't believe this will cause a crash on a release build and can only be triggered by a very specific corrupt parquet file.

It was introduced by IMPALA-4177

Issue Links

• is broken by: IMPALA-4177 Add batch dictionary/RLE decoding in Parquet