[IMPALA-2213] Parquet read can fail if file metadata is stale

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 2.3.0, Impala 2.2.8
• Target Version: Impala 2.3.0, Impala 2.2.8
• Assignee: Skye Wanderman-Milne
• Reporter: Skye Wanderman-Milne
• Created: 17/Aug/15 21:38
• Updated: 10/Nov/15 20:12
• Resolved: 04/Oct/15 23:27

Description

This will cause the same DCHECK to fail as IMPALA-1291, but the cause is different.

IMPALA-1291 will cause failures if the --read_size impalad flag is set very low (below 100KB, default is 8MB), and will fail consistently over any files that are > 100KB in size.

This failure occurs if Impala thinks the file is longer than it actually is, e.g., the file was rewritten and "invalidate metadata" was not called. This could appear as sporadic crashes if the workload involves concurrent reads and writes to the same table.

Here's the stack trace from the DCHECK:

F0922 12:20:03.093201 24325 hdfs-parquet-scanner.cc:897] Check failed: stream_->eosr() 
*** Check failure stack trace: ***
    @           0xf251dd  google::LogMessage::Fail()
    @           0xf2797f  google::LogMessage::SendToLog()
    @           0xf24dcb  google::LogMessage::Flush()
    @           0xf2820d  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f68a73e9f25  impala::HdfsParquetScanner::ProcessFooter()
    @     0x7f68a73e8af6  impala::HdfsParquetScanner::ProcessSplit()
    @     0x7f68a7390e8d  impala::HdfsScanNode::ScannerThread()
    @     0x7f68a73a8dae  boost::_mfi::mf0<>::operator()()
    @     0x7f68a73a8436  boost::_bi::list1<>::operator()<>()
    @     0x7f68a73a7093  boost::_bi::bind_t<>::operator()()
    @     0x7f68a73a594b  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @     0x7f68a5be9d6c  boost::function0<>::operator()()
    @     0x7f68a47cd33c  impala::Thread::SuperviseThread()
    @     0x7f68a47d5d6a  boost::_bi::list4<>::operator()<>()
    @     0x7f68a47d5cb3  boost::_bi::bind_t<>::operator()()
    @     0x7f68a47d5c46  boost::detail::thread_data<>::run()
    @     0x7f68a3cf8ce9  (unknown)
    @     0x7f68a3ad6e9a  start_thread
    @     0x7f68a116accd  (unknown)

Issue Links

• relates to: IMPALA-1291 Parquet read fails if io buffer size is less than the footer size