[IMPALA-1291] Parquet read fails if io buffer size is less than the footer size

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 1.4.1
• Fix Version/s: Impala 2.11.0
• Component/s: Backend
• Labels: crash downgraded
• Target Version: Product Backlog
• Assignee: Zoltán Borók-Nagy
• Reporter: Nong Li
• Created: 22/Sep/14 19:24
• Updated: 02/Nov/17 16:17
• Resolved: 02/Nov/17 16:17

Description

This will practically only happen if --read_size is manually set to be very low (e.g. for testing) so that is is less than HdfsParquetScanner::FOOTER_SIZE (which is currently 102400). It fails with this dcheck:

F0922 12:20:03.093201 24325 hdfs-parquet-scanner.cc:897] Check failed: stream_->eosr() 
*** Check failure stack trace: ***
    @           0xf251dd  google::LogMessage::Fail()
    @           0xf2797f  google::LogMessage::SendToLog()
    @           0xf24dcb  google::LogMessage::Flush()
    @           0xf2820d  google::LogMessageFatal::~LogMessageFatal()
    @     0x7f68a73e9f25  impala::HdfsParquetScanner::ProcessFooter()
    @     0x7f68a73e8af6  impala::HdfsParquetScanner::ProcessSplit()
    @     0x7f68a7390e8d  impala::HdfsScanNode::ScannerThread()
    @     0x7f68a73a8dae  boost::_mfi::mf0<>::operator()()
    @     0x7f68a73a8436  boost::_bi::list1<>::operator()<>()
    @     0x7f68a73a7093  boost::_bi::bind_t<>::operator()()
    @     0x7f68a73a594b  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @     0x7f68a5be9d6c  boost::function0<>::operator()()
    @     0x7f68a47cd33c  impala::Thread::SuperviseThread()
    @     0x7f68a47d5d6a  boost::_bi::list4<>::operator()<>()
    @     0x7f68a47d5cb3  boost::_bi::bind_t<>::operator()()
    @     0x7f68a47d5c46  boost::detail::thread_data<>::run()
    @     0x7f68a3cf8ce9  (unknown)
    @     0x7f68a3ad6e9a  start_thread
    @     0x7f68a116accd  (unknown)

Issue Links

• is related to: IMPALA-2213 Parquet read can fail if file metadata is stale