Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1291

Parquet read fails if io buffer size is less than the footer size

    Details

      Description

      This will practically only happen if --read_size is manually set to be very low (e.g. for testing) so that is is less than HdfsParquetScanner::FOOTER_SIZE (which is currently 102400). It fails with this dcheck:

      F0922 12:20:03.093201 24325 hdfs-parquet-scanner.cc:897] Check failed: stream_->eosr() 
      *** Check failure stack trace: ***
          @           0xf251dd  google::LogMessage::Fail()
          @           0xf2797f  google::LogMessage::SendToLog()
          @           0xf24dcb  google::LogMessage::Flush()
          @           0xf2820d  google::LogMessageFatal::~LogMessageFatal()
          @     0x7f68a73e9f25  impala::HdfsParquetScanner::ProcessFooter()
          @     0x7f68a73e8af6  impala::HdfsParquetScanner::ProcessSplit()
          @     0x7f68a7390e8d  impala::HdfsScanNode::ScannerThread()
          @     0x7f68a73a8dae  boost::_mfi::mf0<>::operator()()
          @     0x7f68a73a8436  boost::_bi::list1<>::operator()<>()
          @     0x7f68a73a7093  boost::_bi::bind_t<>::operator()()
          @     0x7f68a73a594b  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @     0x7f68a5be9d6c  boost::function0<>::operator()()
          @     0x7f68a47cd33c  impala::Thread::SuperviseThread()
          @     0x7f68a47d5d6a  boost::_bi::list4<>::operator()<>()
          @     0x7f68a47d5cb3  boost::_bi::bind_t<>::operator()()
          @     0x7f68a47d5c46  boost::detail::thread_data<>::run()
          @     0x7f68a3cf8ce9  (unknown)
          @     0x7f68a3ad6e9a  start_thread
          @     0x7f68a116accd  (unknown)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                boroknagyz Zoltán Borók-Nagy
                Reporter:
                nong_impala_60e1 Nong Li
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: