Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9449

Crash in LZO scan with fuzz test

    XMLWordPrintableJSON

    Details

    • Target Version:
    • Epic Color:
      ghx-label-1

      Description

      	at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      *** Check failure stack trace: ***
          @          0x4e7bf5c  google::LogMessage::Fail()
          @          0x4e7d801  google::LogMessage::SendToLog()
          @          0x4e7b936  google::LogMessage::Flush()
          @          0x4e7eefd  google::LogMessageFatal::~LogMessageFatal()
          @     0x7fcf1248c2bd  impala::HdfsLzoTextScanner::ReadAndDecompressData()
          @     0x7fcf1248a0da  impala::HdfsLzoTextScanner::ReadData()
          @     0x7fcf1248a635  impala::HdfsLzoTextScanner::FillByteBuffer()
          @          0x269f4ae  impala::HdfsTextScanner::FillByteBufferWrapper()
          @          0x269c047  impala::HdfsTextScanner::FinishScanRange()
          @          0x269f272  impala::HdfsTextScanner::GetNextInternal()
          @     0x7fcf12488b0d  impala::HdfsLzoTextScanner::GetNextInternal()
          @          0x2655808  impala::HdfsScanner::ProcessSplit()
          @          0x27c1a4b  impala::HdfsScanNode::ProcessSplit()
          @          0x27c0c20  impala::HdfsScanNode::ScannerThread()
          @          0x27bff7d  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_18ThreadResourcePoolEENKUlvE_clEv
          @          0x27c2571  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_18ThreadResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
          @          0x1fb39d5  boost::function0<>::operator()()
          @          0x257402a  impala::Thread::SuperviseThread()
          @          0x257c2ae  boost::_bi::list5<>::operator()<>()
          @          0x257c1d2  boost::_bi::bind_t<>::operator()()
          @          0x257c195  boost::detail::thread_data<>::run()
          @          0x3da22f9  thread_proxy
          @     0x7fcfdab10e24  start_thread
          @     0x7fcfd767c34c  __clone
      rypto.CryptoInputStream.read(CryptoInputStr
      
      CORE: ./core.1583079954.31711.impalad
      BINARY: ./be/build/latest/service/impalad
      Core was generated by `/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be'.
      Program terminated with signal SIGABRT, Aborted.
      #0  0x00007f231918b1f7 in raise () from /lib64/libc.so.6
      To enable execution of this file add
      	add-auto-load-safe-path /data0/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/gcc-4.9.2/lib64/libstdc++.so.6.0.20-gdb.py
      line to your configuration file "/var/lib/jenkins/.gdbinit".
      To completely disable this security protection add
      	set auto-load safe-path /
      line to your configuration file "/var/lib/jenkins/.gdbinit".
      For more information about this security protection see the
      "Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
      	info "(gdb)Auto-loading safe path"
      #0  0x00007f231918b1f7 in raise () from /lib64/libc.so.6
      #1  0x00007f231918c8e8 in abort () from /lib64/libc.so.6
      #2  0x0000000004e85504 in google::DumpStackTraceAndExit() ()
      #3  0x0000000004e7bf5d in google::LogMessage::Fail() ()
      #4  0x0000000004e7d802 in google::LogMessage::SendToLog() ()
      #5  0x0000000004e7b937 in google::LogMessage::Flush() ()
      #6  0x0000000004e7eefe in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x00007f2254b1d2be in impala::HdfsLzoTextScanner::ReadAndDecompressData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:582
      #8  0x00007f2254b1b0db in impala::HdfsLzoTextScanner::ReadData (this=0x182062c0, pool=0xdb6cb68) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:307
      #9  0x00007f2254b1b636 in impala::HdfsLzoTextScanner::FillByteBuffer (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:342
      #10 0x000000000269f4af in impala::HdfsTextScanner::FillByteBufferWrapper (this=0x182062c0, pool=0xdb6cb68, eosr=0x7f224091f49e, num_bytes=65536) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:476
      #11 0x000000000269c048 in impala::HdfsTextScanner::FinishScanRange (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:282
      #12 0x000000000269f273 in impala::HdfsTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-text-scanner.cc:467
      #13 0x00007f2254b19b0e in impala::HdfsLzoTextScanner::GetNextInternal (this=0x182062c0, row_batch=0xdb6cb40) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala-lzo/hdfs-lzo-text-scanner.cc:144
      #14 0x0000000002655809 in impala::HdfsScanner::ProcessSplit (this=0x182062c0) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scanner.cc:135
      #15 0x00000000027c1a4c in impala::HdfsScanNode::ProcessSplit (this=0xa3d78d00, filter_ctxs=..., expr_results_pool=0x7f2240920410, scan_range=0x109d1f180, scanner_thread_reservation=0x7f2240920368) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:515
      #16 0x00000000027c0c21 in impala::HdfsScanNode::ScannerThread (this=0xa3d78d00, first_thread=false, scanner_thread_reservation=8192) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:417
      #17 0x00000000027bff7e in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7f2240920b98) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/exec/hdfs-scan-node.cc:338
      #18 0x00000000027c2572 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159
      #19 0x0000000001fb39d6 in boost::function0<void>::operator() (this=0x7f2240920b90) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:771
      #20 0x000000000257402b in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) (name=..., category=..., functor=..., parent_thread_info=0x7f2242720840, thread_started=0x7f224271f540) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/be/src/util/thread.cc:360
      #21 0x000000000257c2af in boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) (this=0x1989013c0, f=@0x1989013b8: 0x2573cc4 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*)>, a=...) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531
      #22 0x000000000257c1d3 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() (this=0x1989013b8) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222
      #23 0x000000000257c196 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::run() (this=0x198901200) at /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116
      #24 0x0000000003da22fa in thread_proxy ()
      #25 0x00007f231c6e2e25 in start_thread () from /lib64/libpthread.so.0
      #26 0x00007f231924e34d in clone () from /lib64/libc.so.6
      

      This crashed all 3 impalads at around the same time:

      Log file created at: 2020/03/01 08:26:07
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:26:07.666946 27213 hdfs-lzo-text-scanner.cc:582] f44a5121256f5b7d:8e0bf92c00000001] Check failed: stream_->eof() 
      
      Log file created at: 2020/03/01 08:25:53
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:25:53.913563 25657 hdfs-lzo-text-scanner.cc:582] 1d4781605704837d:2a54525400000001] Check failed: stream_->eof() 
      
      Log file created at: 2020/03/01 08:25:39
      Running on machine: impala-ec2-centos74-m5-4xlarge-ondemand-0a0f.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      

      There was some noise in the log around the same time, unsure if it's relevant

      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.crypto.CryptoInputStream.read(CryptoInputStream.java:728)
      	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:193)
      hadoopZeroCopyRead: ZeroCopyCursor#read failed error:
      ReadOnlyBufferException: java.nio.ReadOnlyBufferException
      	at java.nio.DirectByteBufferR.put(DirectByteBufferR.java:350)
      	at org.apache.hadoop.crypto.CryptoInputStream.decrypt(CryptoInputStream.java:660)
      	at org.apache.hadoop.cF0301 08:25:39.334376 21960 hdfs-lzo-text-scanner.cc:582] 744cd34df9463fba:610c3d1000000003] Check failed: stream_->eof() 
      *** Check failure stack trace: ***
      

      This was testing with a slightly old impala-lzo commit: https://github.com/cloudera/impala-lzo/commit/36d19a8a4559ec6703e5879684b0d341c75c0202

      I'll attach logs and more info soon.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: