Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8090

DiskIoMgrTest.SyncReadTest hits file_ != nullptr DCHECK in LocalFileReader::ReadFromPos()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 3.2.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Test output:

      45/99 Test #45: disk-io-mgr-test .................***Exception: Other 43.29 sec
      Turning perftools heap leak checking off
      [==========] Running 25 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 25 tests from DiskIoMgrTest
      [ RUN      ] DiskIoMgrTest.SingleWriter
      19/01/16 15:57:09 INFO util.JvmPauseMonitor: Starting JVM pause monitor
      [       OK ] DiskIoMgrTest.SingleWriter (3407 ms)
      [ RUN      ] DiskIoMgrTest.InvalidWrite
      [       OK ] DiskIoMgrTest.InvalidWrite (281 ms)
      [ RUN      ] DiskIoMgrTest.WriteErrors
      [       OK ] DiskIoMgrTest.WriteErrors (235 ms)
      [ RUN      ] DiskIoMgrTest.SingleWriterCancel
      [       OK ] DiskIoMgrTest.SingleWriterCancel (1165 ms)
      [ RUN      ] DiskIoMgrTest.SingleReader
      [       OK ] DiskIoMgrTest.SingleReader (5835 ms)
      [ RUN      ] DiskIoMgrTest.SingleReaderSubRanges
      [       OK ] DiskIoMgrTest.SingleReaderSubRanges (16404 ms)
      [ RUN      ] DiskIoMgrTest.AddScanRangeTest
      [       OK ] DiskIoMgrTest.AddScanRangeTest (1210 ms)
      [ RUN      ] DiskIoMgrTest.SyncReadTest
      *** Check failure stack trace: ***
          @          0x4825dcc
          @          0x4827671
          @          0x48257a6
          @          0x4828d6d
          @          0x1af39ec
          @          0x1ae90a4
          @          0x1ac30ea
          @          0x1accad3
          @          0x1acc660
          @          0x1acbf3e
          @          0x1acb62d
          @          0x1b03671
          @          0x1f79988
          @          0x1f82b60
          @          0x1f82a84
          @          0x1f82a47
          @          0x3751579
          @       0x3ea4807850
          @       0x3ea44e894c
      Wrote minidump to /data/jenkins/workspace/<...>/repos/Impala/logs/be_tests/minidumps/disk-io-mgr-test/5bbf76f7-e5d6-4ac9-bdae9d9b-065c32ec.dmp
      

      Error:

                      Operating system: Linux
                        0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP Tue Jul 16 21:33:24 UTC 2013 x86_64
      CPU: amd64
           family 6 model 45 stepping 7
           8 CPUs
      
      GPU: UNKNOWN
      
      Crash reason:  SIGABRT
      Crash address: 0x45200002fa1
      Process uptime: not available
      
      Thread 205 (crashed)
       0  libc-2.12.so + 0x328e5
          rax = 0x0000000000000000   rdx = 0x0000000000000006
          rcx = 0xffffffffffffffff   rbx = 0x0000000006adf9c0
          rsi = 0x0000000000000563   rdi = 0x0000000000002fa1
          rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc78
           r8 = 0x00007f8009b8fd00    r9 = 0x0000000000000563
          r10 = 0x0000000000000008   r11 = 0x0000000000000202
          r12 = 0x0000000006adfa40   r13 = 0x000000000000001f
          r14 = 0x0000000006ae7384   r15 = 0x0000000006adf9c0
          rip = 0x0000003ea44328e5
          Found by: given as instruction pointer in context
       1  libc-2.12.so + 0x340c5
          rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc80
          rip = 0x0000003ea44340c5
          Found by: stack scanning
       2  disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x21]
          rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc88
          rip = 0x0000000001acbf3e
          Found by: stack scanning
       3  disk-io-mgr-test!google::LogMessage::Flush() + 0x157
          rbx = 0x0000000000000007   rbp = 0x0000000006adf980
          rsp = 0x00007f8009b8fff0   rip = 0x00000000048257a7
          Found by: call frame info
       4  disk-io-mgr-test!google::LogMessageFatal::~LogMessageFatal() + 0xe
          rbx = 0x00007f8009b90110   rbp = 0x00007f8009b903f0
          rsp = 0x00007f8009b90070   r12 = 0x0000000000000001
          r13 = 0x0000000006aee8b8   r14 = 0x000000000c213538
          r15 = 0x0000000000000007   rip = 0x0000000004828d6e
          Found by: call frame info
       5  disk-io-mgr-test!impala::io::LocalFileReader::ReadFromPos(long, unsigned char*, long, long*, bool*) [local-file-reader.cc : 67 + 0x10]
          rbx = 0x0000000000000001   rbp = 0x00007f8009b903f0
          rsp = 0x00007f8009b90090   r12 = 0x0000000000000001
          r13 = 0x0000000006aee8b8   r14 = 0x000000000c213538
          r15 = 0x0000000000000007   rip = 0x0000000001af39ed
          Found by: call frame info
       6  disk-io-mgr-test!impala::io::ScanRange::DoRead(int) [scan-range.cc : 219 + 0x5b]
          rbx = 0x000000000c4f71e0   rbp = 0x00007f8009b90620
          rsp = 0x00007f8009b90400   r12 = 0x0000000001af36e4
          r13 = 0x000000000000000d   r14 = 0x000000000c213538
          r15 = 0x0000000000000007   rip = 0x0000000001ae90a5
          Found by: call frame info
       7  disk-io-mgr-test!impala::io::DiskQueue::DiskThreadLoop(impala::io::DiskIoMgr*) [disk-io-mgr.cc : 425 + 0x17]
          rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b906c0
          rsp = 0x00007f8009b90630   r12 = 0x00007fff99de21c0
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001ac30eb
          Found by: call frame info
       8  disk-io-mgr-test!boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>::operator()(impala::io::DiskQueue*, impala::io::DiskIoMgr*) const [mem_fn_template.hpp : 165 + 0x1e]
          rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b906f0
          rsp = 0x00007f8009b906d0   r12 = 0x00007fff99de21c0
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001accad4
          Found by: call frame info
       9  disk-io-mgr-test!void boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> >::operator()<boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>&, boost::_bi::list0&, int) [bind.hpp : 313 + 0x4f]
          rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b90730
          rsp = 0x00007f8009b90700   r12 = 0x00007fff99de21c0
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001acc661
          Found by: call frame info
      10  disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x22]
          rbx = 0x0000000000000563   rbp = 0x00007f8009b90780
          rsp = 0x00007f8009b90740   r12 = 0x0000000000000000
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001acbf3f
          Found by: call frame info
      11  disk-io-mgr-test!boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >, void>::invoke(boost::detail::function::function_buffer&) [function_template.hpp : 153 + 0xc]
          rbx = 0x0000000000000563   rbp = 0x00007f8009b907b0
          rsp = 0x00007f8009b90790   r12 = 0x0000000000000000
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001acb62e
          Found by: call frame info
      12  disk-io-mgr-test!boost::function0<void>::operator()() const [function_template.hpp : 767 + 0x11]
          rbx = 0x0000000000000563   rbp = 0x00007f8009b907f0
          rsp = 0x00007f8009b907c0   r12 = 0x0000000000000000
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001b03672
          Found by: call frame info
      13  disk-io-mgr-test!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 359 + 0xf]
          rbx = 0x0000000000000563   rbp = 0x00007f8009b90b50
          rsp = 0x00007f8009b90800   r12 = 0x0000000000000000
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
          r15 = 0x0000000000000007   rip = 0x0000000001f79989
          Found by: call frame info
      14  disk-io-mgr-test!void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) [bind.hpp : 525 + 0x15]
          rbx = 0x0000000001f79622   rbp = 0x00007f8009b90bc0
          rsp = 0x00007f8009b90b60   r12 = 0x0000000000000000
          r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
      

      It can be reproduced by looping the test for a few minutes, e.g.

      $ ninja -j $IMPALA_BUILD_THREADS disk-io-mgr-test && while ./be/build/latest/runtime/io/disk-io-mgr-test --gtest_filter=*SyncReadTest*; do date; don
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                dknupp David Knupp
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: