Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8090

DiskIoMgrTest.SyncReadTest hits file_ != nullptr DCHECK in LocalFileReader::ReadFromPos()

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • Impala 3.2.0
    • Impala 3.2.0
    • Backend
    • None

    Description

      Test output:

      45/99 Test #45: disk-io-mgr-test .................***Exception: Other 43.29 sec
Turning perftools heap leak checking off
[==========] Running 25 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 25 tests from DiskIoMgrTest
[ RUN      ] DiskIoMgrTest.SingleWriter
19/01/16 15:57:09 INFO util.JvmPauseMonitor: Starting JVM pause monitor
[       OK ] DiskIoMgrTest.SingleWriter (3407 ms)
[ RUN      ] DiskIoMgrTest.InvalidWrite
[       OK ] DiskIoMgrTest.InvalidWrite (281 ms)
[ RUN      ] DiskIoMgrTest.WriteErrors
[       OK ] DiskIoMgrTest.WriteErrors (235 ms)
[ RUN      ] DiskIoMgrTest.SingleWriterCancel
[       OK ] DiskIoMgrTest.SingleWriterCancel (1165 ms)
[ RUN      ] DiskIoMgrTest.SingleReader
[       OK ] DiskIoMgrTest.SingleReader (5835 ms)
[ RUN      ] DiskIoMgrTest.SingleReaderSubRanges
[       OK ] DiskIoMgrTest.SingleReaderSubRanges (16404 ms)
[ RUN      ] DiskIoMgrTest.AddScanRangeTest
[       OK ] DiskIoMgrTest.AddScanRangeTest (1210 ms)
[ RUN      ] DiskIoMgrTest.SyncReadTest
*** Check failure stack trace: ***
    @          0x4825dcc
    @          0x4827671
    @          0x48257a6
    @          0x4828d6d
    @          0x1af39ec
    @          0x1ae90a4
    @          0x1ac30ea
    @          0x1accad3
    @          0x1acc660
    @          0x1acbf3e
    @          0x1acb62d
    @          0x1b03671
    @          0x1f79988
    @          0x1f82b60
    @          0x1f82a84
    @          0x1f82a47
    @          0x3751579
    @       0x3ea4807850
    @       0x3ea44e894c
Wrote minidump to /data/jenkins/workspace/<...>/repos/Impala/logs/be_tests/minidumps/disk-io-mgr-test/5bbf76f7-e5d6-4ac9-bdae9d9b-065c32ec.dmp

      Error:

                      Operating system: Linux
                  0.0.0 Linux 2.6.32-358.14.1.el6.centos.plus.x86_64 #1 SMP Tue Jul 16 21:33:24 UTC 2013 x86_64
CPU: amd64
     family 6 model 45 stepping 7
     8 CPUs

GPU: UNKNOWN

Crash reason:  SIGABRT
Crash address: 0x45200002fa1
Process uptime: not available

Thread 205 (crashed)
 0  libc-2.12.so + 0x328e5
    rax = 0x0000000000000000   rdx = 0x0000000000000006
    rcx = 0xffffffffffffffff   rbx = 0x0000000006adf9c0
    rsi = 0x0000000000000563   rdi = 0x0000000000002fa1
    rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc78
     r8 = 0x00007f8009b8fd00    r9 = 0x0000000000000563
    r10 = 0x0000000000000008   r11 = 0x0000000000000202
    r12 = 0x0000000006adfa40   r13 = 0x000000000000001f
    r14 = 0x0000000006ae7384   r15 = 0x0000000006adf9c0
    rip = 0x0000003ea44328e5
    Found by: given as instruction pointer in context
 1  libc-2.12.so + 0x340c5
    rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc80
    rip = 0x0000003ea44340c5
    Found by: stack scanning
 2  disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x21]
    rbp = 0x00007f8009b8ffe0   rsp = 0x00007f8009b8fc88
    rip = 0x0000000001acbf3e
    Found by: stack scanning
 3  disk-io-mgr-test!google::LogMessage::Flush() + 0x157
    rbx = 0x0000000000000007   rbp = 0x0000000006adf980
    rsp = 0x00007f8009b8fff0   rip = 0x00000000048257a7
    Found by: call frame info
 4  disk-io-mgr-test!google::LogMessageFatal::~LogMessageFatal() + 0xe
    rbx = 0x00007f8009b90110   rbp = 0x00007f8009b903f0
    rsp = 0x00007f8009b90070   r12 = 0x0000000000000001
    r13 = 0x0000000006aee8b8   r14 = 0x000000000c213538
    r15 = 0x0000000000000007   rip = 0x0000000004828d6e
    Found by: call frame info
 5  disk-io-mgr-test!impala::io::LocalFileReader::ReadFromPos(long, unsigned char*, long, long*, bool*) [local-file-reader.cc : 67 + 0x10]
    rbx = 0x0000000000000001   rbp = 0x00007f8009b903f0
    rsp = 0x00007f8009b90090   r12 = 0x0000000000000001
    r13 = 0x0000000006aee8b8   r14 = 0x000000000c213538
    r15 = 0x0000000000000007   rip = 0x0000000001af39ed
    Found by: call frame info
 6  disk-io-mgr-test!impala::io::ScanRange::DoRead(int) [scan-range.cc : 219 + 0x5b]
    rbx = 0x000000000c4f71e0   rbp = 0x00007f8009b90620
    rsp = 0x00007f8009b90400   r12 = 0x0000000001af36e4
    r13 = 0x000000000000000d   r14 = 0x000000000c213538
    r15 = 0x0000000000000007   rip = 0x0000000001ae90a5
    Found by: call frame info
 7  disk-io-mgr-test!impala::io::DiskQueue::DiskThreadLoop(impala::io::DiskIoMgr*) [disk-io-mgr.cc : 425 + 0x17]
    rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b906c0
    rsp = 0x00007f8009b90630   r12 = 0x00007fff99de21c0
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001ac30eb
    Found by: call frame info
 8  disk-io-mgr-test!boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>::operator()(impala::io::DiskQueue*, impala::io::DiskIoMgr*) const [mem_fn_template.hpp : 165 + 0x1e]
    rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b906f0
    rsp = 0x00007f8009b906d0   r12 = 0x00007fff99de21c0
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001accad4
    Found by: call frame info
 9  disk-io-mgr-test!void boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> >::operator()<boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>&, boost::_bi::list0&, int) [bind.hpp : 313 + 0x4f]
    rbx = 0x000000000c0e0f00   rbp = 0x00007f8009b90730
    rsp = 0x00007f8009b90700   r12 = 0x00007fff99de21c0
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001acc661
    Found by: call frame info
10  disk-io-mgr-test!boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >::operator()() [bind_template.hpp : 20 + 0x22]
    rbx = 0x0000000000000563   rbp = 0x00007f8009b90780
    rsp = 0x00007f8009b90740   r12 = 0x0000000000000000
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001acbf3f
    Found by: call frame info
11  disk-io-mgr-test!boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::io::DiskQueue, impala::io::DiskIoMgr*>, boost::_bi::list2<boost::_bi::value<impala::io::DiskQueue*>, boost::_bi::value<impala::io::DiskIoMgr*> > >, void>::invoke(boost::detail::function::function_buffer&) [function_template.hpp : 153 + 0xc]
    rbx = 0x0000000000000563   rbp = 0x00007f8009b907b0
    rsp = 0x00007f8009b90790   r12 = 0x0000000000000000
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001acb62e
    Found by: call frame info
12  disk-io-mgr-test!boost::function0<void>::operator()() const [function_template.hpp : 767 + 0x11]
    rbx = 0x0000000000000563   rbp = 0x00007f8009b907f0
    rsp = 0x00007f8009b907c0   r12 = 0x0000000000000000
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001b03672
    Found by: call frame info
13  disk-io-mgr-test!impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) [thread.cc : 359 + 0xf]
    rbx = 0x0000000000000563   rbp = 0x00007f8009b90b50
    rsp = 0x00007f8009b90800   r12 = 0x0000000000000000
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8
    r15 = 0x0000000000000007   rip = 0x0000000001f79989
    Found by: call frame info
14  disk-io-mgr-test!void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) [bind.hpp : 525 + 0x15]
    rbx = 0x0000000001f79622   rbp = 0x00007f8009b90bc0
    rsp = 0x00007f8009b90b60   r12 = 0x0000000000000000
    r13 = 0x00007fff99de1a90   r14 = 0x000000000c4f5dc8

      It can be reproduced by looping the test for a few minutes, e.g.

      $ ninja -j $IMPALA_BUILD_THREADS disk-io-mgr-test && while ./be/build/latest/runtime/io/disk-io-mgr-test --gtest_filter=*SyncReadTest*; do date; don

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4249 Clarify lifetime of DiskIoMgr::ScanRange objects

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          Activity

            People

              tarmstrong Tim Armstrong
              dknupp David Knupp
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: