Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5183

buffered-block-mgr-test: Writes did not complete after 500ms

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-8

      Description

      WaitForWrites failed, maybe this is flaky.

      06:56:45       Start 23: buffered-block-mgr-test
      06:56:45 23/88 Test #23: buffered-block-mgr-test ..........***Exception: Other 12.71 sec
      06:56:45 Turning perftools heap leak checking off
      06:56:45 SLF4J: Class path contains multiple SLF4J bindings.
      06:56:45 SLF4J: Found binding in [jar:file:/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/fe/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      06:56:45 SLF4J: Found binding in [jar:file:/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/testdata/target/dependency/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      06:56:45 SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      06:56:45 SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      06:56:45 [==========] Running 31 tests from 1 test case.
      06:56:45 [----------] Global test environment set-up.
      06:56:45 [----------] 31 tests from BufferedBlockMgrTest
      06:56:45 [ RUN      ] BufferedBlockMgrTest.GetNewBlock
      06:56:45 [       OK ] BufferedBlockMgrTest.GetNewBlock (1113 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.GetNewBlockSmallBlocks
      06:56:45 [       OK ] BufferedBlockMgrTest.GetNewBlockSmallBlocks (302 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.Pin
      06:56:45 [       OK ] BufferedBlockMgrTest.Pin (246 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.Eviction
      06:56:45 [       OK ] BufferedBlockMgrTest.Eviction (335 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.Deletion
      06:56:45 [       OK ] BufferedBlockMgrTest.Deletion (255 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.DeleteSingleBlocks
      06:56:45 [       OK ] BufferedBlockMgrTest.DeleteSingleBlocks (245 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.TransferBufferDuringWrite
      06:56:45 [       OK ] BufferedBlockMgrTest.TransferBufferDuringWrite (366 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.Close
      06:56:45 [       OK ] BufferedBlockMgrTest.Close (235 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.DestructDuringWrite
      06:56:45 [       OK ] BufferedBlockMgrTest.DestructDuringWrite (237 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.RuntimeStateTeardown
      06:56:45 [       OK ] BufferedBlockMgrTest.RuntimeStateTeardown (174 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.RuntimeStateTeardownWait
      06:56:45 [       OK ] BufferedBlockMgrTest.RuntimeStateTeardownWait (176 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.RuntimeStateTeardownWriteError
      06:56:45 [       OK ] BufferedBlockMgrTest.RuntimeStateTeardownWriteError (470 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.WriteCompleteWithCancelledRuntimeState
      06:56:45 [       OK ] BufferedBlockMgrTest.WriteCompleteWithCancelledRuntimeState (200 ms)
      06:56:45 [ RUN      ] BufferedBlockMgrTest.WriteError
      06:56:45 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:271: Failure
      06:56:45 Value of: false
      06:56:45   Actual: false
      06:56:45 Expected: true
      06:56:45 Writes did not complete after 500ms
      06:56:45 *** Check failure stack trace: ***
      06:56:45     @          0x291744d  google::LogMessage::Fail()
      06:56:45     @          0x2918cf2  google::LogMessage::SendToLog()
      06:56:45     @          0x2916e27  google::LogMessage::Flush()
      06:56:45     @          0x291a3ee  google::LogMessageFatal::~LogMessageFatal()
      06:56:45     @          0x1326456  impala::TmpFileMgr::WriteHandle::WriteComplete()
      06:56:45     @          0x1325377  impala::TmpFileMgr::FileGroup::WriteComplete()
      06:56:45     @          0x1323fd8  _ZZN6impala10TmpFileMgr9FileGroup5WriteENS_8MemRangeESt8functionIFvRKNS_6StatusEEEPSt10unique_ptrINS0_11WriteHandleESt14default_deleteISA_EEENKUlS6_E_clES6_
      06:56:45     @          0x1326e59  _ZNSt17_Function_handlerIFvRKN6impala6StatusEEZNS0_10TmpFileMgr9FileGroup5WriteENS0_8MemRangeESt8functionIS4_EPSt10unique_ptrINS5_11WriteHandleESt14default_deleteISB_EEEUlS3_E_E9_M_invokeERKSt9_Any_dataS3_
      06:56:45     @          0x1328ca1  std::function<>::operator()()
      06:56:45     @          0x134eac3  impala::DiskIoMgr::HandleWriteFinished()
      06:56:45     @          0x1350940  impala::DiskIoMgr::Write()
      06:56:45     @          0x134f6e3  impala::DiskIoMgr::WorkLoop()
      06:56:45     @          0x135be14  boost::_mfi::mf1<>::operator()()
      06:56:45     @          0x135ba51  boost::_bi::list2<>::operator()<>()
      06:56:45     @          0x135b1a7  boost::_bi::bind_t<>::operator()()
      06:56:45     @          0x135a5fb  boost::detail::function::void_function_obj_invoker0<>::invoke()
      06:56:45     @          0x12c1678  boost::function0<>::operator()()
      06:56:45     @          0x15d13eb  impala::Thread::SuperviseThread()
      06:56:45     @          0x15d9f0e  boost::_bi::list4<>::operator()<>()
      06:56:45     @          0x15d9e51  boost::_bi::bind_t<>::operator()()
      06:56:45     @          0x15d9e14  boost::detail::thread_data<>::run()
      06:56:45     @          0x1b48b2a  thread_proxy
      06:56:45     @       0x3764207851  (unknown)
      06:56:45     @       0x3763ee894d  (unknown)
      06:56:45 Wrote minidump to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/be_tests/minidumps/buffered-block-mgr-test/30c53822-3048-1994-51237fd9-04db9a80.dmp
      06:56:45 Wrote minidump to /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/logs/be_tests/minidumps/buffered-block-mgr-test/30c53822-3048-1994-51237fd9-04db9a80.dmp
      06:56:45 
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        Weird. This has been in there for quite a while and we haven't seen it fail before. Maybe there was a slow disk or some kind of load on the system. Maybe just need to bump the timeout.

        Show
        tarmstrong Tim Armstrong added a comment - Weird. This has been in there for quite a while and we haven't seen it fail before. Maybe there was a slow disk or some kind of load on the system. Maybe just need to bump the timeout.
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5183: increase write wait timeout in BufferedBlockMgrTest

        The timeout was set low enough that some kind of hardware/system
        hiccup had a reasonable chance of causing occasional failures.
        Bumped it to 10s - that should only fail if something is seriously
        wrong.

        Change-Id: Ibbe81eda916730445ca5cf667480ee0cf6b5d429
        Reviewed-on: http://gerrit.cloudera.org:8080/6595
        Reviewed-by: Matthew Jacobs <mj@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5183 : increase write wait timeout in BufferedBlockMgrTest The timeout was set low enough that some kind of hardware/system hiccup had a reasonable chance of causing occasional failures. Bumped it to 10s - that should only fail if something is seriously wrong. Change-Id: Ibbe81eda916730445ca5cf667480ee0cf6b5d429 Reviewed-on: http://gerrit.cloudera.org:8080/6595 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            mjacobs Matthew Jacobs
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development