Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5349

BufferedBlockMgrTest.NoDirsAllocationError failed to write earlier than expected

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-7

      Description

      This is an ASAN build, which may affect timing:

      02:48:15 [ RUN      ] BufferedBlockMgrTest.NoDirsAllocationError
      02:48:15 /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/buffered-block-mgr-test.cc:1239: Failure
      02:48:15 Value of: status_.ok()
      02:48:15   Actual: false
      02:48:15 Expected: true
      02:48:15 Error: Could not create files in any configured scratch directories (--scratch_dirs). See logs for previous errors that may have prevented creating or writing scratch files.
      02:48:15 Opening '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_c0071fba-577c-4c59-95be-a42db2c34721' for write failed with errno=13 description=Error(13): Permission denied
      02:48:15 Opening '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_0d6b4e50-fb7f-44e8-8b59-76a6fd8d56dd' for write failed with errno=13 description=Error(13): Permission denied
      02:48:15 
      02:48:15 *** Check failure stack trace: ***
      02:48:15     @          0x2c3cc26  google::DumpStackTraceAndExit()
      02:48:15     @          0x2c3361d  google::LogMessage::Fail()
      02:48:15     @          0x2c34ec2  google::LogMessage::SendToLog()
      02:48:15     @          0x2c32ff7  google::LogMessage::Flush()
      02:48:15     @          0x2c365be  google::LogMessageFatal::~LogMessageFatal()
      02:48:15     @          0x11a4999  impala::BufferedBlockMgr::~BufferedBlockMgr()
      02:48:15     @          0x11ae5ca  std::_Sp_counted_ptr<>::_M_dispose()
      02:48:15     @          0x10bc8b5  std::_Sp_counted_base<>::_M_release()
      02:48:15     @          0x10b0414  std::__shared_ptr<>::reset()
      02:48:15     @          0x127fcad  impala::RuntimeState::ReleaseResources()
      02:48:15     @          0x1238348  impala::TestEnv::TearDownQueries()
      02:48:15     @          0x10b4f11  impala::BufferedBlockMgrTest::TearDown()
      02:48:15     @          0x2caf923  testing::internal::HandleExceptionsInMethodIfSupported<>()
      02:48:15     @          0x2ca7249  testing::Test::Run()
      02:48:15     @          0x2ca73c8  testing::TestInfo::Run()
      02:48:15     @          0x2ca74a5  testing::TestCase::Run()
      02:48:15     @          0x2ca8728  testing::internal::UnitTestImpl::RunAllTests()
      02:48:15     @          0x2ca8a03  testing::UnitTest::Run()
      02:48:15     @          0x10a8827  main
      02:48:15     @       0x35d0e1ecdd  (unknown)
      02:48:15     @           0xfb8d45  (unknown)
      

        Activity

        Hide
        henryr Henry Robinson added a comment -

        Marking as blocker until we understand how serious this is. Unfortunately the minidumps did not survive long enough for me to gather them.

        Show
        henryr Henry Robinson added a comment - Marking as blocker until we understand how serious this is. Unfortunately the minidumps did not survive long enough for me to gather them.
        Hide
        tarmstrong Tim Armstrong added a comment -

        The assertion in the test is incorrect - asynchronous I/Os are racing with the main test thread and if the I/Os win this happens.

        The fix should be pretty easy but I'll downgrade since it's not a blocker.

        Show
        tarmstrong Tim Armstrong added a comment - The assertion in the test is incorrect - asynchronous I/Os are racing with the main test thread and if the I/Os win this happens. The fix should be pretty easy but I'll downgrade since it's not a blocker.
        Hide
        tarmstrong Tim Armstrong added a comment -

        The same test in BufferPoolTest doesn't have this bug because BufferPool::Unpin() doesn't return a Status

        Show
        tarmstrong Tim Armstrong added a comment - The same test in BufferPoolTest doesn't have this bug because BufferPool::Unpin() doesn't return a Status
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5349: flaky NoDirsAllocationError test

        The assertion was incorrect and racy - it is ok if the write error wins
        the race with the Unpin() calls, causing them to fail.

        Change-Id: I023193b9ad6c6ac0ee114ad77ddf04d7d7185809
        Reviewed-on: http://gerrit.cloudera.org:8080/6953
        Reviewed-by: Henry Robinson <henry@cloudera.com>
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5349 : flaky NoDirsAllocationError test The assertion was incorrect and racy - it is ok if the write error wins the race with the Unpin() calls, causing them to fail. Change-Id: I023193b9ad6c6ac0ee114ad77ddf04d7d7185809 Reviewed-on: http://gerrit.cloudera.org:8080/6953 Reviewed-by: Henry Robinson <henry@cloudera.com> Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            henryr Henry Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development