Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2124

Tablet copy service holds session lock while initializing each session

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.4.0
    • 1.6.0
    • tserver
    • None

    Description

      TabletCopyService holds the service-wide lock while initializing each session. Under heavy load, this can cause starvation, timeouts, and failures. Dan came up with a pstack that had a bunch of threads waiting on the same mutex while the holder was doing I/O:

      #2  0x0000000001a45d72 in kudu::fs::LogBlockManager::OpenBlock(kudu::BlockId const&, std::unique_ptr<kudu::fs::ReadableBlock, std::default_delete<kudu::fs::ReadableBlock> >*) ()
      #3  0x0000000001a33939 in kudu::FsManager::OpenBlock(kudu::BlockId const&, std::unique_ptr<kudu::fs::ReadableBlock, std::default_delete<kudu::fs::ReadableBlock> >*) ()
      #4  0x00000000008e5edf in kudu::tserver::TabletCopySourceSession::OpenBlockUnlocked(kudu::BlockId const&) ()
      #5  0x00000000008e7b46 in kudu::tserver::TabletCopySourceSession::Init() ()
      #6  0x00000000008e1165 in kudu::tserver::TabletCopyServiceImpl::BeginTabletCopySession(kudu::tserver::BeginTabletCopySessionRequestPB const*, kudu::tserver::BeginTabletCopySessionResponsePB*, kudu::rpc::RpcContext*) ()
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            danburkert Dan Burkert
            mpercy Mike Percy
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment