Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1895

DCHECK(non_local_outstanding_writes > 0) in FindBuffer

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Product Backlog
    • Fix Version/s: Impala 2.2
    • Component/s: None
    • Labels:
      None

      Description

      While testing the fix for IMPALA-1890 (repeatedly running test_mem_usage_scaling) with only one impalad I got the following dcheck:

      F0317 16:09:17.021257 31358 buffered-block-mgr.cc:954] Check failed: non_local_outstanding_writes_ > 0 (0 vs. 0) 
      Buffered block mgr
        Num writes outstanding: 0
        Num free io buffers: 0
        Num unpinned blocks: 0
        Num available buffers: -213
        Total pinned buffers: 346
        Unfullfilled reserved buffers: 213
        Remaining memory: 7887616 (#blocks=0)
        Block write threshold: 2
      

      It seems that the problem is that we are checking only before we enter the while loop of waiting for free buffers whether there are any unpinned blocks and outstanding writes.

          if (unpinned_blocks_.empty() && non_local_outstanding_writes_ == 0) return Status::OK; 
      <=== It checks if there is any reason to wait for free buffers.
      
          // Third, this block needs to use a buffer that was unpinned from another block.
          // Get a free buffer from the front of the queue and assign it to the block.
          do {
            SCOPED_TIMER(buffer_wait_timer_);
            // Try to evict unpinned blocks before waiting.
            RETURN_IF_ERROR(WriteUnpinnedBlocks());
            DCHECK_GT(non_local_outstanding_writes_, 0) << endl << DebugInternal();
            buffer_available_cv_.wait(lock);
      ===> At this point we re-acquired the lock and need to check again if there are
      any outstanding writes, somebody else could have sneaked in and took any free
      buffer. <====
            if (is_cancelled_) return Status::CANCELLED;
          } while (free_io_buffers_.empty());
      

        Attachments

          Activity

            People

            • Assignee:
              ippokratis Ippokratis Pandis
              Reporter:
              ippokratis Ippokratis Pandis
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: