Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9737

DCHECK in buffer-pool.cc - min_bytes_to_write <= dirty_unpinned_pages_.bytes()

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • Impala 4.0.0
    • Backend

    Description

      Saw this recently in a dockerised pre-commit tests against a seemingly unrelated change: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/10499/#showFailuresLink (triggered by https://gerrit.cloudera.org/#/c/14666/)

      The error message from the logs is:

      Error Message
      DCHECK found in log file: /home/ubuntu/Impala/logs/ee_tests/impalad_node1.FATAL
      Standard Error
      Log file created at: 2020/05/07 18:07:14
      Running on machine: ip-172-31-3-33
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      F0507 18:07:14.606797 88747 buffer-pool.cc:711] 3f4ad52d42fef180:55b3458000000004] Check failed: min_bytes_to_write <= dirty_unpinned_pages_.bytes() (262144 vs. 0) <BufferPool::Client> 0xedf5d680 name: HASH_JOIN_NODE id=2 ptr=0x25b0e400 write_status:  buffers allocated 262144 num_pages: 0 pinned_bytes: 0 dirty_unpinned_bytes: 0 in_flight_write_bytes: 0 reservation: {<ReservationTracker>: reservation_limit 9223372036854775807 reservation 524288 used_reservation 262144 child_reservations 0 parent:
      <ReservationTracker>: reservation_limit 9223372036854775807 reservation 524288 used_reservation 0 child_reservations 524288 parent:
      <ReservationTracker>: reservation_limit 175112192 reservation 132120576 used_reservation 0 child_reservations 132120576 parent:
      <ReservationTracker>: reservation_limit 10952163328 reservation 326664192 used_reservation 0 child_reservations 326664192 parent:
      NULL}
        0 pinned pages: 
        0 dirty unpinned pages: 
        0 in flight write pages: 
      

      The minidump stack is:

      Operating system: Linux
                        0.0.0 Linux 4.4.0-1081-aws #91-Ubuntu SMP Tue Apr 16 08:21:03 UTC 2019 x86_64
      CPU: amd64
           family 6 model 79 stepping 1
           16 CPUs
      
      GPU: UNKNOWN
      
      Crash reason:  SIGABRT
      Crash address: 0x3e8000010e6
      Process uptime: not available
      
      Thread 418 (crashed)
       0  libc-2.23.so + 0x35428
          rax = 0x0000000000000000   rdx = 0x0000000000000006
          rcx = 0x00007f948e3aa428   rbx = 0x00000000073e2300
          rsi = 0x0000000000015aab   rdi = 0x00000000000010e6
          rbp = 0x00007f9394558c60   rsp = 0x00007f93945588f8
           r8 = 0x0000000000000000    r9 = 0x0000000000000020
          r10 = 0x0000000000000008   r11 = 0x0000000000000202
          r12 = 0x00000000073e2380   r13 = 0x000000000000039a
          r14 = 0x00000000073e9cc4   r15 = 0x00000000073e2300
          rip = 0x00007f948e3aa428
          Found by: given as instruction pointer in context
       1  libc-2.23.so + 0x3702a
          rbp = 0x00007f9394558c60   rsp = 0x00007f9394558900
          rip = 0x00007f948e3ac02a
          Found by: stack scanning
       2  impalad!google::DumpStackTraceAndExit() + 0x24
          rbp = 0x00007f9394558c60   rsp = 0x00007f9394558a30
          rip = 0x0000000005010014
          Found by: stack scanning
       3  impalad!google::LogMessage::Fail() + 0xd
          rbx = 0x00000000073e2300   rbp = 0x00007f9394558c60
          rsp = 0x00007f9394558ae0   rip = 0x0000000005006a6d
          Found by: call frame info
       4  impalad!google::LogMessage::SendToLog() + 0x2b2
          rbx = 0x00000000073e2300   rbp = 0x00007f9394558c60
          rsp = 0x00007f9394558af0   rip = 0x0000000005008312
          Found by: call frame info
       5  impalad!google::LogMessage::Flush() + 0x157
          rbx = 0x00007f9394558ca0   rbp = 0x00007f948ef675a0
          rsp = 0x00007f9394558c70   r12 = 0x00007f9394558c8f
          r13 = 0x0000000000000001   r14 = 0x00007f9394558db0
          r15 = 0x0000000000000001   rip = 0x0000000005006447
          Found by: call frame info
       6  impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe
          rbx = 0x00007f9394558db0   rbp = 0x00007f9394558f40
          rsp = 0x00007f9394558cf0   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x0000000005009a0e
          Found by: call frame info
       7  impalad!impala::BufferPool::Client::WriteDirtyPagesAsync(long) [buffer-pool.cc : 711 + 0xf]
          rbx = 0x0000000000000000   rbp = 0x00007f9394558f40
          rsp = 0x00007f9394558d10   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x0000000002701732
          Found by: call frame info
       8  impalad!impala::BufferPool::Client::CleanPages(std::unique_lock<std::mutex>*, long, bool) [buffer-pool.cc : 691 + 0x16]
          rbx = 0x0000000000000000   rbp = 0x00007f9394559170
          rsp = 0x00007f9394558f50   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x0000000002701147
          Found by: call frame info
       9  impalad!impala::BufferPool::Client::TransferReservationTo(impala::ReservationTracker*, long, bool*) [buffer-pool.cc : 648 + 0x1e]
          rbx = 0x0000000000000000   rbp = 0x00007f9394559200
          rsp = 0x00007f9394559180   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x0000000002700a21
          Found by: call frame info
      10  impalad!impala::BufferPool::ClientHandle::TransferReservationTo(impala::ReservationTracker*, long, bool*) [buffer-pool.cc : 347 + 0x22]
          rbx = 0x0000000000000000   rbp = 0x00007f9394559240
          rsp = 0x00007f9394559210   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x00000000026fce68
          Found by: call frame info
      11  impalad!impala::BufferPool::ClientHandle::TransferReservationTo(impala::BufferPool::ClientHandle*, long, bool*) [buffer-pool.cc : 353 + 0x33]
          rbx = 0x0000000000000000   rbp = 0x00007f93945592c0
          rsp = 0x00007f9394559250   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x00000000026fcf55
          Found by: call frame info
      12  impalad!impala::PhjBuilder::ReturnReservation(impala::BufferPool::ClientHandle*, long) [partitioned-hash-join-builder.cc : 1155 + 0x35]
          rbx = 0x0000000000000000   rbp = 0x00007f93945593b0
          rsp = 0x00007f93945592d0   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x0000000002860a8f
          Found by: call frame info
      13  impalad!impala::PartitionedHashJoinNode::Close(impala::RuntimeState*) [partitioned-hash-join-node.cc : 305 + 0x54]
          rbx = 0x0000000000080000   rbp = 0x00007f93945593f0
          rsp = 0x00007f93945593c0   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x00000000028726fc
          Found by: call frame info
      14  impalad!impala::ExecNode::Close(impala::RuntimeState*) [exec-node.cc : 314 + 0x37]
          rbx = 0x0000000000000000   rbp = 0x00007f93945594e0
          rsp = 0x00007f9394559400   r12 = 0x0000000000000000
          r13 = 0x0000000000000001   r14 = 0x0000000000000001
          r15 = 0x0000000000000001   rip = 0x000000000277437c
          Found by: call frame info
      

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarmstrong Tim Armstrong
            stakiar Sahil Takiar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment