Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4437

Occasional crash in buffered-block-mgr-test

    Details

      Description

      I saw a failure in the exhaustive tests, but it is in a be test that is run as part of core:

      Log file created at: 2016/11/06 04:08:04
      Running on machine: impala-boost-static-burst-slave-0ce0.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      E1106 04:08:04.023739  1412 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_4fb18f92-39bb-4f80-ae7c-dc68b5d420c1': fopen(/tmp/impala-scratch/0:0_4fb18f92-39bb-4f80-ae7c-dc68b5d420c1, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1106 04:08:04.024101  1412 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_4fb18f92-39bb-4f80-ae7c-dc68b5d420c1': fopen(/tmp/impala-scratch/0:0_4fb18f92-39bb-4f80-ae7c-dc68b5d420c1, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1106 04:08:04.348903  1678 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_4811bd63-e6b5-459f-ac40-305f965f29b0': fopen(/tmp/impala-scratch/0:0_4811bd63-e6b5-459f-ac40-305f965f29b0, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1106 04:08:04.349059  1676 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_4811bd63-e6b5-459f-ac40-305f965f29b0': fopen(/tmp/impala-scratch/0:0_4811bd63-e6b5-459f-ac40-305f965f29b0, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1106 04:08:04.503168 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_9a608692-60cf-44df-990e-8477126bb594': Truncate file /tmp/impala-scratch/0:0_9a608692-60cf-44df-990e-8477126bb594 to length 2048 failed with errno 2 (Error(2): No such file or directory)
      E1106 04:08:04.641846 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_fe06514e-4f35-43f0-be84-8fa7550dea0a': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_fe06514e-4f35-43f0-be84-8fa7550dea0a failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.646608 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_796f7476-16d2-4061-9483-aeb05efb0e54': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_796f7476-16d2-4061-9483-aeb05efb0e54 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.646981 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_796f7476-16d2-4061-9483-aeb05efb0e54': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_796f7476-16d2-4061-9483-aeb05efb0e54 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.784018 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_a6243ae5-bd4f-4c98-9f89-3ef7e3069f24': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_a6243ae5-bd4f-4c98-9f89-3ef7e3069f24 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.790413 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_c98a86be-20e3-4dbf-8090-20c4c3a8bd95': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_c98a86be-20e3-4dbf-8090-20c4c3a8bd95 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.791249 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_a6243ae5-bd4f-4c98-9f89-3ef7e3069f24': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_a6243ae5-bd4f-4c98-9f89-3ef7e3069f24 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:04.791865 32684 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_c98a86be-20e3-4dbf-8090-20c4c3a8bd95': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_c98a86be-20e3-4dbf-8090-20c4c3a8bd95 failed with errno=13 description=Error(13): Permission denied
      E1106 04:08:05.514145 32684 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:05.634829 32684 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:05.802711 32684 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:05.998805  3014 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:05.998865  3015 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:06.516338  3016 buffered-block-mgr-test.cc:410]  Iteration 0
      E1106 04:08:06.516345  3017 buffered-block-mgr-test.cc:410]  Iteration 0
      F1106 04:08:07.203595  2971 disk-io-mgr.cc:782] Check failed: !desc->is_client_buffer() 
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4437: hit DCHECK in buffered-block-mgr-test

        The issue was introduced by "IMPALA-3202: DiskIoMgr improvements
        for new buffer pool". The bug is pretty straightforward - I forgot
        to update some error/cancellation paths to handle client-provided
        buffers.

        Testing:
        Was able to reproduce locally before the fix, have run it in a loop for
        a while with the fix and not seen any crashes.

        Change-Id: Ib7e8ffdbec341f6ffbc310fb29df63ab45fed545
        Reviewed-on: http://gerrit.cloudera.org:8080/4973
        Reviewed-by: Henry Robinson <henry@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4437 : hit DCHECK in buffered-block-mgr-test The issue was introduced by " IMPALA-3202 : DiskIoMgr improvements for new buffer pool". The bug is pretty straightforward - I forgot to update some error/cancellation paths to handle client-provided buffers. Testing: Was able to reproduce locally before the fix, have run it in a loop for a while with the fix and not seen any crashes. Change-Id: Ib7e8ffdbec341f6ffbc310fb29df63ab45fed545 Reviewed-on: http://gerrit.cloudera.org:8080/4973 Reviewed-by: Henry Robinson <henry@cloudera.com> Tested-by: Internal Jenkins —
        Hide
        tarmstrong Tim Armstrong added a comment -

        I did some further testing and hit a different DCHECK:

        void DiskIoMgr::ReturnBuffer(BufferDescriptor* buffer_desc) {
          DCHECK(buffer_desc != NULL);
          if (!buffer_desc->status_.ok()) DCHECK(buffer_desc->buffer_ == NULL); <=== here
        
        

        Reopening since it's closely related.

        Show
        tarmstrong Tim Armstrong added a comment - I did some further testing and hit a different DCHECK: void DiskIoMgr::ReturnBuffer(BufferDescriptor* buffer_desc) { DCHECK(buffer_desc != NULL); if (!buffer_desc->status_.ok()) DCHECK(buffer_desc->buffer_ == NULL); <=== here Reopening since it's closely related.
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4437: fix crash in disk-io-mgr

        This fixes another issue where the 'buffer_' field was not set to NULL
        on an error, triggering a DCHECK.

        Testing:
        Added a unit test that triggers the bug on the two different codepaths
        that I fixed.

        Change-Id: Ib76cf5ba8d368b2b37bdc1d2133b8ddcb39f9e00
        Reviewed-on: http://gerrit.cloudera.org:8080/4979
        Reviewed-by: Dan Hecht <dhecht@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4437 : fix crash in disk-io-mgr This fixes another issue where the 'buffer_' field was not set to NULL on an error, triggering a DCHECK. Testing: Added a unit test that triggers the bug on the two different codepaths that I fixed. Change-Id: Ib76cf5ba8d368b2b37bdc1d2133b8ddcb39f9e00 Reviewed-on: http://gerrit.cloudera.org:8080/4979 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development