Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4820

TmpFileMgr can write unencrypted data to disk even when encryption is on

    Details

      Description

      When writing encrypted data to scratch disk, the cancellation process can decrypt the data while the write is still in flight. This could result in some amount of decrypted data being written to disk, defeating the purpose of encryption:

        // Decrypt regardless of whether the write is still in flight or not. An in-flight
        // write may write bogus data to disk but this lets us get some work done while the
        // write is being cancelled.
        Status status;
        if (FLAGS_disk_spill_encryption) {
          status = handle->CheckHashAndDecrypt(buffer);
        }
        handle->WaitForWrite();
      

      This was introduced by commit "IMPALA-3202,IMPALA-2079: rework scratch file I/O" and never made it into any official releases.

        Activity

        Hide
        jbapple Jim Apple added a comment -
        Show
        jbapple Jim Apple added a comment - Patch available: https://gerrit.cloudera.org/#/c/5788/
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-4820: avoid writing unencrypted data during write cancellation

        The bug was that unencrypted data could be written to disk if
        the write was cancelled before it completed. This bug was introduced
        after Impala 2.8.0 was branched in the commit "IMPALA-3202,IMPALA-2079:
        rework scratch file I/O", so does not appear in any released versions
        of Impala.

        The fix is to only start decrypting data after the write is
        complete.

        Testing:
        Added a regression test that reproduced the problem (after adding a
        delay to the write).

        Change-Id: I956bae0685e863f30be23634b29aa076394db184
        Reviewed-on: http://gerrit.cloudera.org:8080/5788
        Tested-by: Impala Public Jenkins
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-4820 : avoid writing unencrypted data during write cancellation The bug was that unencrypted data could be written to disk if the write was cancelled before it completed. This bug was introduced after Impala 2.8.0 was branched in the commit " IMPALA-3202 , IMPALA-2079 : rework scratch file I/O", so does not appear in any released versions of Impala. The fix is to only start decrypting data after the write is complete. Testing: Added a regression test that reproduced the problem (after adding a delay to the write). Change-Id: I956bae0685e863f30be23634b29aa076394db184 Reviewed-on: http://gerrit.cloudera.org:8080/5788 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development