Uploaded image for project: 'Accumulo'
  1. Accumulo
  2. ACCUMULO-998 support encryption at rest
  3. ACCUMULO-981

support pluggable encryption codecs for MapFile when recovering write-ahead logs

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      As with ACCUMULO-980, we need to support encryption for the recovered write-ahead logs that are stored as MapFiles in HDFS. Those MapFiles need to support low-latency seeks, so aligning the encryption with the block compression is a good strategy.

        Issue Links

          Activity

          Hide
          vines John Vines added a comment -

          I know that this is a bit roundabout, but I'm wondering if it would be easier to make the files used for recovery RFiles with the Mutation being transformed to a Key and the columns being written as either one large value or a series of values (using other Key properties to order)

          Show
          vines John Vines added a comment - I know that this is a bit roundabout, but I'm wondering if it would be easier to make the files used for recovery RFiles with the Mutation being transformed to a Key and the columns being written as either one large value or a series of values (using other Key properties to order)
          Hide
          kturner Keith Turner added a comment -

          I'm wondering if it would be easier to make the files used for recovery RFiles with the Mutation being transformed to a Key

          Sorted logs have more than mutations. They have tablet definitions, start/stop compaction events, and mutations for multiple tablets. So its not a 1:1 mapping to RFile.

          Show
          kturner Keith Turner added a comment - I'm wondering if it would be easier to make the files used for recovery RFiles with the Mutation being transformed to a Key Sorted logs have more than mutations. They have tablet definitions, start/stop compaction events, and mutations for multiple tablets. So its not a 1:1 mapping to RFile.
          Hide
          vines John Vines added a comment -

          Drat, worth a shot

          Show
          vines John Vines added a comment - Drat, worth a shot
          Hide
          kturner Keith Turner added a comment -

          Drat, worth a shot

          Its something to think about. The fact that is not a 1:1 mapping, just means it not dead simple. Your comments here and Eric's comments on 1085 hint that we may need to rethink the what the output of log sort is inorder to solve a few issues for 1.6.

          Show
          kturner Keith Turner added a comment - Drat, worth a shot Its something to think about. The fact that is not a 1:1 mapping, just means it not dead simple. Your comments here and Eric's comments on 1085 hint that we may need to rethink the what the output of log sort is inorder to solve a few issues for 1.6.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ctubbsii commented on the pull request:

          https://github.com/apache/accumulo/pull/28#issuecomment-94030167

          @joshelser Regarding what to do about the crypto package... that stuff is still marked as experimental, and some of it is still incomplete for full on-disk encryption (ACCUMULO-981), so whatever we do, I think for now, it should not be moved into the public API, because it may require unstable API changes to finish that work.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ctubbsii commented on the pull request: https://github.com/apache/accumulo/pull/28#issuecomment-94030167 @joshelser Regarding what to do about the crypto package... that stuff is still marked as experimental, and some of it is still incomplete for full on-disk encryption ( ACCUMULO-981 ), so whatever we do, I think for now, it should not be moved into the public API, because it may require unstable API changes to finish that work.

            People

            • Assignee:
              vines John Vines
              Reporter:
              afuchs Adam Fuchs
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development