Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-14

Add column level encryption to ORC files

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 1.6.0
    • None
    • None

    Description

      It would be useful to support column level encryption in ORC files. Since each column and its associated index is stored separately, encrypting a column separately isn't difficult. In terms of key distribution, it would make sense to use an external server like the one in HADOOP-9331.

      Attachments

        Issue Links

        1.
        Create framework for data masking. Sub-task Closed Owen O'Malley   Actions
        2.
        Create sha256 mask Sub-task Closed Sandeep More   Actions
        3.
        Modify InStream and OutStream to optionally encrypt data Sub-task Closed Owen O'Malley   Actions
        4.
        Add support for Key Management Servers (kms) to HadoopShims Sub-task Closed Owen O'Malley   Actions
        5.
        Add unmasked ranges option for redact mask Sub-task Closed Sandeep More   Actions
        6.
        Create in memory KeyProvider class Sub-task Closed Sandeep More   Actions
        7.
        Write documentation for column encryption Sub-task Open Unassigned   Actions
        8.
        Change HadoopShims.KeyProvider to separate createLocalKey and decryptLocalKey Sub-task Closed Owen O'Malley   Actions
        9.
        Separate the compression options from the CompressionCodec Sub-task Closed Owen O'Malley   Actions
        10.
        Extend specification and protobuf definition with column encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        11.
        Extend physicalwriter for encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        12.
        Add the API changes for getting column encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        13.
        Refactor the TreeWriter and WriterContext APIs so that TreeWriters can deal with encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        14.
        Extend the stripe read planner to understand encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        15.
        Update the KeyProvider to match spec Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        16.
        Cleanup API for StreamOptions and CompressionCodec.Options Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        17.
        Modify InStream for column encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        18.
        Fix file merging for column encryption. Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        19.
        Update metadata tools to print encryption information. Sub-task Open Owen O'Malley   Actions
        20.
        Update ReaderImpl to support column encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions
        21.
        Add support for table properties to control column encryption Sub-task Closed Owen O'Malley

        100%

        Original Estimate - Not Specified Original Estimate - Not Specified
        Time Spent - 20m
        Actions

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            omalley Owen O'Malley
            omalley Owen O'Malley
            Votes:
            4 Vote for this issue
            Watchers:
            34 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 3h 40m
                3h 40m

                Slack

                  Issue deployment