Uploaded image for project: 'ORC'
  1. ORC
  2. ORC-14

Add column level encryption to ORC files

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      It would be useful to support column level encryption in ORC files. Since each column and its associated index is stored separately, encrypting a column separately isn't difficult. In terms of key distribution, it would make sense to use an external server like the one in HADOOP-9331.

        Attachments

          Issue Links

          1.
          Create framework for data masking. Sub-task Closed Owen O'Malley  
          2.
          Create sha256 mask Sub-task Closed Sandeep More  
          3.
          Modify InStream and OutStream to optionally encrypt data Sub-task Resolved Owen O'Malley  
          4.
          Add support for Key Management Servers (kms) to HadoopShims Sub-task Closed Owen O'Malley  
          5.
          Add unmasked ranges option for redact mask Sub-task Closed Sandeep More  
          6.
          Create in memory KeyProvider class Sub-task Closed Sandeep More  
          7.
          Write documentation for column encryption Sub-task Open Unassigned  
          8.
          Change HadoopShims.KeyProvider to separate createLocalKey and decryptLocalKey Sub-task Closed Owen O'Malley  
          9.
          Separate the compression options from the CompressionCodec Sub-task Resolved Owen O'Malley  
          10.
          Extend specification and protobuf definition with column encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          11.
          Extend physicalwriter for encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          12.
          Add the API changes for getting column encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          13.
          Refactor the TreeWriter and WriterContext APIs so that TreeWriters can deal with encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          14.
          Extend the stripe read planner to understand encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          15.
          Update the KeyProvider to match spec Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          16.
          Cleanup API for StreamOptions and CompressionCodec.Options Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          17.
          Modify InStream for column encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          18.
          Fix file merging for column encryption. Sub-task Open Owen O'Malley  
          19.
          Update metadata tools to print encryption information. Sub-task Open Owen O'Malley  
          20.
          Update ReaderImpl to support column encryption Sub-task Resolved Owen O'Malley

          100%

          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20m
          21.
          Add support for table properties to control column encryption Sub-task Open Owen O'Malley  

            Activity

              People

              • Assignee:
                owen.omalley Owen O'Malley
                Reporter:
                owen.omalley Owen O'Malley
              • Votes:
                3 Vote for this issue
                Watchers:
                31 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h