Details

    • Sub-task
    • Status: Patch Available
    • Major
    • Resolution: Unresolved
    • 3.0.0-alpha1
    • None
    • security

    Description

      The work defined here is to extend the Compression Codec framework to the encryption framework for handling encryption and decryption specific requirements in Hadoop.
      The targets of the encryption framework are:
      1. Establishes a common abstraction of the API level that can be shared by all crypto codec implementations as well as users that use the API.
      2. Provides a foundation for other components in Hadoop such as Map Reduce or HBase to support encryption features.

      The design document is available in the HADOOP-9331.

      Attachments

        1. HADOOP-9333.patch
          93 kB
          Haifeng Chen
        2. HADOOP-9333.patch
          92 kB
          Haifeng Chen

        Issue Links

          Activity

            jerrychenhf Haifeng Chen added a comment -

            Patch attached.

            jerrychenhf Haifeng Chen added a comment - Patch attached.
            jerrychenhf Haifeng Chen added a comment -

            The patched submitted. Please help review. Comments are welcomed.

            jerrychenhf Haifeng Chen added a comment - The patched submitted. Please help review. Comments are welcomed.
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12570715/HADOOP-9333.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified test files.

            -1 one of tests included doesn't have a timeout.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//testReport/
            Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570715/HADOOP-9333.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. -1 one of tests included doesn't have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-common.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2227//console This message is automatically generated.
            jerrychenhf Haifeng Chen added a comment -

            New patch that fixed the findbugs and adding timeout in test.

            jerrychenhf Haifeng Chen added a comment - New patch that fixed the findbugs and adding timeout in test.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified test files.

            +1 tests included appear to have a timeout.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2228//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2228//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2228//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2228//console This message is automatically generated.

            Jerry, I did not look at the patch in detail. At first glance it seems you'll be able to use a single codec, how are you planning to support different compression/encryption codec combinations? Or it is already there?

            tucu00 Alejandro Abdelnur added a comment - Jerry, I did not look at the patch in detail. At first glance it seems you'll be able to use a single codec, how are you planning to support different compression/encryption codec combinations? Or it is already there?
            jerrychenhf Haifeng Chen added a comment -

            tucu00, The crypto codec provide high level abstraction related to a codec that needs a crypto context. When considering the case that we do compression before the encryption, a compression codec can be configured as part of the crypto codec configuration and let the crypto codec implemenation handling the compression by using the specified compression codec and then do encryption after that. And the decryption process is reverse.

            jerrychenhf Haifeng Chen added a comment - tucu00 , The crypto codec provide high level abstraction related to a codec that needs a crypto context. When considering the case that we do compression before the encryption, a compression codec can be configured as part of the crypto codec configuration and let the crypto codec implemenation handling the compression by using the specified compression codec and then do encryption after that. And the decryption process is reverse.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch
            against trunk revision .

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified test files.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 javadoc. There were no new javadoc warning messages.

            +1 eclipse:eclipse. The patch built with eclipse:eclipse.

            +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3700//testReport/
            Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3700//console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3700//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3700//console This message is automatically generated.
            hadoopqa Hadoop QA added a comment -



            -1 overall



            Vote Subsystem Runtime Comment
            0 pre-patch 14m 32s Pre-patch trunk compilation is healthy.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
            +1 javac 7m 32s There were no new javac warning messages.
            +1 javadoc 9m 32s There were no new javadoc warning messages.
            +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
            -1 checkstyle 1m 4s The applied patch generated 196 new checkstyle issues (total was 317, now 511).
            -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
            +1 install 1m 33s mvn install still works.
            +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
            +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
            +1 common tests 22m 46s Tests passed in hadoop-common.
                59m 39s  



            Subsystem Report/Notes
            Patch URL http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch
            Optional Tests javadoc javac unit findbugs checkstyle
            git revision trunk / 6ae2a0d
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/diffcheckstylehadoop-common.txt
            whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/whitespace.txt
            hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/testrun_hadoop-common.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/testReport/
            Java 1.7.0_55
            uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/console

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 14m 32s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 javac 7m 32s There were no new javac warning messages. +1 javadoc 9m 32s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 4s The applied patch generated 196 new checkstyle issues (total was 317, now 511). -1 whitespace 0m 1s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 1m 41s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 common tests 22m 46s Tests passed in hadoop-common.     59m 39s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12570744/HADOOP-9333.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 6ae2a0d checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/diffcheckstylehadoop-common.txt whitespace https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/whitespace.txt hadoop-common test log https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/artifact/patchprocess/testrun_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/6426/console This message was automatically generated.

            People

              Unassigned Unassigned
              jerrychenhf Haifeng Chen
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 168h
                  168h
                  Remaining:
                  Remaining Estimate - 168h
                  168h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified