Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.1
    • Component/s: security
    • Labels:
      None

      Description

      This is to track changes for restoring security in 0.22 branch.

      1. performance_22_vs_22sec_vs_22secon.pdf
        37 kB
        Benoy Antony
      2. performance_22_vs_22sec.pdf
        33 kB
        Benoy Antony
      3. SecurityTestPlan_results.pdf
        52 kB
        Benoy Antony
      4. test_patch_results
        13 kB
        Benoy Antony

        Issue Links

          Activity

          Konstantin Shvachko made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Fix Version/s 0.22.1 [ 12319240 ]
          Resolution Fixed [ 1 ]
          Hide
          Konstantin Shvachko added a comment -

          Yes. This was completed. Resolving.

          Show
          Konstantin Shvachko added a comment - Yes. This was completed. Resolving.
          Hide
          Harsh J added a comment -

          Hi Benoy/Konstantin,

          Do we still need this open?

          Show
          Harsh J added a comment - Hi Benoy/Konstantin, Do we still need this open?
          Gavin made changes -
          Link This issue depends upon MAPREDUCE-4240 [ MAPREDUCE-4240 ]
          Gavin made changes -
          Link This issue depends on MAPREDUCE-4240 [ MAPREDUCE-4240 ]
          Hide
          Konstantin Shvachko added a comment -

          Created a quick followup jira MAPREDUCE-4318.

          Show
          Konstantin Shvachko added a comment - Created a quick followup jira MAPREDUCE-4318 .
          Konstantin Shvachko made changes -
          Link This issue depends upon MAPREDUCE-4318 [ MAPREDUCE-4318 ]
          Konstantin Shvachko made changes -
          Link This issue depends upon MAPREDUCE-4314 [ MAPREDUCE-4314 ]
          Hide
          Benoy Antony added a comment -
          Show
          Benoy Antony added a comment - Tested Distcp with and without hftp. It works fine. Also tested the same on a cluster with security turned off. Made a trivial change in HftpFileSystem in the patch for HDFS-1860 . The order of patches to be applied are as below : MAPREDUCE-4240 HDFS-3402 MAPREDUCE-4243 HADOOP-7338 MAPREDUCE-2178 MAPREDUCE-4244 HDFS-3403 HADOOP-7119 HADOOP-8381 HADOOP-7674 HADOOP-7645 HADOOP-7621 HDFS-1860 MAPREDUCE-2420 MAPREDUCE-2452 HADOOP-7272 HDFS-1584 HADOOP-7215 MAPREDUCE-2651 HADOOP-7115 MAPREDUCE-2353 MAPREDUCE-2377 MAPREDUCE-2103 MAPREDUCE-2376 MAPREDUCE-2224 MAPREDUCE-4246 HADOOP-8383 MAPREDUCE-4247 MAPREDUCE-4248 MAPREDUCE-4249
          Hide
          Konstantin Boudnik added a comment -

          This is indeed looks good and complete enough. I see here quite a bit of test scenarios we did for first Y! security release.
          +1 on the changes. Let's commit it.

          Show
          Konstantin Boudnik added a comment - This is indeed looks good and complete enough. I see here quite a bit of test scenarios we did for first Y! security release. +1 on the changes. Let's commit it.
          Hide
          Konstantin Shvachko added a comment -

          Sounds like a pretty comprehensive test plan to me.
          I see that impersonation tests cover the Oozie case scenarios. Did you test DistCp with hftp and without? I believe the WebUI cases should cover that, but worth asking. Please comment if you did try it.

          The benchmarks look really good. I remember seeing similar numbers when security was first tested in then 0.20 branch.

          Good job fixing findbugs. I agree the remaining few are just the specific use cases.

          I am +1 on the changes overall and will be glad to start committing soon if there are no objections.

          Show
          Konstantin Shvachko added a comment - Sounds like a pretty comprehensive test plan to me. I see that impersonation tests cover the Oozie case scenarios. Did you test DistCp with hftp and without? I believe the WebUI cases should cover that, but worth asking. Please comment if you did try it. The benchmarks look really good. I remember seeing similar numbers when security was first tested in then 0.20 branch. Good job fixing findbugs. I agree the remaining few are just the specific use cases. I am +1 on the changes overall and will be glad to start committing soon if there are no objections.
          Benoy Antony made changes -
          Attachment SecurityTestPlan_results.pdf [ 12528826 ]
          Attachment performance_22_vs_22sec.pdf [ 12528827 ]
          Attachment performance_22_vs_22sec_vs_22secon.pdf [ 12528828 ]
          Attachment test_patch_results [ 12528829 ]
          Hide
          Benoy Antony added a comment -

          Test-patch results:
          There were issues in all the 3 projects. I fixed some of the issues and uploaded them to respective jiras .
          But some of the issues cannot be resolved. I am attaching the results and explanations for the -1s.

          test Plan :
          The executed test plan is attached.

          Performance Results:
          Performance comparison results between 22, 22 with security patches (security turned off and on) are attached.

          Show
          Benoy Antony added a comment - Test-patch results: There were issues in all the 3 projects. I fixed some of the issues and uploaded them to respective jiras . But some of the issues cannot be resolved. I am attaching the results and explanations for the -1s. test Plan : The executed test plan is attached. Performance Results: Performance comparison results between 22, 22 with security patches (security turned off and on) are attached.
          Hide
          Konstantin Shvachko added a comment -

          Ran all unit tests for 3 projects. All passed, except for one TestFileArgs. MAPREDUCE-4249 fixed it.
          Things look good now.

          So I propose that

          1. Let's fix findbugs if there are any related to these patches.
          2. Attach the test plan doc describing what testing was executed.
          3. Make two types of benchmarks
            1. Current 0.22.1 vs the patched non-secure version.
            2. Compare secure and non-secure versions with the patch applied.

          Also it would be good to have somebody with security background to look into it.
          Volunteers are very much welcome.

          Show
          Konstantin Shvachko added a comment - Ran all unit tests for 3 projects. All passed, except for one TestFileArgs. MAPREDUCE-4249 fixed it. Things look good now. So I propose that Let's fix findbugs if there are any related to these patches. Attach the test plan doc describing what testing was executed. Make two types of benchmarks Current 0.22.1 vs the patched non-secure version. Compare secure and non-secure versions with the patch applied. Also it would be good to have somebody with security background to look into it. Volunteers are very much welcome.
          Hide
          Benoy Antony added a comment -

          Adding more patch for the streaming test failure - MAPREDUCE-4249

          Show
          Benoy Antony added a comment - Adding more patch for the streaming test failure - MAPREDUCE-4249
          Benoy Antony made changes -
          Link This issue is depended upon by MAPREDUCE-4249 [ MAPREDUCE-4249 ]
          Show
          Benoy Antony added a comment - I have moved the sub tasks to the right projects and then linked them . The correct order to apply the patches is as follows: The order to apply the patches is as follows: MAPREDUCE-4240 HDFS-3402 MAPREDUCE-4243 HADOOP-7338 MAPREDUCE-2178 MAPREDUCE-4244 HDFS-3403 HADOOP-7119 HADOOP-8381 HADOOP-7674 HADOOP-7645 HADOOP-7621 MAPREDUCE-2420 MAPREDUCE-2452 HADOOP-7272 HDFS-1584 HADOOP-7215 MAPREDUCE-2651 HADOOP-7115 MAPREDUCE-2353 MAPREDUCE-2377 MAPREDUCE-2103 MAPREDUCE-2376 MAPREDUCE-2224 MAPREDUCE-4246 HADOOP-8383 MAPREDUCE-4247 MAPREDUCE-4248
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-4248 [ MAPREDUCE-4248 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-4247 [ MAPREDUCE-4247 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-4246 [ MAPREDUCE-4246 ]
          Benoy Antony made changes -
          Link This issue depends upon HDFS-3403 [ HDFS-3403 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-4244 [ MAPREDUCE-4244 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-4243 [ MAPREDUCE-4243 ]
          Benoy Antony made changes -
          Link This issue depends upon HDFS-3402 [ HDFS-3402 ]
          Hide
          Konstantin Shvachko added a comment -

          Some sub-tasks need to be moved to the right projects.

          Show
          Konstantin Shvachko added a comment - Some sub-tasks need to be moved to the right projects.
          Konstantin Shvachko made changes -
          Link This issue depends on MAPREDUCE-4240 [ MAPREDUCE-4240 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-1584 [ HADOOP-1584 ]
          Benoy Antony made changes -
          Link This issue depends upon HDFS-1584 [ HDFS-1584 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2353 [ MAPREDUCE-2353 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2377 [ MAPREDUCE-2377 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2224 [ MAPREDUCE-2224 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2376 [ MAPREDUCE-2376 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2103 [ MAPREDUCE-2103 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2651 [ MAPREDUCE-2651 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-1584 [ HADOOP-1584 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2452 [ MAPREDUCE-2452 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7115 [ HADOOP-7115 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7215 [ HADOOP-7215 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7272 [ HADOOP-7272 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2420 [ MAPREDUCE-2420 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7621 [ HADOOP-7621 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7645 [ HADOOP-7645 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7674 [ HADOOP-7674 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7119 [ HADOOP-7119 ]
          Benoy Antony made changes -
          Link This issue depends upon HADOOP-7338 [ HADOOP-7338 ]
          Benoy Antony made changes -
          Link This issue depends upon MAPREDUCE-2178 [ MAPREDUCE-2178 ]
          Hide
          Konstantin Shvachko added a comment -

          Sounds like a plan, Benoy.
          It will be good to have a test plan laid out here. I think a small cluster testing is sufficient, since we are restoring / porting the feature, not introducing.
          Also I don't think we should create a separate branch at this point. Testing is going on on internal branch and I will be able to commit after satisfactory results are reported.

          Show
          Konstantin Shvachko added a comment - Sounds like a plan, Benoy. It will be good to have a test plan laid out here. I think a small cluster testing is sufficient, since we are restoring / porting the feature, not introducing. Also I don't think we should create a separate branch at this point. Testing is going on on internal branch and I will be able to commit after satisfactory results are reported.
          Benoy Antony made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Benoy Antony made changes -
          Assignee Benoy Antony [ benoyantony ]
          Hide
          Benoy Antony added a comment -

          I have been  working on incorporating security into Hadoop 0.22. I am assembling changes in my internal branch.The work is close to completion in terms of development and testing. 

          I started by reverting  MAPREDUCE-2767.
          The main jiras that I have incorporated are 

          MAPREDUCE-2178 - Race condition in LinuxTaskController permissions handling
          HADOOP-7119 add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles
           
          I have incorporated the following patches from 1.0 addressing bugs and vulnerabilities
           
          MAPREDUCE-2420. JobTracker should be able to renew delegation token over HTTP
          MAPREDUCE-2452 - Delegation token cancellation shouldn't hold global JobTracker lock
          HADOOP  7272  Remove unnecessary security related info logs
          HDFS-1584 - Adds a check for whether relogin is needed to getDelegationToken in HftpFileSystem(Kan Zhang)
          HADOOP-7215. RPC clients must use network interface corresponding to the host in the client's kerberos principal key
          MAPREDUCE-2651 Race condition in Linux Task Controller for job log directory creation
          HADOOP-7115. Reduces the number of calls to getpwuid_r and getpwgid_r, by implementing a cache in NativeIO. (ddas)
          MAPREDUCE-2377. task-controller fails to parse configuration if it doesn't end in \n
          MAPREDUCE-2103 task-controller shouldn't require o-r permissions
          MAPREDUCE-2376 test-task-controller fails if run as a userid < 1000
          Fixing an issue to do with setting of correct groups for tasks
           
          I  fixed some issues found during testing for which I will file new jiras. They include
           
          1) Task Termination by TT when fatal error occurs
          2) Failure in deleting user directories when security is enabled
          3) some build changes are needed
           
          I have been testing the secure hadoop on cluster of 6 nodes. I have also tested with security disabled to make sure that unsecured version still works.
           
          Once I am done with remaining work, I can start adding patches to the above jiras as well as create jiras for additional issues. 

          Show
          Benoy Antony added a comment - I have been  working on incorporating security into Hadoop 0.22. I am assembling changes in my internal branch.The work is close to completion in terms of development and testing.  I started by reverting   MAPREDUCE-2767 . The main jiras that I have incorporated are  MAPREDUCE-2178 - Race condition in LinuxTaskController permissions handling HADOOP-7119  add Kerberos HTTP SPNEGO authentication support to Hadoop JT/NN/DN/TT web-consoles   I have incorporated the following patches from 1.0 addressing bugs and vulnerabilities   MAPREDUCE-2420 . JobTracker should be able to renew delegation token over HTTP MAPREDUCE-2452 - Delegation token cancellation shouldn't hold global JobTracker lock HADOOP   7272   Remove unnecessary security related info logs HDFS-1584  - Adds a check for whether relogin is needed to getDelegationToken in HftpFileSystem(Kan Zhang) HADOOP-7215 . RPC clients must use network interface corresponding to the host in the client's kerberos principal key MAPREDUCE-2651  Race condition in Linux Task Controller for job log directory creation HADOOP-7115 . Reduces the number of calls to getpwuid_r and getpwgid_r, by implementing a cache in NativeIO. (ddas) MAPREDUCE-2377 . task-controller fails to parse configuration if it doesn't end in \n MAPREDUCE-2103  task-controller shouldn't require o-r permissions MAPREDUCE-2376  test-task-controller fails if run as a userid < 1000 Fixing an issue to do with setting of correct groups for tasks   I  fixed some issues found during testing for which I will file new jiras. They include   1) Task Termination by TT when fatal error occurs 2) Failure in deleting user directories when security is enabled 3) some build changes are needed   I have been testing the secure hadoop on cluster of 6 nodes. I have also tested with security disabled to make sure that unsecured version still works.   Once I am done with remaining work, I can start adding patches to the above jiras as well as create jiras for additional issues. 
          Konstantin Shvachko made changes -
          Link This issue is related to MAPREDUCE-2767 [ MAPREDUCE-2767 ]
          Hide
          Konstantin Shvachko added a comment -

          Please link required jiras and create subtasks here.

          Show
          Konstantin Shvachko added a comment - Please link required jiras and create subtasks here.
          Konstantin Shvachko made changes -
          Link This issue is related to HADOOP-4487 [ HADOOP-4487 ]
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop Common [ 12310240 ]
          Key MAPREDUCE-4222 HADOOP-8357
          Affects Version/s 0.22.0 [ 12314296 ]
          Affects Version/s 0.22.0 [ 12314184 ]
          Target Version/s 0.22.1 [ 12319242 ] 0.22.1 [ 12319240 ]
          Component/s security [ 12312526 ]
          Component/s security [ 12313041 ]
          Konstantin Shvachko created issue -

            People

            • Assignee:
              Benoy Antony
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development