Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: security
    • Labels:
      None

      Description

      HADOOP-8225 broke MR when security is off. MR was changed to stop re-reading the credentials that UGI had already read, and to stop putting those tokens back into the UGI where they already were. UGI only reads a credentials file when security is enabled, but MR uses tokens (ie. job token) even when security is disabled...

      1. HADOOP-8725.patch
        0.9 kB
        Daryn Sharp

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Affects Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1176/)
          HADOOP-8725. MR is broken when security is off (daryn via bobby) (Revision 1376929)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1176/ ) HADOOP-8725 . MR is broken when security is off (daryn via bobby) (Revision 1376929) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1145/)
          HADOOP-8725. MR is broken when security is off (daryn via bobby) (Revision 1376929)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1145 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1145/ ) HADOOP-8725 . MR is broken when security is off (daryn via bobby) (Revision 1376929) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #354 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/354/)
          svn merge -c 1376929 FIXES: HADOOP-8725. MR is broken when security is off (daryn via bobby) (Revision 1376933)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376933
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #354 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/354/ ) svn merge -c 1376929 FIXES: HADOOP-8725 . MR is broken when security is off (daryn via bobby) (Revision 1376933) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376933 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Hide
          Alejandro Abdelnur added a comment -

          @Jason and Bobby, I don't have any Hadoop ENV set in my environment, but whatever it was it is not happenining anymore on trunk. thx

          Show
          Alejandro Abdelnur added a comment - @Jason and Bobby, I don't have any Hadoop ENV set in my environment, but whatever it was it is not happenining anymore on trunk. thx
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2697 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2697/)
          HADOOP-8725. MR is broken when security is off (daryn via bobby) (Revision 1376929)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2697 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2697/ ) HADOOP-8725 . MR is broken when security is off (daryn via bobby) (Revision 1376929) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2633 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2633/)
          HADOOP-8725. MR is broken when security is off (daryn via bobby) (Revision 1376929)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2633 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2633/ ) HADOOP-8725 . MR is broken when security is off (daryn via bobby) (Revision 1376929) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376929 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          Robert Joseph Evans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 2.2.0-alpha [ 12322473 ]
          Resolution Fixed [ 1 ]
          Hide
          Robert Joseph Evans added a comment -

          I checked this into trunk, branch-2, branch-2.1.0-alpha, and branch-0.23.

          Thanks Daryn for a quick fix to this.

          Show
          Robert Joseph Evans added a comment - I checked this into trunk, branch-2, branch-2.1.0-alpha, and branch-0.23. Thanks Daryn for a quick fix to this.
          Hide
          Robert Joseph Evans added a comment -

          Because non-secure mode is completely broken currently, and this patch does seem to fix the issues. Also because Jason and I both seem to have validated that the MR tests pass with this fix, I am going to check this in, simply because it seems better then the current state of trunk. If we find other errors we can fix them on a different JIRA.

          Show
          Robert Joseph Evans added a comment - Because non-secure mode is completely broken currently, and this patch does seem to fix the issues. Also because Jason and I both seem to have validated that the MR tests pass with this fix, I am going to check this in, simply because it seems better then the current state of trunk. If we find other errors we can fix them on a different JIRA.
          Hide
          Jason Lowe added a comment -

          I was able to reproduce the errors Alejandro is seeing in TestMRJobs twice in a row. However when I emptied my environment of Hadoop variables (e.g.: HADOOP_PREFIX, HADOOP_CONF_DIR, etc.) then the test passed twice in a row. So apparently that test is sensitive to its environment.

          Show
          Jason Lowe added a comment - I was able to reproduce the errors Alejandro is seeing in TestMRJobs twice in a row. However when I emptied my environment of Hadoop variables (e.g.: HADOOP_PREFIX, HADOOP_CONF_DIR, etc.) then the test passed twice in a row. So apparently that test is sensitive to its environment.
          Hide
          Robert Joseph Evans added a comment -

          I did run all of the mr tests. The only one that failed is TestProcfsBasedProcessTree, which seems to be a race condition. I ran several jobs on a single node cluster with this patch and they all passed. so I am still a +1, but I would like others to chime in on if their tests pass.

          Show
          Robert Joseph Evans added a comment - I did run all of the mr tests. The only one that failed is TestProcfsBasedProcessTree, which seems to be a race condition. I ran several jobs on a single node cluster with this patch and they all passed. so I am still a +1, but I would like others to chime in on if their tests pass.
          Hide
          Alejandro Abdelnur added a comment - - edited

          mmh, jenkings come back while I was writing my prev comment. But I believe jenkins is not testing MR, can somebody verify running mvn clean test -Dtest=TestMRJobs?

          Show
          Alejandro Abdelnur added a comment - - edited mmh, jenkings come back while I was writing my prev comment. But I believe jenkins is not testing MR, can somebody verify running mvn clean test -Dtest=TestMRJobs ?
          Hide
          Alejandro Abdelnur added a comment -

          I've just run TestMRJobs with the patch and I'm still seen errors

          
          -------------------------------------------------------
          Running org.apache.hadoop.mapreduce.v2.TestMRJobs
          Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 119.838 sec <<< FAILURE!
          
          Results :
          
          Failed tests:   testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs)
            testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs)
            testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs)
          

          Unless they are unrelated, the issue is still there.

          Show
          Alejandro Abdelnur added a comment - I've just run TestMRJobs with the patch and I'm still seen errors ------------------------------------------------------- Running org.apache.hadoop.mapreduce.v2.TestMRJobs Tests run: 4, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 119.838 sec <<< FAILURE! Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) Unless they are unrelated, the issue is still there.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542227/HADOOP-8725.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542227/HADOOP-8725.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1355//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          I have manually verified that the changes fix the problem. The change is small and looks good to me +1 assuming Jenkins comes back OK.

          Show
          Robert Joseph Evans added a comment - I have manually verified that the changes fix the problem. The change is small and looks good to me +1 assuming Jenkins comes back OK.
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Daryn Sharp made changes -
          Attachment HADOOP-8725.patch [ 12542227 ]
          Hide
          Daryn Sharp added a comment -

          I'm not sure how to easily write a test since there's not a supported way to set envs in java. Well, that's non-extremely-dirty-and-hacky by using reflection to bypass the immutable map.

          Show
          Daryn Sharp added a comment - I'm not sure how to easily write a test since there's not a supported way to set envs in java. Well, that's non-extremely-dirty-and-hacky by using reflection to bypass the immutable map.
          Daryn Sharp made changes -
          Field Original Value New Value
          Link This issue is broken by HADOOP-8225 [ HADOOP-8225 ]
          Daryn Sharp created issue -

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development