Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4554

Job Credentials are not transmitted if security is turned off

    Details

    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary .

      These credentials get submitted during job submission and are made available to the task processes.

      In HADOOP 1, these credentials get submitted and routed to task processes even if security was off.
      In HADOOP 2 , these credentials are transmitted only when the security is turned on.

      This should be changed for two reasons:
      1) It is not backward compatible.

      2) Credentials should be passed even if security is turned off .

      1. MR_4554_trunk.patch
        14 kB
        Benoy Antony
      2. MR_4554_trunk.patch
        15 kB
        Benoy Antony
      3. MR_4554_trunk.patch
        15 kB
        Benoy Antony
      4. MR_4554_trunk.patch
        15 kB
        Benoy Antony
      5. MR_4554_trunk.patch
        15 kB
        Benoy Antony
      6. MR_4554_trunk.patch
        16 kB
        Benoy Antony
      7. MR_4554_trunk.patch
        16 kB
        Benoy Antony
      8. MR_4554_trunk.patch
        16 kB
        Benoy Antony
      9. MR_4554_trunk.patch
        16 kB
        Benoy Antony

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/)
          MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/ ) MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/)
          MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/ ) MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #399 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/399/)
          svn merge -c 1395769 FIXES: MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395774)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395774
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #399 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/399/ ) svn merge -c 1395769 FIXES: MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395774) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395774 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2849 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2849/)
          MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2849 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2849/ ) MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2825 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2825/)
          MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2825 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2825/ ) MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2887 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2887/)
          MAPREDUCE-4554. Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2887 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2887/ ) MAPREDUCE-4554 . Job Credentials are not transmitted if security is turned off (Benoy Antony via bobby) (Revision 1395769) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395769 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/CredentialsTestJob.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/TestMRCredentials.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks for the patch Benoy +1,

          I put this into trunk, branch-2, and branch-0.23.

          Show
          Robert Joseph Evans added a comment - Thanks for the patch Benoy +1, I put this into trunk, branch-2, and branch-0.23.
          Hide
          Daryn Sharp added a comment -

          +1 Bobby is OOO for probably the next week. This is a joint +1 since he requested trivial changes.

          Show
          Daryn Sharp added a comment - +1 Bobby is OOO for probably the next week. This is a joint +1 since he requested trivial changes.
          Hide
          Benoy Antony added a comment -

          Attached the patch which addresses Robert's comments in regards to sleep job related names and comments in the test class.
          It also passed all the unit tests and received a +1 from Jenkins.

          Show
          Benoy Antony added a comment - Attached the patch which addresses Robert's comments in regards to sleep job related names and comments in the test class. It also passed all the unit tests and received a +1 from Jenkins.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547532/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2904//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2904//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547532/MR_4554_trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2904//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2904//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          +1 After Bobby is satisfied. Great job!

          Show
          Daryn Sharp added a comment - +1 After Bobby is satisfied. Great job!
          Hide
          Robert Joseph Evans added a comment -

          I am not really sure what happened here with the test failure. The test passes for me when I run it myself. Looking more closely at the failures, two of them seem to be because the recovery could not find the jhist file to recover from. One of them appears to be a bit of a race, but I don't really know for sure. Either way I don't think they are caused by the patch. I think it is more likely that they are caused by two people running the tests at the same time on the same box. Local runner collides that way.

          The patch looks OK to me for the most part. I would like to see the comments and class names in CredentialsTestJob updated, because they still refer to sleep, almost like they were copied and pasted from sleep job. But that is it.

          Show
          Robert Joseph Evans added a comment - I am not really sure what happened here with the test failure. The test passes for me when I run it myself. Looking more closely at the failures, two of them seem to be because the recovery could not find the jhist file to recover from. One of them appears to be a bit of a race, but I don't really know for sure. Either way I don't think they are caused by the patch. I think it is more likely that they are caused by two people running the tests at the same time on the same box. Local runner collides that way. The patch looks OK to me for the most part. I would like to see the comments and class names in CredentialsTestJob updated, because they still refer to sleep, almost like they were copied and pasted from sleep job. But that is it.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547329/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestRecovery

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2899//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2899//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547329/MR_4554_trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestRecovery +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2899//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2899//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12547009/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapred.lib.aggregate.TestAggregates
          org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat
          org.apache.hadoop.mapred.TestTaskCommit
          org.apache.hadoop.mapred.TestFieldSelection
          org.apache.hadoop.mapreduce.TestLocalRunner
          org.apache.hadoop.mapred.TestUserDefinedCounters
          org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
          org.apache.hadoop.mapreduce.TestMROutputFormat
          org.apache.hadoop.mapred.TestLineRecordReader
          org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection
          org.apache.hadoop.mapred.TestLocalMRNotification
          org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
          org.apache.hadoop.mapred.lib.TestChainMapReduce
          org.apache.hadoop.mapreduce.TestMapReduce
          org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge
          org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator
          org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner
          org.apache.hadoop.mapred.lib.TestMultipleOutputs
          org.apache.hadoop.mapred.TestJavaSerialization
          org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
          org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs
          org.apache.hadoop.mapred.TestCollect
          org.apache.hadoop.fs.slive.TestSlive
          org.apache.hadoop.mapred.join.TestDatamerge
          org.apache.hadoop.mapreduce.TestMapCollection
          org.apache.hadoop.fs.TestDFSIO
          org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates
          org.apache.hadoop.mapred.TestMapRed
          org.apache.hadoop.mapred.TestFileOutputFormat
          org.apache.hadoop.mapreduce.TestValueIterReset
          org.apache.hadoop.mapred.TestMapOutputType
          org.apache.hadoop.mapred.TestJobCounters
          org.apache.hadoop.fs.TestFileSystem
          org.apache.hadoop.conf.TestNoDefaultsJobConf
          org.apache.hadoop.mapred.TestReporter
          org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator
          org.apache.hadoop.mapreduce.lib.chain.TestChainErrors
          org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain
          org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs
          org.apache.hadoop.mapred.TestComparators
          org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain
          org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547009/MR_4554_trunk.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapred.lib.aggregate.TestAggregates org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat org.apache.hadoop.mapred.TestTaskCommit org.apache.hadoop.mapred.TestFieldSelection org.apache.hadoop.mapreduce.TestLocalRunner org.apache.hadoop.mapred.TestUserDefinedCounters org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath org.apache.hadoop.mapreduce.TestMROutputFormat org.apache.hadoop.mapred.TestLineRecordReader org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection org.apache.hadoop.mapred.TestLocalMRNotification org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper org.apache.hadoop.mapred.lib.TestChainMapReduce org.apache.hadoop.mapreduce.TestMapReduce org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner org.apache.hadoop.mapred.lib.TestMultipleOutputs org.apache.hadoop.mapred.TestJavaSerialization org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs org.apache.hadoop.mapred.TestCollect org.apache.hadoop.fs.slive.TestSlive org.apache.hadoop.mapred.join.TestDatamerge org.apache.hadoop.mapreduce.TestMapCollection org.apache.hadoop.fs.TestDFSIO org.apache.hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates org.apache.hadoop.mapred.TestMapRed org.apache.hadoop.mapred.TestFileOutputFormat org.apache.hadoop.mapreduce.TestValueIterReset org.apache.hadoop.mapred.TestMapOutputType org.apache.hadoop.mapred.TestJobCounters org.apache.hadoop.fs.TestFileSystem org.apache.hadoop.conf.TestNoDefaultsJobConf org.apache.hadoop.mapred.TestReporter org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator org.apache.hadoop.mapreduce.lib.chain.TestChainErrors org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs org.apache.hadoop.mapred.TestComparators org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2889//console This message is automatically generated.
          Hide
          Benoy Antony added a comment -

          fixing the spacing issue on the new files

          Show
          Benoy Antony added a comment - fixing the spacing issue on the new files
          Hide
          Benoy Antony added a comment -

          Thanks Robert. I'll fix the tab issue.

          Show
          Benoy Antony added a comment - Thanks Robert. I'll fix the tab issue.
          Hide
          Robert Joseph Evans added a comment -

          The changes look OK to me too for the most part, but the new files you have added are using tabs, and the spacing appears to be off a bit. Our style guide says that we should be using 2 spaces for each indentation, not tabs. Please fix it.

          Show
          Robert Joseph Evans added a comment - The changes look OK to me too for the most part, but the new files you have added are using tabs, and the spacing appears to be off a bit. Our style guide says that we should be using 2 spaces for each indentation, not tabs. Please fix it.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12544256/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544256/MR_4554_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2832//console This message is automatically generated.
          Hide
          Benoy Antony added a comment -

          Attaching patch with the following changes

          1) Fixed TestStagingCleanup to fix the test failures.

          2) Fixed TestMRCredentials to fix the deprecation warnings

          Show
          Benoy Antony added a comment - Attaching patch with the following changes 1) Fixed TestStagingCleanup to fix the test failures. 2) Fixed TestMRCredentials to fix the deprecation warnings
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543901/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 2061 javac compiler warnings (more than the trunk's current 2059 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543901/MR_4554_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The applied patch generated 2061 javac compiler warnings (more than the trunk's current 2059 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2819//console This message is automatically generated.
          Hide
          Benoy Antony added a comment -

          Suppressing the deprecation warning due to the use of MiniMRCluster

          Show
          Benoy Antony added a comment - Suppressing the deprecation warning due to the use of MiniMRCluster
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543333/MR_4554_trunk.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          -1 javac. The applied patch generated 2063 javac compiler warnings (more than the trunk's current 2059 warnings).

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

          org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup
          org.apache.hadoop.mapred.TestMiniMRClasspath

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543333/MR_4554_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. -1 javac. The applied patch generated 2063 javac compiler warnings (more than the trunk's current 2059 warnings). +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup org.apache.hadoop.mapred.TestMiniMRClasspath +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2813//console This message is automatically generated.
          Hide
          Benoy Antony added a comment -

          Attaching the new patch with Daryn's suggested improvement.

          Show
          Benoy Antony added a comment - Attaching the new patch with Daryn's suggested improvement.
          Hide
          Daryn Sharp added a comment -

          Minor suggestion, in TaskAttemptImpl you could replace addAll with taskCredentials = new Credentials(credentials);.

          Otherwise, +1. Good job!

          Show
          Daryn Sharp added a comment - Minor suggestion, in TaskAttemptImpl you could replace addAll with taskCredentials = new Credentials(credentials); . Otherwise, +1. Good job!
          Hide
          Benoy Antony added a comment -

          Attaching latest patch after merging changes from trunk.

          Show
          Benoy Antony added a comment - Attaching latest patch after merging changes from trunk.
          Hide
          Benoy Antony added a comment -

          A correction : HADOOP-8225 introduced the problem mentioned in HADOOP-8726 .

          Show
          Benoy Antony added a comment - A correction : HADOOP-8225 introduced the problem mentioned in HADOOP-8726 .
          Hide
          Benoy Antony added a comment -

          The included test passes in 1.0 , but fails on trunk.
          The attached patch for 1.1 is a test only patch.
          The patch for trunk includes test plus the required fix.

          Show
          Benoy Antony added a comment - The included test passes in 1.0 , but fails on trunk. The attached patch for 1.1 is a test only patch. The patch for trunk includes test plus the required fix.
          Hide
          Benoy Antony added a comment -

          It works in MR 1.0 , but coes not work in trunk as mentioned in description.

          HADOOP-8225 did not introduce the problem mentioned in this issue.
          HADOOP-8225 introduced the problem mentioned in HADOOP-8276 .

          Show
          Benoy Antony added a comment - It works in MR 1.0 , but coes not work in trunk as mentioned in description. HADOOP-8225 did not introduce the problem mentioned in this issue. HADOOP-8225 introduced the problem mentioned in HADOOP-8276 .
          Hide
          Daryn Sharp added a comment -

          Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary.

          I don't think passing secrets them via mapreduce.job.credentials.binary ever worked? MR code appears to explicitly pass only tokens.

          Question: did HADOOP-8225 actually introduce these issues? As best as I can tell, these are additional bugs that also aren't handled? I presume this issue manifests only when an insecure cluster accesses a secure cluster?

          Show
          Daryn Sharp added a comment - Credentials (secret keys) can be passed to a job via mapreduce.job.credentials.json or mapreduce.job.credentials.binary. I don't think passing secrets them via mapreduce.job.credentials.binary ever worked? MR code appears to explicitly pass only tokens. Question: did HADOOP-8225 actually introduce these issues? As best as I can tell, these are additional bugs that also aren't handled? I presume this issue manifests only when an insecure cluster accesses a secure cluster?
          Hide
          Benoy Antony added a comment -

          Attaching a patch which has the test only so that failure can be observed. This will be removed later.

          Show
          Benoy Antony added a comment - Attaching a patch which has the test only so that failure can be observed. This will be removed later.
          Hide
          Benoy Antony added a comment -

          Thanks so much for reviewing, Daryn.

          I have reviewed HADOOP-8725/HADOOP-8225. This issue is not resolved with those patches.

          When Credentials are transmitted from Job Submitter to the tasks, there are a few other places where isSecurityEnabled check is made.
          In those places, the credentials are passed further only if isSecurityEnabled is true .
          This patch fixes it.

          This patch also adds a test which submits a job with secrets and then verifies that secrets are available to tasks.
          Without the patch, the test fails. I am attaching a patch with the test only so that the test failure can be observed.

          Meanwhile I found an issue HADOOP-8225 which results in a situation in which secrets (not tokens) present in credentials are not available to MR tasks. I opened HADOOP-8726 to fix that. I attached an initial patch to unblock this issue. It will be great if you could review it.

          If MAPREDUCE-4554 and HADOOP-8726 patches applied, the above mentioned test will pass.

          Show
          Benoy Antony added a comment - Thanks so much for reviewing, Daryn. I have reviewed HADOOP-8725 / HADOOP-8225 . This issue is not resolved with those patches. When Credentials are transmitted from Job Submitter to the tasks, there are a few other places where isSecurityEnabled check is made. In those places, the credentials are passed further only if isSecurityEnabled is true . This patch fixes it. This patch also adds a test which submits a job with secrets and then verifies that secrets are available to tasks. Without the patch, the test fails. I am attaching a patch with the test only so that the test failure can be observed. Meanwhile I found an issue HADOOP-8225 which results in a situation in which secrets (not tokens) present in credentials are not available to MR tasks. I opened HADOOP-8726 to fix that. I attached an initial patch to unblock this issue. It will be great if you could review it. If MAPREDUCE-4554 and HADOOP-8726 patches applied, the above mentioned test will pass.
          Hide
          Benoy Antony added a comment -

          Attaching a new patch for trunk. The previous patch simply comments out the securityenabled checks. This patch removes those checks.

          Show
          Benoy Antony added a comment - Attaching a new patch for trunk. The previous patch simply comments out the securityenabled checks. This patch removes those checks.
          Hide
          Daryn Sharp added a comment -

          Could you please look at those jiras and explain if this patch is still needed, and if so why? Those jiras make tokens propagate to tasks.

          Show
          Daryn Sharp added a comment - Could you please look at those jiras and explain if this patch is still needed, and if so why? Those jiras make tokens propagate to tasks.
          Hide
          Daryn Sharp added a comment -

          I'll definitely review. I need to make sure it doesn't collide with HADOOP-8725/HADOOP-8225.

          Show
          Daryn Sharp added a comment - I'll definitely review. I need to make sure it doesn't collide with HADOOP-8725 / HADOOP-8225 .
          Hide
          Benoy Antony added a comment -

          Could I please get a review of this patch ?

          Show
          Benoy Antony added a comment - Could I please get a review of this patch ?
          Hide
          Benoy Antony added a comment -

          The patch adds a test case for 1.1 and 2.0 .
          It also removes security on/off checks when transmitting credentials.

          Show
          Benoy Antony added a comment - The patch adds a test case for 1.1 and 2.0 . It also removes security on/off checks when transmitting credentials.

            People

            • Assignee:
              Benoy Antony
              Reporter:
              Benoy Antony
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development