Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4612

job summary file permissions not set when its created

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 3.0.0, 2.0.2-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: None
    • Labels:
      None

      Description

      The job summary file permissions are not set when its written out by to the done intermediate directory. The conf and jhist files are both set properly but the summary file doesn't follow the same path.

      This can cause the summary file to not be copied to the done directory if the default umask is set to be restrictive (like 600) and the mapred user can't read it.

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320355 ]
          Thomas Graves made changes -
          Link This issue is related to MAPREDUCE-4636 [ MAPREDUCE-4636 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/)
          MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/ ) MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/)
          MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/ ) MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/)
          svn merge -c 1379584 FIXES: MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379586)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379586
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/ ) svn merge -c 1379584 FIXES: MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379586) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379586 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2691/)
          MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2691 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2691/ ) MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2664 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2664/)
          MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2664 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2664/ ) MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2727 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2727/)
          MAPREDUCE-4612. job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2727 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2727/ ) MAPREDUCE-4612 . job summary file permissions not set when its created (tgraves via bobby) (Revision 1379584) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379584 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java
          Robert Joseph Evans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Fix Version/s 2.2.0-alpha [ 12322471 ]
          Resolution Fixed [ 1 ]
          Hide
          Robert Joseph Evans added a comment -

          Thanks Tom,

          I put this into trunk, branch-2, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks Tom, I put this into trunk, branch-2, and branch-0.23
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543332/MAPREDUCE-4612.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2805//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2805//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543332/MAPREDUCE-4612.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2805//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2805//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543332/MAPREDUCE-4612.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2804//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2804//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543332/MAPREDUCE-4612.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2804//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2804//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The patch looks good to me. +1 assuming Jenkins comes back OK barring the missing test.

          Show
          Robert Joseph Evans added a comment - The patch looks good to me. +1 assuming Jenkins comes back OK barring the missing test.
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Thomas Graves made changes -
          Field Original Value New Value
          Attachment MAPREDUCE-4612.patch [ 12543332 ]
          Hide
          Thomas Graves added a comment -

          Patch to fix the permission on the job summary file. Unfortunately no unit tests for this. I tested it manually on the single node cluster.

          I will file a separate jira to write more unit tests for the JobHistoryEventHandler

          Show
          Thomas Graves added a comment - Patch to fix the permission on the job summary file. Unfortunately no unit tests for this. I tested it manually on the single node cluster. I will file a separate jira to write more unit tests for the JobHistoryEventHandler
          Thomas Graves created issue -

            People

            • Assignee:
              Thomas Graves
              Reporter:
              Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development