Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4633

history server doesn't set permissions on all subdirs

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 3.0.0, 2.0.2-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: jobhistoryserver
    • Labels:
      None

      Description

      The job history server creates a bunch of subdirectories under the "done" directory. They are like 2012/09/03/000000. It only sets the permissions on the last one, ie 000000 to 770. So the 2012/09/03 aren't explicitly set so if the umask is more restrictive, they won't be set as it expects.

        Issue Links

          Activity

          Thomas Graves created issue -
          Thomas Graves made changes -
          Field Original Value New Value
          Assignee Thomas Graves [ tgraves ]
          Thomas Graves made changes -
          Priority Major [ 3 ] Critical [ 2 ]
          Hide
          Thomas Graves added a comment -

          Vaidya uses the history files and this issue is blocking it being used.

          Show
          Thomas Graves added a comment - Vaidya uses the history files and this issue is blocking it being used.
          Thomas Graves made changes -
          Link This issue is blocked by HDFS-3890 [ HDFS-3890 ]
          Hide
          Thomas Graves added a comment -

          Preliminary patch that relies on HDFS-3890.

          Show
          Thomas Graves added a comment - Preliminary patch that relies on HDFS-3890 .
          Thomas Graves made changes -
          Attachment MAPREDUCE-4633.patch [ 12543868 ]
          Thomas Graves made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.23.3, 2.2.0-alpha [ 12320060, 12322471 ] 0.23.3, 3.0.0, 2.2.0-alpha [ 12320060, 12320355, 12322471 ]
          Thomas Graves made changes -
          Link This issue is related to MAPREDUCE-4636 [ MAPREDUCE-4636 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12543868/MAPREDUCE-4633.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2818//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2818//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543868/MAPREDUCE-4633.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2818//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2818//console This message is automatically generated.
          Hide
          Robert Joseph Evans added a comment -

          The change looks fine to me. It would be nice to have a unit test for this, but all it really would do is prove that HDFS-3890 is correct. I am a +1 on this once HDFS-3890 goes in.

          Show
          Robert Joseph Evans added a comment - The change looks fine to me. It would be nice to have a unit test for this, but all it really would do is prove that HDFS-3890 is correct. I am a +1 on this once HDFS-3890 goes in.
          Hide
          Robert Joseph Evans added a comment -

          Thanks Tom,

          +1 I put this into trunk, branch-2, branch-0.23 and 0.23.3

          Show
          Robert Joseph Evans added a comment - Thanks Tom, +1 I put this into trunk, branch-2, branch-0.23 and 0.23.3
          Robert Joseph Evans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Fix Version/s 2.2.0-alpha [ 12322471 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2693 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2693/)
          MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2693 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2693/ ) MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2756 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2756/)
          MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2756 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2756/ ) MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2717 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2717/)
          MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2717 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2717/ ) MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/)
          MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1158/ ) MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/)
          MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1189 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1189/ ) MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381614) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381614 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #367 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/367/)
          svn merge -c 1381614 FIXES: MAPREDUCE-4633. history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381619)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #367 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/367/ ) svn merge -c 1381614 FIXES: MAPREDUCE-4633 . history server doesn't set permissions on all subdirs (tgraves via bobby) (Revision 1381619) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1381619 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobHistoryUtils.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
          Arun C Murthy made changes -
          Fix Version/s 3.0.0 [ 12320355 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Thomas Graves
              Reporter:
              Thomas Graves
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development