History files are given world readable permissions.

• Key: HADOOP-4783
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Blocker
• Assignee: Amareshwari Sriramadasu
• Reporter: Hemanth Yamijala
• Affects Version/s: 0.19.0
• Fix Version/s: 0.20.0

Description

It is found that history files are being created with permissions 0777. On shared clusters this is opening up too much. However, there is a requirement to allow Chukwa to read history files. (See HADOOP-4705). This issue is to set up appropriate permissions for the files to be as restrictive as required, while still fixing the problem for Chukwa.

Comments

[Nigel Daley]

Hemanth, since HADOOP-4782 is a blocker for 0.20, why isn't this issue too? They seem to go together.

[Hemanth Yamijala]

Yes, I think it should be a blocker. Marking as such.

[Amareshwari Sriramadasu]

I would propose we have 0710 permission for history directory and 0740 permission for history files, sothat clients like Chukwa and Hadoop-vaidya would read history files. Thoughts?

[Amareshwari Sriramadasu]

patch changing the permissiond properly

[Amareshwari Sriramadasu]

test-patch result:

     [exec]
     [exec] +1 overall.
     [exec]
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec]
     [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
     [exec]
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec]
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
     [exec]
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec]
     [exec]     +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
     [exec]
     [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
     [exec]
     [exec]

unit tests passed on my machine.

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401101/patch-4783.txt
against trunk revision 748861.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/23/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/23/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/23/console

This message is automatically generated.

[Jothi Padmanabhan]

conf.xml should have 0740 permissions too.

[Amareshwari Sriramadasu]

Patch with review comments incorporated.

[Amareshwari Sriramadasu]

test-patch result:
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]

All unit tests passed on my machine

[Jothi Padmanabhan]

Patch looks good.

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401371/patch-4783-1.txt
against trunk revision 750237.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/47/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/47/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/47/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/47/console

This message is automatically generated.

[Hemanth Yamijala]

I had an offline discussion with Rajiv on this. We felt that for the time being 740 for files and 750 for the history directory should be sufficient.

[Amareshwari Sriramadasu]

Patch chaging the history directory permissions to 750

[Jothi Padmanabhan]

+1

[Amareshwari Sriramadasu]

The same patch applies to branch 0.20 without the testcase changes, since TestJobHistory does not exist in 0.20

[Amareshwari Sriramadasu]

Patch updated with trunk

[Hadoop QA]

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402112/patch-4783-2.txt
against trunk revision 753052.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

+1 contrib tests. The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/83/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/83/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/83/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/83/console

This message is automatically generated.

[Amareshwari Sriramadasu]

resubmitting to hudson

[Hadoop QA]

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402118/patch-4783-2.txt
against trunk revision 753113.

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 3 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 findbugs. The patch does not introduce any new Findbugs warnings.

+1 Eclipse classpath. The patch retains Eclipse classpath integrity.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed core unit tests.

-1 contrib tests. The patch failed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/85/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/85/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/85/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/85/console

This message is automatically generated.

[Hemanth Yamijala]

Code changes look good to me. +1

[Hemanth Yamijala]

Also, the test case failure seems unrelated to this patch. It is in testProxyFilter.

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20. The patch applied cleanly to the branch as well, I guess because HADOOP-4191 is committed now to the branch.

Thanks, Amareshwari !

[Hudson]

Integrated in Hadoop-trunk #780 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/780/)
. Change permissions on history files on the jobtracker to be only group readable instead of world readable. Contributed by Amareshwari Sriramadasu.