Hadoop YARN
  1. Hadoop YARN
  2. YARN-47 [Umbrella] Security issues in YARN
  3. YARN-578

NodeManager should use SecureIOUtils for serving and aggregating logs

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0-beta
    • Component/s: nodemanager
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Log servlets for serving logs and the ShuffleService for serving intermediate outputs both should use SecureIOUtils for avoiding symlink attacks.

      1. YARN-578-20130529.patch
        26 kB
        Omkar Vinit Joshi
      2. YARN-578-20130528.patch
        22 kB
        Omkar Vinit Joshi
      3. YARN-578-20130528.1.patch
        26 kB
        Omkar Vinit Joshi
      4. YARN-578-20130520.patch
        21 kB
        Omkar Vinit Joshi
      5. YARN-578-20130520.branch-2.patch
        21 kB
        Omkar Vinit Joshi
      6. YARN-578-20130520.1.patch
        22 kB
        Omkar Vinit Joshi
      7. YARN-578-20130506.patch
        11 kB
        Omkar Vinit Joshi
      8. yarn-578-20130426.patch
        6 kB
        Omkar Vinit Joshi

        Issue Links

          Activity

          Hide
          Omkar Vinit Joshi added a comment -

          There are 3 issues related to symlink attacks in serving logs and ShuffleService

          • Index file (file.out.index) :- [Location - SpillRecord.SpillRecord() - FSDataInputStream ] Here we are directly trying to read from file.out.index file (So the potential problem is that we ShuffleHandler may end up reading files of yarn user or yarn group user. [ yarn:yarn is running nodemanager ]
          • Map output file (file.out) :- [Location - ShuffleHandler.sendMapOutput() - RandomAccessFile ] Here too we are directly accessing file.out file.
          • Container Logs :- [Location - ContainerLogsPage.printLogs() - FileInputStream ] Here we are directly accessing container logs as yarn:yarn user.

          At present SecureIOUtils supports only FileInputStream, so I am adding support for 2 more streams, FSDataInputStream (This is required if you want a stream to be position readable or seekable) and RandomAccessFile. Filing a separate JIRA for this. HADOOP-9511

          Show
          Omkar Vinit Joshi added a comment - There are 3 issues related to symlink attacks in serving logs and ShuffleService Index file (file.out.index) :- [Location - SpillRecord.SpillRecord() - FSDataInputStream ] Here we are directly trying to read from file.out.index file (So the potential problem is that we ShuffleHandler may end up reading files of yarn user or yarn group user. [ yarn:yarn is running nodemanager ] Map output file (file.out) :- [Location - ShuffleHandler.sendMapOutput() - RandomAccessFile ] Here too we are directly accessing file.out file. Container Logs :- [Location - ContainerLogsPage.printLogs() - FileInputStream ] Here we are directly accessing container logs as yarn:yarn user. At present SecureIOUtils supports only FileInputStream, so I am adding support for 2 more streams, FSDataInputStream (This is required if you want a stream to be position readable or seekable) and RandomAccessFile. Filing a separate JIRA for this. HADOOP-9511
          Hide
          Omkar Vinit Joshi added a comment -

          Fix for SecureIOUtils

          Show
          Omkar Vinit Joshi added a comment - Fix for SecureIOUtils
          Hide
          Omkar Vinit Joshi added a comment -

          Updating the patch. Not sure about the junit test if it is required here.
          For ContainerLogsPage in case of an IOException not printing the whole exception stack on NodeManager UI; instead logging it.

          Show
          Omkar Vinit Joshi added a comment - Updating the patch. Not sure about the junit test if it is required here. For ContainerLogsPage in case of an IOException not printing the whole exception stack on NodeManager UI; instead logging it.
          Hide
          Omkar Vinit Joshi added a comment -

          Testing :- This patch I have tested on Secure setup.[ Ubuntu 12.04, single node using kerberos].

          Show
          Omkar Vinit Joshi added a comment - Testing :- This patch I have tested on Secure setup.[ Ubuntu 12.04, single node using kerberos].
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Can you use this only for YARN changes i.e. serving logs and open a separate MAPREDUCE ticket for ShuffleHandler?

          For the YARN changes:

          • Remove the comment above the code which talks about SecureIOUtils
          • I think we should separate the exception message to clearly say whether this was an permission-issue or something else.
          Show
          Vinod Kumar Vavilapalli added a comment - Can you use this only for YARN changes i.e. serving logs and open a separate MAPREDUCE ticket for ShuffleHandler? For the YARN changes: Remove the comment above the code which talks about SecureIOUtils I think we should separate the exception message to clearly say whether this was an permission-issue or something else.
          Hide
          Omkar Vinit Joshi added a comment -

          This issue will now only track yarn related changes (ContainerLogsPage). Creating separate jira for mapreduce issue. MAPREDUCE-5208

          Show
          Omkar Vinit Joshi added a comment - This issue will now only track yarn related changes (ContainerLogsPage). Creating separate jira for mapreduce issue. MAPREDUCE-5208
          Hide
          Omkar Vinit Joshi added a comment -

          Fixing both comments and throwing different exceptions for different scenarios in ContainerLogsPage. Adding test. Verified it on Secure setup with NativeIO enabled.

          Show
          Omkar Vinit Joshi added a comment - Fixing both comments and throwing different exceptions for different scenarios in ContainerLogsPage. Adding test. Verified it on Secure setup with NativeIO enabled.
          Hide
          Omkar Vinit Joshi added a comment -

          Adding krb5.conf file. Needed for Testing container logs page.

          Show
          Omkar Vinit Joshi added a comment - Adding krb5.conf file. Needed for Testing container logs page.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Minor nits:

          ContainerLogsPage:

          • Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check?
          • The exception message is confusing. Let us say that the authenticated user is foo and the application-submitter is bar. The message talks about bar not having permissions to read the file which is totally confusing to foo. We should instead say something in the lines of "The log-file generated by the application-submitter foo has invalid permissions, so not showing etc.."
          • You don't need the unnecessary string concatenation: ' doesn't have permissions to read " + "log file :"
          Show
          Vinod Kumar Vavilapalli added a comment - Minor nits: ContainerLogsPage: Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check? The exception message is confusing. Let us say that the authenticated user is foo and the application-submitter is bar. The message talks about bar not having permissions to read the file which is totally confusing to foo. We should instead say something in the lines of "The log-file generated by the application-submitter foo has invalid permissions, so not showing etc.." You don't need the unnecessary string concatenation: ' doesn't have permissions to read " + "log file :"
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Okay, I just had an enlightening experience and I realized we need to fix more issues:

          • LogAggregationService can ignore these permissions and upload sensitive files! Please fix this and write a test to verify that it doesn't happen.
          • It seems like when logs are deleted, we are using the correct user to delete them. But can you write tests to validate this for two cases (1) when log-aggregation is enabled and (2) when it isn't.
          Show
          Vinod Kumar Vavilapalli added a comment - Okay, I just had an enlightening experience and I realized we need to fix more issues: LogAggregationService can ignore these permissions and upload sensitive files! Please fix this and write a test to verify that it doesn't happen. It seems like when logs are deleted, we are using the correct user to delete them. But can you write tests to validate this for two cases (1) when log-aggregation is enabled and (2) when it isn't.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          This ticket is only addressing logs - fixing the title.

          Show
          Vinod Kumar Vavilapalli added a comment - This ticket is only addressing logs - fixing the title.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/)
          HADOOP-9511. Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578. Contributed by Omkar Vinit Joshi. (Revision 1479010)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3712 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3712/ ) HADOOP-9511 . Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578 . Contributed by Omkar Vinit Joshi. (Revision 1479010) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/)
          HADOOP-9511. Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578. Contributed by Omkar Vinit Joshi. (Revision 1479010)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #202 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/202/ ) HADOOP-9511 . Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578 . Contributed by Omkar Vinit Joshi. (Revision 1479010) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/)
          HADOOP-9511. Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578. Contributed by Omkar Vinit Joshi. (Revision 1479010)

          Result = FAILURE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1391 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1391/ ) HADOOP-9511 . Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578 . Contributed by Omkar Vinit Joshi. (Revision 1479010) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/)
          HADOOP-9511. Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578. Contributed by Omkar Vinit Joshi. (Revision 1479010)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1418 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1418/ ) HADOOP-9511 . Adding support for additional input streams (FSDataInputStream and RandomAccessFile) in SecureIOUtils so as to help YARN-578 . Contributed by Omkar Vinit Joshi. (Revision 1479010) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1479010 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SecureIOUtils.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/TestSecureIOUtils.java
          Hide
          Omkar Vinit Joshi added a comment -

          Thanks vinod..

          Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check?

          No.. as both of them are throwing IOException only (with different messages.. should we fix the exception type for both of them??) and they will occur for the same SecureIOUtils.open call.

          The exception message is confusing. Let us say that the authenticated user is foo and the application-submitter is bar. The message talks about bar not having permissions to read the file which is totally confusing to foo. We should instead say something in the lines of "The log-file generated by the application-submitter foo has invalid permissions, so not showing etc.."

          updated the message

          You don't need the unnecessary string concatenation: ' doesn't have permissions to read " + "log file :"

          Yeah fixed it.

          LogAggregationService can ignore these permissions and upload sensitive files! Please fix this and write a test to verify that it doesn't happen.

          Fixed. added test

          It seems like when logs are deleted, we are using the correct user to delete them. But can you write tests to validate this for two cases (1) when log-aggregation is enabled and (2) when it isn't.

          1) added test for it.
          2) is already verified.

          Show
          Omkar Vinit Joshi added a comment - Thanks vinod.. Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check? No.. as both of them are throwing IOException only (with different messages.. should we fix the exception type for both of them??) and they will occur for the same SecureIOUtils.open call. The exception message is confusing. Let us say that the authenticated user is foo and the application-submitter is bar. The message talks about bar not having permissions to read the file which is totally confusing to foo. We should instead say something in the lines of "The log-file generated by the application-submitter foo has invalid permissions, so not showing etc.." updated the message You don't need the unnecessary string concatenation: ' doesn't have permissions to read " + "log file :" Yeah fixed it. LogAggregationService can ignore these permissions and upload sensitive files! Please fix this and write a test to verify that it doesn't happen. Fixed. added test It seems like when logs are deleted, we are using the correct user to delete them. But can you write tests to validate this for two cases (1) when log-aggregation is enabled and (2) when it isn't. 1) added test for it. 2) is already verified.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12583925/YARN-578-20130520.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/964//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/964//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583925/YARN-578-20130520.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/964//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/964//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12583940/YARN-578-20130520.branch-2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/967//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/967//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583940/YARN-578-20130520.branch-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/967//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/967//console This message is automatically generated.
          Hide
          Omkar Vinit Joshi added a comment -

          forgot to add krb5.conf file...attaching it.

          Show
          Omkar Vinit Joshi added a comment - forgot to add krb5.conf file...attaching it.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12583947/YARN-578-20130520.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/968//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/968//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583947/YARN-578-20130520.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/968//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/968//console This message is automatically generated.
          Hide
          Sandy Ryza added a comment -

          Hi Omkar. The patch is looking good. A couple nits:

          • You should use a LOG message instead of System.out.println in ContainerLogsPage, and the "Opeing" in it is a typo.
          • "Exception Reading log file." Reading should not be capitalized.
          • Unnecessary whitespace change on this line: logAggregationService.getRemoteNodeLogFileForApp(appId, this.user));
          • "' doesn't own requested log file :syslog". There should be a space between the colon and syslog.
          Show
          Sandy Ryza added a comment - Hi Omkar. The patch is looking good. A couple nits: You should use a LOG message instead of System.out.println in ContainerLogsPage, and the "Opeing" in it is a typo. "Exception Reading log file." Reading should not be capitalized. Unnecessary whitespace change on this line: logAggregationService.getRemoteNodeLogFileForApp(appId, this.user)); "' doesn't own requested log file :syslog". There should be a space between the colon and syslog.
          Hide
          Omkar Vinit Joshi added a comment -

          Thanks Sandy Ryza for reviewing..
          Fixed all the comments..

          Show
          Omkar Vinit Joshi added a comment - Thanks Sandy Ryza for reviewing.. Fixed all the comments..
          Hide
          Sandy Ryza added a comment -

          Thanks Omkar Vinit Joshi, looks good to me now.

          Show
          Sandy Ryza added a comment - Thanks Omkar Vinit Joshi , looks good to me now.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585057/YARN-578-20130528.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1010//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1010//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585057/YARN-578-20130528.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1010//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1010//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Few comments on the latest patch:

          • LogValue: If we run into an exception reading one file, like the security exception, the remaining logs will be completely skipped. Not directly caused by your patch but it gets worse with the patch. I think we should fix it here.
          • LogValue: When such security exception happens, we should log it specifically so as to help debugging later. When we have log-aggregation status exposed to users, we can add a diagnostic message there.
          • Also please add a test verifying the above.
          • ContainerLogsPage: the info log message isn't useful, can be removed.

          Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check?

          Missed this?

          Show
          Vinod Kumar Vavilapalli added a comment - Few comments on the latest patch: LogValue: If we run into an exception reading one file, like the security exception, the remaining logs will be completely skipped. Not directly caused by your patch but it gets worse with the patch. I think we should fix it here. LogValue: When such security exception happens, we should log it specifically so as to help debugging later. When we have log-aggregation status exposed to users, we can add a diagnostic message there. Also please add a test verifying the above. ContainerLogsPage: the info log message isn't useful, can be removed. Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check? Missed this?
          Hide
          Omkar Vinit Joshi added a comment -

          Thanks Vinod ..

          LogValue: If we run into an exception reading one file, like the security exception, the remaining logs will be completely skipped. Not directly caused by your patch but it gets worse with the patch. I think we should fix it here.

          sounds good... Now making sure that Log aggregator will try to upload all container log files. In case of an exception it will log an error. (error will also be added into aggregated log file corresponding to log file entry for future purpose). Added test case to verify this. (test case for log aggregation for different log files for same container with different ownerships is bit difficult with current implementation unless we make two append calls). Adding test for two containers trying to append log files one after the other and then verifying aggregated log file output.

          LogValue: When such security exception happens, we should log it specifically so as to help debugging later. When we have log-aggregation status exposed to users, we can add a diagnostic message there.

          Fixed.

          Also please add a test verifying the above.

          added..

          ContainerLogsPage: the info log message isn't useful, can be removed.

          removed..

          Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check?

          humm.. thought about it ..was earlier thinking to keep it in same block.... separating try catch for open call and other read calls.

          Show
          Omkar Vinit Joshi added a comment - Thanks Vinod .. LogValue: If we run into an exception reading one file, like the security exception, the remaining logs will be completely skipped. Not directly caused by your patch but it gets worse with the patch. I think we should fix it here. sounds good... Now making sure that Log aggregator will try to upload all container log files. In case of an exception it will log an error. (error will also be added into aggregated log file corresponding to log file entry for future purpose). Added test case to verify this. (test case for log aggregation for different log files for same container with different ownerships is bit difficult with current implementation unless we make two append calls). Adding test for two containers trying to append log files one after the other and then verifying aggregated log file output. LogValue: When such security exception happens, we should log it specifically so as to help debugging later. When we have log-aggregation status exposed to users, we can add a diagnostic message there. Fixed. Also please add a test verifying the above. added.. ContainerLogsPage: the info log message isn't useful, can be removed. removed.. Instead of matching messages in the exception block, why not separate the try {} catch {} block for the SecureIOUtils check? humm.. thought about it ..was earlier thinking to keep it in same block.... separating try catch for open call and other read calls.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585133/YARN-578-20130528.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1018//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1018//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585133/YARN-578-20130528.1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1018//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1018//console This message is automatically generated.
          Hide
          Omkar Vinit Joshi added a comment -

          Updating test case...
          Making sure that in a single container now we have two files (stderr owned by user and stdout owned by some randomUser). Log aggregation should fail for first one but should succeed for the later. As this method doesn't throw any exception when either container log file is not found or it is a symlink; we actually don't need to test it for another container log files after this container log files are read.

          Show
          Omkar Vinit Joshi added a comment - Updating test case... Making sure that in a single container now we have two files (stderr owned by user and stdout owned by some randomUser). Log aggregation should fail for first one but should succeed for the later. As this method doesn't throw any exception when either container log file is not found or it is a symlink; we actually don't need to test it for another container log files after this container log files are read.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585297/YARN-578-20130529.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1027//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1027//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585297/YARN-578-20130529.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 4 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1027//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1027//console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1, this looks good. Checking this in.

          Show
          Vinod Kumar Vavilapalli added a comment - +1, this looks good. Checking this in.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I believe we need a separate branch-2 patch. Can you please upload one? Tx.

          Show
          Vinod Kumar Vavilapalli added a comment - I believe we need a separate branch-2 patch. Can you please upload one? Tx.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3803 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3803/)
          YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3803 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3803/ ) YARN-578 . Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Hide
          Omkar Vinit Joshi added a comment -

          The patch is same for both branch-2 and trunk.. compiled and tested locally.

          Show
          Omkar Vinit Joshi added a comment - The patch is same for both branch-2 and trunk.. compiled and tested locally.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Committed this to trunk and branch-2. Thanks Omkar!

          Show
          Vinod Kumar Vavilapalli added a comment - Committed this to trunk and branch-2. Thanks Omkar!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #225 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/225/)
          YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #225 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/225/ ) YARN-578 . Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1415 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1415/)
          YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672)

          Result = FAILURE
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1415 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1415/ ) YARN-578 . Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1441/)
          YARN-578. Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672)

          Result = SUCCESS
          vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1441 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1441/ ) YARN-578 . Fixed NM to use SecureIOUtils for reading and aggregating logs. Contributed by Omkar Vinit Joshi. (Revision 1487672) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487672 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/krb5.conf /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/AppLogAggregatorImpl.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerLogsPage.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java

            People

            • Assignee:
              Omkar Vinit Joshi
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development