Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha, 0.23.3
    • Component/s: nodemanager
    • Labels:
      None

      Description

      The NM is exhausting its fds because it's not closing fs instances when the app is finished.

      1. MAPREDUCE-4323.patch
        8 kB
        Daryn Sharp
      2. MAPREDUCE-4323.patch
        11 kB
        Jason Lowe
      3. MAPREDUCE-4323.patch
        12 kB
        Jason Lowe
      4. MAPREDUCE-4323.patch
        23 kB
        Jason Lowe

        Issue Links

          Activity

          Daryn Sharp created issue -
          Hide
          Daryn Sharp added a comment -

          In particular, DFSClient maintains a socket cache. Closed sockets are not detected until another connection is needed, or the client is closed. That's another issue, but the NM's failure to close filesystems for a user after the app completes causes a leak of sockets in the CLOSE_WAIT state that eventually exhaust fds for the process.

          Calling FileSystem.closeAllForUGI, as the JT does, is troublesome that it may close the fs for other apps running as that user. One approach is to partition the fs cache to allow each app to maintain its own cache of filesystems. See HADOOP-8490 for possible approaches, which would allow the closing of the app's filesystems ala the JT.

          Also note that failure to close filesystems causes all future jobs to use the configuration of the first job. This will be very problematic, so it's imperative to ensure apps each get their own cached instances.

          Show
          Daryn Sharp added a comment - In particular, DFSClient maintains a socket cache. Closed sockets are not detected until another connection is needed, or the client is closed. That's another issue, but the NM's failure to close filesystems for a user after the app completes causes a leak of sockets in the CLOSE_WAIT state that eventually exhaust fds for the process. Calling FileSystem.closeAllForUGI , as the JT does, is troublesome that it may close the fs for other apps running as that user. One approach is to partition the fs cache to allow each app to maintain its own cache of filesystems. See HADOOP-8490 for possible approaches, which would allow the closing of the app's filesystems ala the JT. Also note that failure to close filesystems causes all future jobs to use the configuration of the first job. This will be very problematic, so it's imperative to ensure apps each get their own cached instances.
          Daryn Sharp made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-8490 [ HADOOP-8490 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          This looks like a problem of the newly added socket cache. Once it is fixed (say, it is removed for the sake of discussion), are there other problems?

          Show
          Tsz Wo Nicholas Sze added a comment - This looks like a problem of the newly added socket cache. Once it is fixed (say, it is removed for the sake of discussion), are there other problems?
          Hide
          Todd Lipcon added a comment -

          See comment on HADOOP-8490: I think the NM should just be side-stepping the FS cache, so it can explicitly close the FS when necessary.

          Show
          Todd Lipcon added a comment - See comment on HADOOP-8490 : I think the NM should just be side-stepping the FS cache, so it can explicitly close the FS when necessary.
          Hide
          Daryn Sharp added a comment -

          FileSystem.closeAllForUGI is actually a reasonable approach. Each request is creating a new ugi so there's no issue with "pulling the rug out" from underneath other fs users.

          Show
          Daryn Sharp added a comment - FileSystem.closeAllForUGI is actually a reasonable approach. Each request is creating a new ugi so there's no issue with "pulling the rug out" from underneath other fs users.
          Daryn Sharp made changes -
          Assignee Daryn Sharp [ daryn ]
          Hide
          Daryn Sharp added a comment -

          Patch ensures closeAllForUGI is called in the event of a service init failure, or when the aggregation thread is done.

          Show
          Daryn Sharp added a comment - Patch ensures closeAllForUGI is called in the event of a service init failure, or when the aggregation thread is done.
          Daryn Sharp made changes -
          Attachment MAPREDUCE-4323.patch [ 12539698 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12539698/MAPREDUCE-4323.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539698/MAPREDUCE-4323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2715//console This message is automatically generated.
          Daryn Sharp made changes -
          Summary NM leaks sockets NM log aggregator leaks filesystems
          Hide
          Jonathan Eagles added a comment -

          This patch seems like it partially addresses the problem. Something else still seems to be leaking sockets.

          Show
          Jonathan Eagles added a comment - This patch seems like it partially addresses the problem. Something else still seems to be leaking sockets.
          Hide
          Jonathan Eagles added a comment -

          Thanks to Jason Lowe for pointing out a simple way for me to check the socket leak.

          while true; do ls /proc/<nm process id>/fd/ | wc; sleep 1; done
          
          Show
          Jonathan Eagles added a comment - Thanks to Jason Lowe for pointing out a simple way for me to check the socket leak. while true ; do ls /proc/<nm process id>/fd/ | wc; sleep 1; done
          Devaraj K made changes -
          Link This issue duplicates MAPREDUCE-4340 [ MAPREDUCE-4340 ]
          Hide
          Jason Lowe added a comment -

          Daryn and I did some more digging. It looks like the resource localization being performed by the ContainerLocalizer is also leaking filesystems.

          Show
          Jason Lowe added a comment - Daryn and I did some more digging. It looks like the resource localization being performed by the ContainerLocalizer is also leaking filesystems.
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Daryn Sharp added a comment -

          Yes, I should have commented on the title change. Log aggregation is leaking, which I fixed in this patch, but something else is also leaking. I suspected localization, and Jason Lowe and I tracked it down so I'll fix it in this patch too.

          Show
          Daryn Sharp added a comment - Yes, I should have commented on the title change. Log aggregation is leaking, which I fixed in this patch, but something else is also leaking. I suspected localization, and Jason Lowe and I tracked it down so I'll fix it in this patch too.
          Hide
          Jason Lowe added a comment -

          Updated patch to trunk and closed filesystems in ContainerLocalizer which fixes the other filesystem leak.

          Show
          Jason Lowe added a comment - Updated patch to trunk and closed filesystems in ContainerLocalizer which fixes the other filesystem leak.
          Jason Lowe made changes -
          Attachment MAPREDUCE-4323.patch [ 12541063 ]
          Jason Lowe made changes -
          Summary NM log aggregator leaks filesystems NM leaks filesystems
          Jason Lowe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 0.23.2, 2.1.0-alpha [ 12317654, 12319851, 12321442 ] 0.24.0, 0.23.3, 2.2.0-alpha [ 12317654, 12320060, 12322471 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541063/MAPREDUCE-4323.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2734//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2734//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541063/MAPREDUCE-4323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2734//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2734//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I think the additional closeAllForUGI needs to be added like this to guard against exceptions in the other cleanup operations causing the fs close to be skipped.

          } finally {
            try {
              if (exec != null) {
                exec.shutdownNow();
              }
              LocalDirAllocator.removeContext(appCacheDirContextName);
            } finally {
              closeFileSystems(ugi);
            }
          }
          

          You probably want to break out the closeAllForUGI into closeFileSystems ala the aggregator part of the patch so mocking a test will be easier.

          Show
          Daryn Sharp added a comment - I think the additional closeAllForUGI needs to be added like this to guard against exceptions in the other cleanup operations causing the fs close to be skipped. } finally { try { if (exec != null ) { exec.shutdownNow(); } LocalDirAllocator.removeContext(appCacheDirContextName); } finally { closeFileSystems(ugi); } } You probably want to break out the closeAllForUGI into closeFileSystems ala the aggregator part of the patch so mocking a test will be easier.
          Hide
          Jason Lowe added a comment -

          Updated patch per review comments.

          Show
          Jason Lowe added a comment - Updated patch per review comments.
          Jason Lowe made changes -
          Attachment MAPREDUCE-4323.patch [ 12541075 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541075/MAPREDUCE-4323.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2736//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2736//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541075/MAPREDUCE-4323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2736//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2736//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Please add a localizer test similar to my aggregation test, otherwise, looks good!

          Show
          Daryn Sharp added a comment - Please add a localizer test similar to my aggregation test, otherwise, looks good!
          Jason Lowe made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Assignee Daryn Sharp [ daryn ] Jason Lowe [ jlowe ]
          Hide
          Jason Lowe added a comment -

          Added test for ContainerLocalizer closing filesystems after localization completes.

          Show
          Jason Lowe added a comment - Added test for ContainerLocalizer closing filesystems after localization completes.
          Jason Lowe made changes -
          Attachment MAPREDUCE-4323.patch [ 12541122 ]
          Jason Lowe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12541122/MAPREDUCE-4323.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2739//testReport/
          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2739//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541122/MAPREDUCE-4323.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2739//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2739//console This message is automatically generated.
          Hide
          Jonathan Eagles added a comment -

          +1 Filesystems are not leaked using the proc fd tests above

          Show
          Jonathan Eagles added a comment - +1 Filesystems are not leaked using the proc fd tests above
          Jonathan Eagles made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Fix Version/s 2.1.0-alpha [ 12321442 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2670 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2670/)
          MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2670 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2670/ ) MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2635 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2635/)
          MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164)

          Result = FAILURE
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2635 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2635/ ) MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2606 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2606/)
          MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2606 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2606/ ) MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #350 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/350/)
          svn merge -c 1375164 from trunk FIXES: MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375185)

          Result = SUCCESS
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375185
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #350 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/350/ ) svn merge -c 1375164 from trunk FIXES: MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375185) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375185 Files : /hadoop/common/branches/branch-0.23/hadoop-yarn-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/branches/branch-0.23/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1141/)
          MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164)

          Result = FAILURE
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1141 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1141/ ) MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1173/)
          MAPREDUCE-4323. NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164)

          Result = FAILURE
          jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164
          Files :

          • /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java
          • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1173/ ) MAPREDUCE-4323 . NM leaks filesystems (Jason Lowe via jeagles) (Revision 1375164) Result = FAILURE jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1375164 Files : /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestContainerLocalizer.java /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java
          Arun C Murthy made changes -
          Project Hadoop Map/Reduce [ 12310941 ] Hadoop YARN [ 12313722 ]
          Key MAPREDUCE-4323 YARN-58
          Affects Version/s 0.23.0 [ 12315570 ]
          Affects Version/s 0.24.0 [ 12317654 ]
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Target Version/s 0.24.0, 0.23.3, 2.2.0-alpha [ 12317654, 12320060, 12322471 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Fix Version/s 0.23.3 [ 12322841 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Fix Version/s 2.1.0-alpha [ 12321442 ]
          Component/s nodemanager [ 12319323 ]
          Component/s nodemanager [ 12315341 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322548 ]
          Fix Version/s 2.1.0-alpha [ 12322547 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          1d 1h 35m 2 Jason Lowe 15/Aug/12 21:14
          Open Open Patch Available Patch Available
          68d 3h 29m 3 Jason Lowe 15/Aug/12 21:33
          Patch Available Patch Available Resolved Resolved
          4d 22h 1 Jonathan Eagles 20/Aug/12 19:33
          Resolved Resolved Closed Closed
          51d 23h 14m 1 Arun C Murthy 11/Oct/12 18:48

            People

            • Assignee:
              Jason Lowe
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development