Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4801

ShuffleHandler can generate large logs due to prematurely closed channels

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 2.0.1-alpha
    • Fix Version/s: 3.0.0, 2.0.3-alpha, 0.23.5
    • Component/s: None
    • Labels:
      None

      Description

      We ran into an instance where many nodes on a cluster ran out of disk space because the nodemanager logs were huge. Examining the logs showed many, many shuffle errors due to either ClosedChannelException or IOException from "Connection reset by peer" or "Broken pipe".

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/)
        MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131)

        Result = FAILURE
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1259 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1259/ ) MAPREDUCE-4801 . ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/)
        MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1228 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1228/ ) MAPREDUCE-4801 . ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #437 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/437/)
        svn merge -c 1410131 FIXES: MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410133)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410133
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #437 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/437/ ) svn merge -c 1410131 FIXES: MAPREDUCE-4801 . ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410133) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410133 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/)
        MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #38 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/38/ ) MAPREDUCE-4801 . ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3030/)
        MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3030 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3030/ ) MAPREDUCE-4801 . ShuffleHandler can generate large logs due to prematurely closed channels (jlowe via bobby) (Revision 1410131) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1410131 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
        Hide
        Robert Joseph Evans added a comment -

        Thanks Jason,

        I put this into trunk, branch-2, branch-0.23, and branch-0.23.5

        Show
        Robert Joseph Evans added a comment - Thanks Jason, I put this into trunk, branch-2, branch-0.23, and branch-0.23.5
        Hide
        Robert Joseph Evans added a comment -

        The patch looks good. +1 I'll check it in.

        Show
        Robert Joseph Evans added a comment - The patch looks good. +1 I'll check it in.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12553716/MAPREDUCE-4801.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3035//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3035//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12553716/MAPREDUCE-4801.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3035//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3035//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Patch to ignore errors from closed connections and IOExceptions related to the client closing the connection early.

        Show
        Jason Lowe added a comment - Patch to ignore errors from closed connections and IOExceptions related to the client closing the connection early.
        Hide
        Jason Lowe added a comment -

        I believe this is caused by the behavior of reducers during the shuffle when they receive the shuffle header containing the size and then the MergeManager decides that's too much data to receive right now. In that case it doesn't read the subsequent map data and just closes the socket. That leads to IOExceptions when the ShuffleHandler tries to push the data to the closed socket.

        Show
        Jason Lowe added a comment - I believe this is caused by the behavior of reducers during the shuffle when they receive the shuffle header containing the size and then the MergeManager decides that's too much data to receive right now. In that case it doesn't read the subsequent map data and just closes the socket. That leads to IOExceptions when the ShuffleHandler tries to push the data to the closed socket.
        Hide
        Jason Lowe added a comment -

        Some sample exceptions:

        2012-11-15 12:47:02,365 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: 
        java.io.IOException: Connection reset by peer
                at sun.nio.ch.FileDispatcher.read0(Native Method)
                at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
                at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
                at sun.nio.ch.IOUtil.read(IOUtil.java:204)
                at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:321)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        2012-11-15 12:47:02,366 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: 0x01901fc1, /xx.xx.xx.xx:xx => /xx.xx.xx.xx:xx] EXCEPTION: java.io.IOException: Connection reset by peer
        2012-11-15 12:47:02,366 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: 
        java.nio.channels.ClosedChannelException
                at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:616)
                at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:592)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:355)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        ...
        2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: 
        java.io.IOException: Broken pipe
                at sun.nio.ch.FileDispatcher.write0(Native Method)
                at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
                at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
                at sun.nio.ch.IOUtil.write(IOUtil.java:56)
                at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
                at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239)
                at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:469)
                at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387)
                at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
                at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
                at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:68)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:255)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:124)
                at org.jboss.netty.channel.Channels.write(Channels.java:611)
                at org.jboss.netty.channel.Channels.write(Channels.java:578)
                at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
                at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:477)
                at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:397)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:148)
                at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:116)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: 0x01fd225b, /xx.xx.xx.xx:xx => /xx.xx.xx.xx:xx] EXCEPTION: java.io.IOException: Broken pipe
        2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: 
        java.nio.channels.ClosedChannelException
                at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636)
                at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:592)
                at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:512)
                at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387)
                at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137)
                at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76)
                at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:68)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:255)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:124)
                at org.jboss.netty.channel.Channels.write(Channels.java:611)
                at org.jboss.netty.channel.Channels.write(Channels.java:578)
                at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259)
                at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:477)
                at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:397)
                at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:148)
                at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:116)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506)
                at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
                at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
                at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
                at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
                at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
                at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                at java.lang.Thread.run(Thread.java:619)
        
        Show
        Jason Lowe added a comment - Some sample exceptions: 2012-11-15 12:47:02,365 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237) at sun.nio.ch.IOUtil.read(IOUtil.java:204) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:321) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-11-15 12:47:02,366 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: 0x01901fc1, /xx.xx.xx.xx:xx => /xx.xx.xx.xx:xx] EXCEPTION: java.io.IOException: Connection reset by peer 2012-11-15 12:47:02,366 [New I/O server worker #1-14] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:616) at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:592) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:355) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) ... 2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100) at sun.nio.ch.IOUtil.write(IOUtil.java:56) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.jboss.netty.channel.socket.nio.SocketSendBufferPool$PooledSendBuffer.transferTo(SocketSendBufferPool.java:239) at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:469) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:68) at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:255) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:124) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:477) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:397) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:148) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:116) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error [id: 0x01fd225b, /xx.xx.xx.xx:xx => /xx.xx.xx.xx:xx] EXCEPTION: java.io.IOException: Broken pipe 2012-11-15 12:47:02,367 [New I/O server worker #1-15] ERROR org.apache.hadoop.mapred.ShuffleHandler: Shuffle error: java.nio.channels.ClosedChannelException at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:636) at org.jboss.netty.channel.socket.nio.NioWorker.close(NioWorker.java:592) at org.jboss.netty.channel.socket.nio.NioWorker.write0(NioWorker.java:512) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:387) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:137) at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:76) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:68) at org.jboss.netty.handler.stream.ChunkedWriteHandler.flush(ChunkedWriteHandler.java:255) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleDownstream(ChunkedWriteHandler.java:124) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:259) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.sendMapOutput(ShuffleHandler.java:477) at org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:397) at org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:148) at org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:116) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:302) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.unfoldAndfireMessageReceived(ReplayingDecoder.java:522) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:506) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:443) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349) at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619)

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development