[HDFS-7191] WebHDFS prematurely closes connections under high concurrent loads

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Haohui Mai
• Created: 04/Oct/14 00:29
• Updated: 14/Mar/15 00:31
• Resolved: 14/Mar/15 00:31

Description

We're seeing the DN prematurely closes APPEND connections:

{noformat] 2014-09-22 23:53:12,721 WARN org.apache.hadoop.hdfs.web.resources.ExceptionHandler: INTERNAL_SERVER_ERROR java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:59) at org.mortbay.io.nio.SelectChannelEndPoint.updateKey(SelectChannelEndPoint.java:325) at org.mortbay.io.nio.SelectChannelEndPoint.blockReadable(SelectChannelEndPoint.java:242) at org.mortbay.jetty.HttpParser$Input.blockForContent(HttpParser.java:1169) at org.mortbay.jetty.HttpParser$Input.read(HttpParser.java:1122) at java.io.InputStream.read(InputStream.java:85) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:84) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods.put(DatanodeWebHdfsMethods.java:239) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods.access$000(DatanodeWebHdfsMethods.java:87) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$1.run(DatanodeWebHdfsMethods.java:205) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods$1.run(DatanodeWebHdfsMethods.java:202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.hdfs.server.datanode.web.resources.DatanodeWebHdfsMethods.put(DatanodeWebHdfsMethods.java:202) {noformat}