Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.6.1, 1.7.0
Description
Running the test RestServerEndpointITCase#testRequestInterleaving in a loop may trigger the HTTP server to send a response with Content-Type: application/octet-stream, which causes the test to fail. The expected Content-Type is application/json. Note that the REST handler used for testing, can only return json responses. The failure can likely be triggered for other tests inside RestServerEndpointITCase as well. The behavior has not been observed on Linux so far.
To run the test in a loop, apply the git patch in the attachment, and execute the following command:
mvn clean integration-test -pl flink-runtime -am -Dtest=RestServerEndpointITCase -Dfast -DfailIfNoTests=false -Dsurefire.skipAfterFailureCount=1 -Dlog.dir=/path/to/log-dir -Dlog4j.configuration=file:///path/to/flink/tools/log4j-travis.properties
After a while you may see the following stacktrace in the test's log file:
================================================================================ 15:25:45,619 INFO org.apache.flink.runtime.rest.RestServerEndpointITCase - ================================================================================ Test testRequestInterleaving[5253](org.apache.flink.runtime.rest.RestServerEndpointITCase) is running. -------------------------------------------------------------------------------- 15:25:45,620 WARN org.apache.flink.runtime.rest.RestServerEndpointITCase$TestRestServerEndpoint - Upload directory /private/var/folders/43/ghpk0br14m99tnl9b86lwfsm0000gn/T/junit7127204190502548819/junit6384422818182277795/flink-web-upload does not exist, or has been deleted externally. Previously uploaded files are no longer available. 15:25:45,620 INFO org.apache.flink.runtime.rest.RestServerEndpointITCase$TestRestServerEndpoint - Created directory /private/var/folders/43/ghpk0br14m99tnl9b86lwfsm0000gn/T/junit7127204190502548819/junit6384422818182277795/flink-web-upload for file uploads. 15:25:45,620 INFO org.apache.flink.runtime.rest.RestClient - Rest client endpoint started. 15:25:45,620 INFO org.apache.flink.runtime.rest.RestServerEndpointITCase$TestRestServerEndpoint - Starting rest endpoint. 15:25:45,620 INFO org.apache.flink.runtime.rest.RestServerEndpointITCase$TestRestServerEndpoint - Rest endpoint listening at localhost:52841 15:25:45,631 ERROR org.apache.flink.runtime.rest.RestClient - Response was not valid JSON. org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParseException: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens at [Source: org.apache.flink.shaded.netty4.io.netty.buffer.ByteBufInputStream@7823f1cb; line: 1, column: 2] at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.base.ParserMinimalBase._throwInvalidSpace(ParserMinimalBase.java:472) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipWSOrEnd(UTF8StreamJsonParser.java:2891) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:707) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3792) at org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2272) at org.apache.flink.runtime.rest.RestClient$ClientHandler.readRawResponse(RestClient.java:505) at org.apache.flink.runtime.rest.RestClient$ClientHandler.channelRead0(RestClient.java:453) at org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.flink.shaded.netty4.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.flink.shaded.netty4.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.flink.shaded.netty4.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) at org.apache.flink.shaded.netty4.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) at org.apache.flink.shaded.netty4.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) at org.apache.flink.shaded.netty4.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) at org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884) at java.lang.Thread.run(Thread.java:748) 15:25:45,633 ERROR org.apache.flink.runtime.rest.RestClient - Unexpected plain-text response:
Find below the HTTP response as captured by Wireshark. The paypload is a single 0 byte.
0000 48 54 54 50 2f 31 2e 30 20 32 30 30 20 4f 4b 0d HTTP/1.0 200 OK. 0010 0a 43 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 61 .Content-type: a 0020 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 pplication/octet 0030 2d 73 74 72 65 61 6d 0d 0a 43 6f 6e 74 65 6e 74 -stream..Content 0040 2d 6c 65 6e 67 74 68 3a 20 31 0d 0a 0d 0a -length: 1....
Attachments
Attachments
Issue Links
- relates to
-
FLINK-10586 Running RestServerEndpointITCase may cause Kernel Panic
- Closed
- links to