Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22436

twitter datastream connector hangs, CRLF expected at end of chunk

    XMLWordPrintableJSON

    Details

      Description

      Sorry for selecting Connectors/Common, Twitter didn't show up in the Connectors / "X" list, I'm not sure why that is.

       

      in addition to item FLINK-22435 I found this exception as well when the recent version (1.12) twitter connector just hangs. 

       

      basically I am following the example here:

      https://github.com/apache/flink/blob/master/flink-examples/flink-examples-streaming/src/main/java/org/apache/flink/streaming/examples/twitter/TwitterExample.java

       

      which just seems to run indefinitely but not have any data throughput/results or exceptions.

       

      task manager stdout (not logs) looks something like this:

       

      WARNING: Please consider reporting this to the maintainers of org.apache.flink.shaded.akka.org.jboss.netty.util.internal.ByteBufferUtilWARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operationsWARNING: All illegal access operations will be denied in a future releaseMay 01, 2021 8:37:04 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at java.base/java.lang.Thread.run(Thread.java:832)
      May 01, 2021 8:39:06 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at java.base/java.lang.Thread.run(Thread.java:832)
      May 01, 2021 8:49:48 AM org.apache.flink.twitter.shaded.com.google.common.io.Closeables closeWARNING: IOException thrown while closing Closeable.org.apache.http.MalformedChunkCodingException: CRLF expected at end of chunk    at org.apache.http.impl.io.ChunkedInputStream.getChunkSize(ChunkedInputStream.java:253)    at org.apache.http.impl.io.ChunkedInputStream.nextChunk(ChunkedInputStream.java:225)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:184)    at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:213)    at org.apache.http.impl.io.ChunkedInputStream.close(ChunkedInputStream.java:315)    at org.apache.http.conn.BasicManagedEntity.streamClosed(BasicManagedEntity.java:166)    at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)    at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)    at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)    at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)    at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:94)    at org.apache.flink.twitter.shaded.com.google.common.io.Closeables.close(Closeables.java:77)    at com.twitter.hbc.httpclient.Connection.close(Connection.java:64)    at com.twitter.hbc.httpclient.ClientBase.run(ClientBase.java:148)    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)    at java.base/java.lang.Thread.run(Thread.java:832)
      

       

      So the part that stood out to me was "CRLF expected at end of chunk", then stacktrace.

      I'm not sure which is the cause of twitter connector not working between this and FLINK-22435 but I'm reporting both just in case.

       

      I can provide more info if needed but my setup is really basic/simple.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              perezjasonr Jason Perez
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: