Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-12500

Dataflow SocketException (SSLException) error while trying to send message from Cloud Pub/Sub to BigQuery

    XMLWordPrintableJSON

    Details

      Description

      Issue: I found an issue of seeing SSLException when using beam version (>2.25) and could not upgrade with advanced version of beam in the dataflow job (streaming). Only versions below 2.25 is possible to work. Could not be sure if this issue is connected to [1]

      [1]https://bugs.openjdk.java.net/browse/JDK-8241372

      LogError message from worker: java.lang.RuntimeException: java.net.SocketException: Connection reset org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:932) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.insertAll(BigQueryServicesImpl.java:972) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.flushRows(StreamingWriteFn.java:195) org.apache.beam.sdk.io.gcp.bigquery.StreamingWriteFn.finishBundle(StreamingWriteFn.java:149) Caused by: java.net.SocketException: Connection reset java.net.SocketInputStream.read(SocketInputStream.java:210) java.net.SocketInputStream.read(SocketInputStream.java:141) sun.security.ssl.InputRecord.readFully(InputRecord.java:465) sun.security.ssl.InputRecord.read(InputRecord.java:503) sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940) sun.security.ssl.AppInputStream.read(AppInputStream.java:105) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read1(BufferedInputStream.java:286) java.io.BufferedInputStream.read(BufferedInputStream.java:345) sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552) sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609) sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696) java.io.FilterInputStream.read(FilterInputStream.java:133) sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3393) com.google.api.client.http.javanet.NetHttpResponse$SizeValidatingInputStream.read(NetHttpResponse.java:164) java.io.FilterInputStream.read(FilterInputStream.java:133) java.io.FilterInputStream.read(FilterInputStream.java:107) com.google.common.io.ByteStreams.exhaust(ByteStreams.java:273) com.google.api.client.http.ConsumingInputStream.close(ConsumingInputStream.java:40) java.util.zip.InflaterInputStream.close(InflaterInputStream.java:227) java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136) com.fasterxml.jackson.core.json.UTF8StreamJsonParser._closeInput(UTF8StreamJsonParser.java:254) com.fasterxml.jackson.core.base.ParserBase.close(ParserBase.java:369) com.google.api.client.json.jackson2.JacksonParser.close(JacksonParser.java:47) com.google.api.client.json.JsonParser.parse(JsonParser.java:363) com.google.api.client.json.JsonParser.parse(JsonParser.java:335) com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:79) com.google.api.client.json.JsonObjectParser.parseAndClose(JsonObjectParser.java:73) com.google.api.client.http.HttpResponse.parseAs(HttpResponse.java:451) com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:591) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$DatasetServiceImpl.lambda$insertAll$1(BigQueryServicesImpl.java:857) org.apache.beam.sdk.io.gcp.bigquery.BigQueryServicesImpl$BoundedExecutorService$SemaphoreCallable.call(BigQueryServicesImpl.java:1304) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhangsteven steven zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified