Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6624

Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 3.0, Impala 2.12.0
    • Impala 3.0, Impala 2.12.0
    • Distributed Exec
    • None

    Description

      During stress testing in a secure 140 node cluster, Impalad ran into the following errors. This is supposed to be fixed in KUDU-2218. The fix for KUDU-2218 has already been cherry-picked to Impala code base at this commit and the build should have this commit. It's unclear if Impala may be missing other commits or the issue in KUDU-2218 is not completely fixed.

      Assigning to sailesh to lead the investigation. Please feel free to reassign to me if you are swamped Sailesh.

      W0307 03:31:04.512100 158268 connection.cc:659] client connection to 10.17.221.47:27000 send error: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
W0307 03:31:04.524086 158268 connection.cc:153] Shutting down client connection to 10.17.221.47:27000 with pending inbound data (11/16 bytes received, last active 0 ns ago, status=Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad writ
e retry:s3_pkt.c:874)
E0307 03:31:04.535635 123156 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
W0307 03:31:04.536145 158268 connection.cc:190] Error closing socket: Network error: TlsSocket::Close: Success
E0307 03:31:04.584370 140087 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
I0307 03:31:04.697773 158412 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from 10.17.221.15:33716 (request call id 509466) took 125221ms. Request Metrics: {}
E0307 03:31:04.707012 64577 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:04.767437 123164 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:04.786669 117111 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:04.792443 118554 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:04.912823 108328 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.221110 64484 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.228492 167981 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.232076 117126 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.412305 69586 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.427347 64667 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.430274 65641 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.436692 66206 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.437369 116174 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.515347 115108 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.545945 66826 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.752233 68861 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.793612 117106 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.799756 102340 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.801004 107447 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.832300 138449 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:05.881510 66751 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:06.278340 138373 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:06.278870 116990 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:06.280494 136840 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:06.490084 66207 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:07.231269 67227 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
E0307 03:31:07.339190 66752 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874

      Attachments

        Issue Links

          is blocked by

          Bug - A problem which impairs or prevents the functions of the product. KUDU-2334 OutboundTransfer::TransferStarted() isn't stateful enough with TLS socket

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              kwho Michael Ho
              kwho Michael Ho
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: