Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6624

Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 3.0, Impala 2.12.0
    • Fix Version/s: Impala 3.0, Impala 2.12.0
    • Component/s: Distributed Exec
    • Labels:
      None

      Description

      During stress testing in a secure 140 node cluster, Impalad ran into the following errors. This is supposed to be fixed in KUDU-2218. The fix for KUDU-2218 has already been cherry-picked to Impala code base at this commit and the build should have this commit. It's unclear if Impala may be missing other commits or the issue in KUDU-2218 is not completely fixed.

      Assigning to Sailesh Mukil to lead the investigation. Please feel free to reassign to me if you are swamped Sailesh.

      W0307 03:31:04.512100 158268 connection.cc:659] client connection to 10.17.221.47:27000 send error: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      W0307 03:31:04.524086 158268 connection.cc:153] Shutting down client connection to 10.17.221.47:27000 with pending inbound data (11/16 bytes received, last active 0 ns ago, status=Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad writ
      e retry:s3_pkt.c:874)
      E0307 03:31:04.535635 123156 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      W0307 03:31:04.536145 158268 connection.cc:190] Error closing socket: Network error: TlsSocket::Close: Success
      E0307 03:31:04.584370 140087 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      I0307 03:31:04.697773 158412 rpcz_store.cc:255] Call impala.DataStreamService.TransmitData from 10.17.221.15:33716 (request call id 509466) took 125221ms. Request Metrics: {}
      E0307 03:31:04.707012 64577 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:04.767437 123164 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:04.786669 117111 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:04.792443 118554 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:04.912823 108328 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.221110 64484 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.228492 167981 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.232076 117126 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.412305 69586 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.427347 64667 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.430274 65641 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.436692 66206 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.437369 116174 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.515347 115108 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.545945 66826 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.752233 68861 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.793612 117106 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.799756 102340 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.801004 107447 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.832300 138449 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:05.881510 66751 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:06.278340 138373 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:06.278870 116990 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:06.280494 136840 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:06.490084 66207 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:07.231269 67227 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      E0307 03:31:07.339190 66752 krpc-data-stream-sender.cc:335] channel send to 10.17.221.47:27000 failed: TransmitData() to 10.17.221.47:27000 failed: Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c:874
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                kwho Michael Ho
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: