[IMPALA-8577] Crash during OpenSSLSocket.read

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Epic Color: ghx-label-5
• Assignee: Sahil Takiar
• Reporter: David Rorke
• Created: 22/May/19 20:12
• Updated: 28/Sep/20 18:11
• Resolved: 28/Sep/20 18:11

Description

Impalad crashed while running a TPC-DS 10 TB run against S3. Excerpt from the stack trace (hs_err log file attached with more complete stack):

Stack: [0x00007f3d095bc000,0x00007f3d09dbc000],  sp=0x00007f3d09db9050,  free space=8180k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [impalad+0x2528a33]  tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)+0x133
C  [impalad+0x2528e0f]  tcmalloc::ThreadCache::Scavenge()+0x3f
C  [impalad+0x266468a]  operator delete(void*)+0x32a
C  [libcrypto.so.10+0x6e70d]  CRYPTO_free+0x1d
J 5709  org.wildfly.openssl.SSLImpl.freeBIO0(J)V (0 bytes) @ 0x00007f3d4dadf9f9 [0x00007f3d4dadf940+0xb9]
J 5708 C1 org.wildfly.openssl.SSLImpl.freeBIO(J)V (5 bytes) @ 0x00007f3d4dfd0dfc [0x00007f3d4dfd0d80+0x7c]
J 5158 C1 org.wildfly.openssl.OpenSSLEngine.shutdown()V (78 bytes) @ 0x00007f3d4de4fe2c [0x00007f3d4de4f720+0x70c]
J 5758 C1 org.wildfly.openssl.OpenSSLEngine.closeInbound()V (51 bytes) @ 0x00007f3d4de419cc [0x00007f3d4de417c0+0x20c]
J 2994 C2 org.wildfly.openssl.OpenSSLEngine.unwrap(Ljava/nio/ByteBuffer;[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (892 bytes) @ 0x00007f3d4db8da34 [0x00007f3d4db8c900+0x1134]
J 3161 C2 org.wildfly.openssl.OpenSSLSocket.read([BII)I (810 bytes) @ 0x00007f3d4dd64cb0 [0x00007f3d4dd646c0+0x5f0]
J 5090 C2 com.amazonaws.thirdparty.apache.http.impl.io.SessionInputBufferImpl.fillBuffer()I (97 bytes) @ 0x00007f3d4ddd9ee0 [0x00007f3d4ddd9e40+0xa0]
J 5846 C1 com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.fillInputBuffer(I)I (48 bytes) @ 0x00007f3d4d7acb24 [0x00007f3d4d7ac7a0+0x384]
J 5845 C1 com.amazonaws.thirdparty.apache.http.impl.BHttpConnectionBase.isStale()Z (31 bytes) @ 0x00007f3d4d7ad49c [0x00007f3d4d7ad220+0x27c]

The crash may not be easy to reproduce. I've run this test multiple times and only crashed once. I have a core file if needed.

Issue Links

• is related to: HADOOP-16346 Stabilize S3A OpenSSL support
• is related to: HADOOP-16050 S3A SSL connections should use OpenSSL