[IMPALA-2280] webserver-test be test ASAN failure: heap-use-after-free

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.3.0
• Labels: broken-build crash
• Target Version: Impala 2.3.0
• Assignee: Henry Robinson
• Reporter: Daniel Hecht
• Created: 01/Sep/15 23:49
• Updated: 04/Sep/15 18:18
• Resolved: 04/Sep/15 18:18

Description

webserver-test failed in ASAN:
http://sandbox.jenkins.cloudera.com/view/Impala/view/Nightly-Builds/job/impala-master-cdh5-trunk-ASAN/116

From test log:

[ RUN      ] Webserver.SslWithPrivateKeyPasswordTest
=================================================================
==9143==ERROR: AddressSanitizer: heap-use-after-free on address 0x60400003ac28 at pc 0xf203e6 bp 0x7fff7348b9c0 sp 0x7fff7348b998
READ of size 9 at 0x60400003ac28 thread T0
    #0 0xf203e5 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf203e5)
    #1 0x10272fd (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x10272fd)
    #2 0x103414b (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x103414b)
    #3 0x10159d3 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x10159d3)
    #4 0xf3733c (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf3733c)
    #5 0x23deb67 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23deb67)
    #6 0x23d9cd7 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23d9cd7)
    #7 0x23c5eff (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c5eff)
    #8 0x23c671d (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c671d)
    #9 0x23c6d26 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c6d26)
    #10 0x23cbe66 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23cbe66)
    #11 0x23dfd15 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23dfd15)
    #12 0x23dab37 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23dab37)
    #13 0x23caa44 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23caa44)
    #14 0xf3a51c (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf3a51c)
    #15 0x30bb21ed5c (/lib64/libc-2.12.so+0x1ed5c)
    #16 0xf305cc (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf305cc)
0x60400003ac28 is located 24 bytes inside of 34-byte region [0x60400003ac10,0x60400003ac32)
freed by thread T0 here:
    #0 0xf22ad4 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf22ad4)
    #1 0x38d0e9d4c8 (/usr/lib64/libstdc++.so.6.0.13+0x9d4c8)
    #2 0xf3733c (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf3733c)
    #3 0x23deb67 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23deb67)
    #4 0x23d9cd7 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23d9cd7)
    #5 0x23c5eff (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c5eff)
    #6 0x23c671d (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c671d)
    #7 0x23c6d26 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23c6d26)
    #8 0x23cbe66 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23cbe66)
    #9 0x23dfd15 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23dfd15)
    #10 0x23dab37 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23dab37)
    #11 0x23caa44 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0x23caa44)
    #12 0xf3a51c (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf3a51c)
    #13 0x30bb21ed5c (/lib64/libc-2.12.so+0x1ed5c)
previously allocated by thread T0 here:
    #0 0xf22914 (/data/2/jenkins/workspace/impala-master-cdh5-trunk-ASAN/repos/Impala/be/build/debug/util/webserver-test+0xf22914)
    #1 0x38d0e9c3c8 (/usr/lib64/libstdc++.so.6.0.13+0x9c3c8)
Shadow bytes around the buggy address:
  0x0c087ffff530: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087ffff540: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087ffff550: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087ffff560: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c087ffff570: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c087ffff580: fa fa fd fd fd[fd]fd fa fa fa 00 00 00 00 07 fa
  0x0c087ffff590: fa fa 00 00 00 00 03 fa fa fa 00 00 00 00 06 fa
  0x0c087ffff5a0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c087ffff5b0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c087ffff5c0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
  0x0c087ffff5d0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap right redzone:    fb
  Freed heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==9143==ABORTING
<end of output>
Test time =   0.21 sec
----------------------------------------------------------
Test Failed.
"webserver-test" end time: Sep 01 12:28 PDT
"webserver-test" time elapsed: 00:00:00
----------------------------------------------------------

Here's the decoded callstack:

(gdb) x /i 0xf203e5
   0xf203e5 <strlen+277>:       add    %cl,-0x73(%rax)
(gdb) x /i 0x10272fd
   0x10272fd <sq_strdup+23>:    decl   -0x77(%rax)
(gdb) x /i 0x103414b
   0x103414b <sq_start+493>:    decl   -0x75(%rax)
(gdb) x /i 0x10159d3
   0x10159d3 <impala::Webserver::Start()+2595>: add    %cl,0x64(%rbx,%rcx,4)
(gdb) x /i 0xf3733c
   0xf3733c <Webserver_SslWithPrivateKeyPasswordTest_Test::TestBody()+908>:     add    %cl,-0x73(%rax)

henryr, can you take a look?

Issue Links

• is related to: IMPALA-2051 Allow Squeasel to accept certificates and private keys from separate files