Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2422

Subprocess hang at startup in TSAN builds

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.8.0
    • Component/s: test
    • Labels:
      None

      Description

      I'm seeing a small percentage of test timeouts caused by a hang in subprocess starting. I managed to catch one in the act and attach to the subprocess in gdb. It's hung before 'exec' here:

      * 1    Thread 0x7f1feebdb440 (LWP 9453) "disk_failure-it" __sanitizer::internal_sched_yield () at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:414
      (gdb) bt
      #0  __sanitizer::internal_sched_yield () at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cc:414
      #1  0x00000000004c39db in Do (this=<synthetic pointer>) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_mutex.cc:195
      #2  __tsan::Mutex::Lock (this=this@entry=0x600001000000) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_mutex.cc:235
      #3  0x00000000004c72ed in GenericScopedLock (mu=0x600001000000, this=<synthetic pointer>) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_mutex.h:189
      #4  TraceSwitch (thr=<optimized out>) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl.cc:552
      #5  __tsan::__tsan_trace_switch () at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl.cc:581
      #6  0x00000000004da3df in __tsan_trace_switch_thunk () at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl_amd64.S:53
      #7  0x00000000004cfabc in TraceAddEvent (thr=<optimized out>, addr=0, typ=__tsan::EventTypeFuncExit, fs=...) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl.h:845
      #8  FuncExit (thr=<optimized out>) at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_rtl.cc:997
      #9  __tsan_func_exit () at /data/1/todd/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interface_inl.h:108
      #10 0x00007f1fe06a3ee5 in safe_strtou32_base (str=<optimized out>, value=<optimized out>, base=<optimized out>) at ../../src/kudu/gutil/strings/numbers.cc:717
      #11 0x00007f1fe06a4426 in safe_strtou32 (str=0xfffffffffffc04c0 <error: Cannot access memory at address 0xfffffffffffc04c0>, value=0x1fffff) at ../../src/kudu/gutil/strings/numbers.cc:773
      #12 0x00007f1fe288e42f in kudu::(anonymous namespace)::CloseNonStandardFDs (fd_dir=0x7bb400000000) at ../../src/kudu/util/subprocess.cc:144
      #13 0x00007f1fe288d3b4 in kudu::Subprocess::Start (this=<optimized out>) at ../../src/kudu/util/subprocess.cc:418
      #14 0x00007f1fee0ddcf0 in kudu::cluster::ExternalDaemon::StartProcess (this=<optimized out>, user_flags=...) at ../../src/kudu/mini-cluster/external_mini_cluster.cc:821
      

        Attachments

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: