Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2765

tsan failure in ToolTest.TestLoadgenAutoFlushBackgroundRandom

    XMLWordPrintableJSON

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: n/a
    • Component/s: None
    • Labels:
      None

      Description

      ToolTest.TestLoadgenAutoFlushBackgroundRandom failed with the following error in tsan

      ==================
      WARNING: ThreadSanitizer: destroy of a locked mutex (pid=1076)
          #0 pthread_rwlock_destroy /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1313 (kudu+0x4b4474)
          #1 glog_internal_namespace_::Mutex::~Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 (libglog.so.0+0x16488)
          #2 cxa_at_exit_wrapper(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:386 (kudu+0x484803)
      
        and:
          #0 operator delete(void*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:119 (kudu+0x523cf1)
          #1 google::protobuf::FieldDescriptorProto::~FieldDescriptorProto() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor.pb.cc:4916:47 (libprotobuf.so.14+0x19c3b1)
          #2 google::protobuf::internal::GenericTypeHandler<google::protobuf::FieldDescriptorProto>::Delete(google::protobuf::FieldDescriptorProto*, google::protobuf::Arena*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:615:7 (libprotobuf.so.14+0x1973b1)
          #3 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<google::protobuf::FieldDescriptorProto>::TypeHandler>() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:1429 (libprotobuf.so.14+0x1973b1)
          #4 google::protobuf::RepeatedPtrField<google::protobuf::FieldDescriptorProto>::~RepeatedPtrField() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:1892 (libprotobuf.so.14+0x1973b1)
          #5 google::protobuf::DescriptorProto::~DescriptorProto() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor.pb.cc:3528 (libprotobuf.so.14+0x1973b1)
          #6 google::protobuf::DescriptorProto::~DescriptorProto() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor.pb.cc:3525:37 (libprotobuf.so.14+0x197519)
          #7 google::protobuf::internal::GenericTypeHandler<google::protobuf::DescriptorProto>::Delete(google::protobuf::DescriptorProto*, google::protobuf::Arena*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:615:7 (libprotobuf.so.14+0x18e8c1)
          #8 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<google::protobuf::DescriptorProto>::TypeHandler>() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:1429 (libprotobuf.so.14+0x18e8c1)
          #9 google::protobuf::RepeatedPtrField<google::protobuf::DescriptorProto>::~RepeatedPtrField() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/repeated_field.h:1892 (libprotobuf.so.14+0x18e8c1)
          #10 google::protobuf::FileDescriptorProto::~FileDescriptorProto() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor.pb.cc:1426 (libprotobuf.so.14+0x18e8c1)
          #11 google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor_database.cc:322:1 (libprotobuf.so.14+0x182dcd)
          #12 google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/descriptor.cc:1315:3 (libprotobuf.so.14+0x13b705)
          #13 google::protobuf::protobuf_google_2fprotobuf_2ftype_2eproto::(anonymous namespace)::AddDescriptorsImpl() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/type.pb.cc:240:3 (libprotobuf.so.14+0x237c10)
          #14 google::protobuf::internal::FunctionClosure0::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/callback.h:129:5 (libprotobuf.so.14+0xd330b)
          #15 google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/once.cc:83:14 (libprotobuf.so.14+0xd5d6a)
          #16 google::protobuf::GoogleOnceInit(long*, void (*)()) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/once.h:128:5 (libprotobuf.so.14+0x237b66)
          #17 google::protobuf::protobuf_google_2fprotobuf_2ftype_2eproto::AddDescriptors() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/type.pb.cc:251 (libprotobuf.so.14+0x237b66)
          #18 google::protobuf::protobuf_google_2fprotobuf_2fapi_2eproto::(anonymous namespace)::AddDescriptorsImpl() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/api.pb.cc:179:3 (libprotobuf.so.14+0x12ca1d)
          #19 google::protobuf::internal::FunctionClosure0::Run() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/callback.h:129:5 (libprotobuf.so.14+0xd330b)
          #20 google::protobuf::GoogleOnceInitImpl(long*, google::protobuf::Closure*) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/once.cc:83:14 (libprotobuf.so.14+0xd5d6a)
          #21 google::protobuf::GoogleOnceInit(long*, void (*)()) /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/stubs/once.h:128:5 (libprotobuf.so.14+0xcf0a6)
          #22 google::protobuf::protobuf_google_2fprotobuf_2fapi_2eproto::AddDescriptors() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/api.pb.cc:185 (libprotobuf.so.14+0xcf0a6)
          #23 google::protobuf::protobuf_google_2fprotobuf_2fapi_2eproto::StaticDescriptorInitializer::StaticDescriptorInitializer() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/api.pb.cc:190 (libprotobuf.so.14+0xcf0a6)
          #24 __cxx_global_var_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/api.pb.cc:192 (libprotobuf.so.14+0xcf0a6)
          #25 _GLOBAL__sub_I_api.pb.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/protobuf-3.4.1/src/google/protobuf/api.pb.cc (libprotobuf.so.14+0xcf0a6)
          #26 call_init /build/eglibc-SvCtMH/eglibc-2.19/elf/dl-init.c:78 (ld-linux-x86-64.so.2+0x102d9)
      
        Location is global 'google::vmodule_lock' of size 64 at 0x7efc2674e878 (libglog.so.0+0x00000023c878)
      
        Mutex M132 (0x7efc2674e878) created at:
          #0 pthread_rwlock_init /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/llvm-6.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1304 (kudu+0x4b46e4)
          #1 glog_internal_namespace_::Mutex::Mutex() /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:247:19 (libglog.so.0+0xcca6)
          #2 __cxx_global_var_init.15 /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/vlog_is_on.cc:120 (libglog.so.0+0xcca6)
          #3 _GLOBAL__sub_I_vlog_is_on.cc /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/vlog_is_on.cc (libglog.so.0+0xcca6)
          #4 call_init /build/eglibc-SvCtMH/eglibc-2.19/elf/dl-init.c:78 (ld-linux-x86-64.so.2+0x102d9)
      
      SUMMARY: ThreadSanitizer: destroy of a locked mutex /data/somelongdirectorytoavoidrpathissues/src/kudu/thirdparty/src/glog-0.3.5/src/base/mutex.h:249:30 in glog_internal_namespace_::Mutex::~Mutex()
      ==================
      ThreadSanitizer: reported 1 warnings
      /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/kudu-tool-test.cc:1996: Failure
      Failed
      Bad status: Runtime error: /tmp/dist-test-taskvH0aH1/build/tsan/bin/kudu: process exited with non-zero status 66
      /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/tools/kudu-tool-test.cc:2033: Failure
      Expected: RunLoadgen(5, { "--buffer_flush_watermark_pct=0.125", "--buffer_size_bytes=65536", "--buffers_num=8", "--num_rows_per_thread=16", "--num_threads=1", "--run_scan", "--string_len=8", "--use_random", }, "bench_auto_flush_background_random") doesn't generate new fatal failures in the current thread.
        Actual: it does.
      

        Attachments

        1. kudu-tool-test.0.txt
          907 kB
          Hao Hao

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hahao Hao Hao
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: