Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3359

expr-test: StartWithEphemeralPorts() crashes if impalad fails to start

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Duplicate
    • Impala 2.5.0
    • None
    • Infrastructure
    • None

    Description

      If FindWithEphemeralPorts() returns the same port for several different services in StartWithEphemeralPorts(), StartWithClientServers() will crash when it calls delete impala.

      It might not be easy to fix the crash (as it's deep into Thrift), but maybe let's make it so that StartWithEphemeralPorts() can't try to use the same port for e.g. beeswax_port and hs2_port/

      Log from {[expr-test}}

      I0414 04:12:38.979552 16994 impala-server.cc:1837] Impala Beeswax Service listening on 62874
I0414 04:12:38.980504 16994 impala-server.cc:1859] Impala HiveServer2 Service listening on 52011
I0414 04:12:38.981437 16994 impala-server.cc:1879] ImpalaInternalService listening on 52011
I0414 04:12:38.981679 16994 thrift-server.cc:431] ThriftServer 'backend' started on port: 52011 <----- Duplicate ephemeral port
E0414 04:12:38.983886 17166 thrift-server.cc:176] ThriftServer 'hiveserver2-frontend' (on port: 52011) exited due to TException: Could not bind: Transport endpoint is not connected
E0414 04:12:38.984000 16994 thrift-server.cc:165] ThriftServer 'hiveserver2-frontend' (on port: 52011) did not start correctly

      Stacktrace

      #0  0x00007ffff4d6ecc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff4d720d8 in __GI_abort () at abort.c:89
#2  0x00007ffff4d67b86 in __assert_fail_base (fmt=0x7ffff4eb8830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x27d06b8 "!ret",
    file=file@entry=0x27d05c8 "/data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp", line=line@entry=81,
    function=function@entry=0x27d2800 <boost::condition_variable::~condition_variable()::__PRETTY_FUNCTION__> "boost::condition_variable::~condition_variable()") at assert.c:92
#3  0x00007ffff4d67c32 in __GI___assert_fail (assertion=0x27d06b8 "!ret", file=0x27d05c8 "/data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp", line=81,
    function=0x27d2800 <boost::condition_variable::~condition_variable()::__PRETTY_FUNCTION__> "boost::condition_variable::~condition_variable()") at assert.c:101
#4  0x0000000001401b82 in boost::condition_variable::~condition_variable (this=0x6f8ff20, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/thread/pthread/condition_variable_fwd.hpp:81
#5  0x000000000156f235 in impala::ImpalaServer::~ImpalaServer (this=0x6f8fe00, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/service/impala-server.cc:706
#6  0x000000000156f3c2 in impala::ImpalaServer::~ImpalaServer (this=0x6f8fe00, __in_chrg=<optimized out>, __vtt_parm=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/service/impala-server.cc:706
#7  0x00000000015af1d2 in boost::checked_delete<impala::ImpalaServer> (x=0x6f8fe00) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#8  0x00000000015bbf44 in boost::detail::sp_counted_impl_p<impala::ImpalaServer>::dispose (this=0x83bfa80) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#9  0x000000000116226c in boost::detail::sp_counted_base::release (this=0x83bfa80) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#10 0x00000000011622fb in boost::detail::shared_count::~shared_count (this=0x83c9c00, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/shared_count.hpp:443
#11 0x0000000001587ee2 in boost::shared_ptr<impala::ImpalaServer>::~shared_ptr (this=0x83c9bf8, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/shared_ptr.hpp:323
#12 0x00000000015af6a4 in impala::ImpalaInternalService::~ImpalaInternalService (this=0x83c9bf0, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/service/impala-internal-service.h:29
#13 0x00000000015af6e0 in impala::ImpalaInternalService::~ImpalaInternalService (this=0x83c9bf0, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/service/impala-internal-service.h:29
#14 0x00000000015af718 in boost::checked_delete<impala::ImpalaInternalService> (x=0x83c9bf0) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#15 0x00000000015bbdfa in boost::detail::sp_counted_impl_p<impala::ImpalaInternalService>::dispose (this=0x83bfce0) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#16 0x000000000116226c in boost::detail::sp_counted_base::release (this=0x83bfce0) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#17 0x00000000011622fb in boost::detail::shared_count::~shared_count (this=0x83ca380, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/shared_count.hpp:443
#18 0x000000000158248c in boost::shared_ptr<impala::ImpalaInternalServiceIf>::~shared_ptr (this=0x83ca378, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/shared_ptr.hpp:323
#19 0x00000000018bdf16 in impala::ImpalaInternalServiceProcessor::~ImpalaInternalServiceProcessor (this=0x83ca360, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.h:803
#20 0x00000000018bdf52 in impala::ImpalaInternalServiceProcessor::~ImpalaInternalServiceProcessor (this=0x83ca360, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.h:803
#21 0x00000000015af780 in boost::checked_delete<impala::ImpalaInternalServiceProcessor> (x=0x83ca360) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#22 0x00000000015bbdb8 in boost::detail::sp_counted_impl_p<impala::ImpalaInternalServiceProcessor>::dispose (this=0x83bfd00) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#23 0x000000000116226c in boost::detail::sp_counted_base::release (this=0x83bfd00) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#24 0x00000000011622fb in boost::detail::shared_count::~shared_count (this=0x83bfe70, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/shared_count.hpp:443
#25 0x000000000142e92c in boost::shared_ptr<apache::thrift::TProcessor>::~shared_ptr (this=0x83bfe68, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/shared_ptr.hpp:323
#26 0x000000000195d280 in apache::thrift::TSingletonProcessorFactory::~TSingletonProcessorFactory (this=0x83bfe60, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/thrift-0.9.0-p2/include/thrift/TProcessor.h:218
#27 0x000000000195d2bc in apache::thrift::TSingletonProcessorFactory::~TSingletonProcessorFactory (this=0x83bfe60, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/thrift-0.9.0-p2/include/thrift/TProcessor.h:218
#28 0x000000000195d2f4 in boost::checked_delete<apache::thrift::TSingletonProcessorFactory> (x=0x83bfe60) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#29 0x000000000195dacc in boost::detail::sp_counted_impl_p<apache::thrift::TSingletonProcessorFactory>::dispose (this=0x83bfe80) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#30 0x000000000238657a in apache::thrift::server::TThreadedServer::~TThreadedServer() ()
#31 0x00000000023866b9 in apache::thrift::server::TThreadedServer::~TThreadedServer() ()
#32 0x000000000143a0ba in boost::checked_delete<apache::thrift::server::TServer> (x=0x7e94700) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#33 0x0000000001438465 in boost::scoped_ptr<apache::thrift::server::TServer>::~scoped_ptr (this=0x86eea40, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:82
#34 0x000000000143696a in impala::ThriftServer::~ThriftServer (this=0x86eea00, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/rpc/thrift-server.h:40
#35 0x00000000014369da in boost::checked_delete<impala::ThriftServer> (x=0x86eea00) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/core/checked_delete.hpp:34
#36 0x00000000014348c3 in boost::scoped_ptr<impala::ThriftServer>::~scoped_ptr (this=0x77c4ee8, __in_chrg=<optimized out>) at /data/henry/src/cloudera/impala-toolchain/boost-1.57.0/include/boost/smart_ptr/scoped_ptr.hpp:82
#37 0x0000000001667fd8 in impala::InProcessImpalaServer::~InProcessImpalaServer (this=0x77c4ec0, __in_chrg=<optimized out>) at /data/henry/src/cloudera/Impala/be/src/testutil/in-process-servers.h:39
#38 0x000000000166643b in impala::InProcessImpalaServer::StartWithEphemeralPorts (statestore_host=..., statestore_port=0) at /data/henry/src/cloudera/Impala/be/src/testutil/in-process-servers.cc:68
#39 0x00000000011602c1 in main (argc=1, argv=0x7fffffff2258) at /data/henry/src/cloudera/Impala/be/src/exprs/expr-test.cc:5779

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5499 session-expiry-test failed because of conflicting ephemeral ports

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              sailesh Sailesh Mukil
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: