Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:

      Description

      Impala does not start under ASAN anymore. Our ASAN build is hitting this, and I can reproduce locally.

      Here's the stack:

      Stack: [0x00007f5d20999000,0x00007f5d2119a000],  sp=0x00007f5d211963d0,  free space=8180k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C  [impalad+0xc718bb]  base::subtle::NoBarrier_CompareAndSwap(int volatile*, int, int)+0x1b
      C  [impalad+0xc71875]  base::SpinLock::Lock()+0x15
      C  [impalad+0xf417d5]  impala::SimpleMetric<unsigned long, (impala::TMetricKind::type)0>::value()+0x95
      C  [impalad+0x126e494]  impala::SumGauge<unsigned long>::CalculateValue()+0x124
      C  [impalad+0xf41808]  impala::SimpleMetric<unsigned long, (impala::TMetricKind::type)0>::value()+0xc8
      C  [impalad+0x126ce93]  impala::SimpleMetric<unsigned long, (impala::TMetricKind::type)0>::ToLegacyJson(rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*)+0xb3
      C  [impalad+0x127093a]  impala::MetricGroup::CMCompatibleCallback(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*)+0x4ca
      C  [impalad+0x12750bd]  void boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::MetricGroup, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>, boost::_bi::list3<boost::_bi::value<impala::MetricGroup*>, boost::arg<1>, boost::arg<2> > >::operator()<std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*&)+0xad
      C  [impalad+0x1274e30]  boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::MetricGroup, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>, boost::_bi::list3<boost::_bi::value<impala::MetricGroup*>, boost::arg<1>, boost::arg<2> > >, void, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::invoke(boost::detail::function::function_buffer&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*)+0xb0
      C  [impalad+0x13083a6]  boost::function2<void, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*>::operator()(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, rapidjson::GenericDocument<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >*) const+0xc6
      C  [impalad+0x1306c6f]  impala::Webserver::RenderUrlWithTemplate(std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > > const&, impala::Webserver::UrlHandler const&, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*, impala::ContentType*)+0x28f
      C  [impalad+0x130648f]  impala::Webserver::BeginRequestCallback(sq_connection*, sq_request_info*)+0x50f
      C  [impalad+0x1322ed0]  boost::_bi::bind_t<impala::ThriftClientImpl*, boost::_mfi::mf4<impala::ThriftClientImpl*, impala::ClientCache<impala::ImpalaBackendClient>, impala::TNetworkAddress const&, void**, std::string, bool>, boost::_bi::list_av_5<impala::ClientCache<impala::ImpalaBackendClient>*, boost::arg<1>, boost::arg<2>, std::string, bool>::type> boost::bind<impala::ThriftClientImpl*, boost::_mfi::mf4<impala::ThriftClientImpl*, impala::ClientCache<impala::ImpalaBackendClient>, impala::TNetworkAddress const&, void**, std::string, bool>, impala::ClientCache<impala::ImpalaBackendClient>*, boost::arg<1>, boost::arg<2>, std::string, bool>(boost::_mfi::mf4<impala::ThriftClientImpl*, impala::ClientCache<impala::ImpalaBackendClient>, impala::TNetworkAddress const&, void**, std::string, bool>, impala::ClientCache<impala::ImpalaBackendClient>*, boost::arg<1>, boost::arg<2>, std::string, bool)+0x190
      C  [impalad+0x132290e]  impala::_TGetHadoopConfigResponse__isset::_TGetHadoopConfigResponse__isset()+0x2e
      C  [impalad+0x1322347]  __gnu_cxx::new_allocator<std::pair<impala::TUniqueId const, impala::QueryState*> >::~new_allocator()+0x7
      

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        I introduced this with commit 955b257cfb5: "IMPALA-5073: Part 1: add option to use mmap() for buffer pool". I actually had the code right on an earlier iteration but moved an #endif without thinking through the consequences.

        Show
        tarmstrong Tim Armstrong added a comment - I introduced this with commit 955b257cfb5: " IMPALA-5073 : Part 1: add option to use mmap() for buffer pool". I actually had the code right on an earlier iteration but moved an #endif without thinking through the consequences.
        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5230: fix non-functional impalad under ASAN

        The bug is that TCMalloc metrics are not initialised under
        ASAN but other metrics that reference them are created. As
        soon as a debug webpage tries to access that metric, Impala
        crashes.

        The bug was introduced by commit 955b257cfb5: "IMPALA-5073:
        Part 1: add option to use mmap() for buffer pool".

        Testing:
        Started the cluster under ASAN, ran some query tests, checked the /memz
        and /metrics pages.

        Change-Id: Ic71fa57e0444f470398e16513e4ba6e7f71565e4
        Reviewed-on: http://gerrit.cloudera.org:8080/6694
        Reviewed-by: Alex Behm <alex.behm@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5230 : fix non-functional impalad under ASAN The bug is that TCMalloc metrics are not initialised under ASAN but other metrics that reference them are created. As soon as a debug webpage tries to access that metric, Impala crashes. The bug was introduced by commit 955b257cfb5: " IMPALA-5073 : Part 1: add option to use mmap() for buffer pool". Testing: Started the cluster under ASAN, ran some query tests, checked the /memz and /metrics pages. Change-Id: Ic71fa57e0444f470398e16513e4ba6e7f71565e4 Reviewed-on: http://gerrit.cloudera.org:8080/6694 Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development