[IMPALA-9767] ASAN crash during coordinator runtime filter updates

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: asan broken-build crash
• Epic Color: ghx-label-13
• Assignee: Riza Suminto
• Reporter: Sahil Takiar
• Created: 20/May/20 18:35
• Updated: 23/Feb/21 17:18
• Resolved: 23/Feb/21 17:18

Description

ASAN crash output:

Error MessageAddress Sanitizer message detected in /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/logs/ee_tests/impalad.ERRORStandard Error==4808==ERROR: AddressSanitizer: heap-use-after-free on address 0x7f6288cbe818 at pc 0x00000199f6fe bp 0x7f63c1a8b270 sp 0x7f63c1a8aa20
READ of size 1048576 at 0x7f6288cbe818 thread T73 (rpc reactor-552)
    #0 0x199f6fd in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904
    #1 0x19a1f57 in read_msghdr(void*, __sanitizer::__sanitizer_msghdr*, long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2781
    #2 0x19a46c3 in __interceptor_sendmsg /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:2796
    #3 0x372034d in kudu::Socket::Writev(iovec const*, int, long*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/net/socket.cc:447:3
    #4 0x331c095 in kudu::rpc::OutboundTransfer::SendBuffer(kudu::Socket&) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/transfer.cc:227:26
    #5 0x3324da1 in kudu::rpc::Connection::WriteHandler(ev::io&, int) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/connection.cc:802:31
    #6 0x52ca4e2 in ev_invoke_pending (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52ca4e2)
    #7 0x32aeadc in kudu::rpc::ReactorThread::InvokePendingCb(ev_loop*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:196:3
    #8 0x52cdb03 in ev_run (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x52cdb03)
    #9 0x32aecd1 in kudu::rpc::ReactorThread::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:497:9
    #10 0x32c08db in boost::_bi::bind_t<void, boost::_mfi::mf0<void, kudu::rpc::ReactorThread>, boost::_bi::list1<boost::_bi::value<kudu::rpc::ReactorThread*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #11 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #12 0x2144b29 in kudu::Thread::SuperviseThread(void*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:675:3
    #13 0x7f6c0bcf4e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
    #14 0x7f6c0885834c in __clone (/lib64/libc.so.6+0xf834c)

0x7f6288cbe818 is located 24 bytes inside of 1052640-byte region [0x7f6288cbe800,0x7f6288dbf7e0)
freed by thread T114 here:
    #0 0x1a773e0 in operator delete(void*) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:137
    #1 0x7f6c090faed3 in __gnu_cxx::new_allocator<char>::deallocate(char*, unsigned long) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:110
    #2 0x7f6c090faed3 in std::string::_Rep::_M_destroy(std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:449
    #3 0x7f6c090faed3 in std::string::_Rep::_M_dispose(std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:249
    #4 0x7f6c090faed3 in std::string::reserve(unsigned long) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:511
    #5 0x2781865 in impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
    #6 0x26d57d5 in impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
    #7 0x266bd65 in impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
    #8 0x27a1eed in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
    #9 0x3312e70 in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
    #10 0x3312231 in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
    #11 0x22f557b in impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
    #12 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #13 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #14 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
    #15 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
    #16 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #17 0x41a8939 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)

previously allocated by thread T109 here:
    #0 0x1a76668 in operator new(unsigned long) /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_new_delete.cc:92
    #1 0x7f6c090fa168 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/ext/new_allocator.h:104
    #2 0x7f6c090fa168 in std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:607
    #3 0x7f6c090fbb7b in _S_construct_aux<char const*> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1743
    #4 0x7f6c090fbb7b in _S_construct<char const*> /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.h:1764
    #5 0x7f6c090fbb7b in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) /mnt/source/gcc/build-4.9.2/x86_64-unknown-linux-gnu/libstdc++-v3/include/bits/basic_string.tcc:207
    #6 0x338b610 in impala::Coordinator::FilterState::ApplyUpdate(impala::UpdateFilterParamsPB const&, impala::Coordinator*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1417:51
    #7 0x338a4c0 in impala::Coordinator::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/coordinator.cc:1315:12
    #8 0x2781865 in impala::ClientRequestState::UpdateFilter(impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/client-request-state.cc:1451:11
    #9 0x26d57d5 in impala::ImpalaServer::UpdateFilter(impala::UpdateFilterResultPB*, impala::UpdateFilterParamsPB const&, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impala-server.cc:2694:19
    #10 0x266bd65 in impala::DataStreamService::UpdateFilter(impala::UpdateFilterParamsPB const*, impala::UpdateFilterResultPB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/data-stream-service.cc:119:44
    #11 0x27a1eed in std::_Function_handler<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*), impala::DataStreamServiceIf::DataStreamServiceIf(scoped_refptr<kudu::MetricEntity> const&, scoped_refptr<kudu::rpc::ResultTracker> const&)::$_5>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2
    #12 0x3312e70 in std::function<void (google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14
    #13 0x3312231 in kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3
    #14 0x22f557b in impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15
    #15 0x22fe3fb in boost::_bi::bind_t<void, boost::_mfi::mf0<void, impala::ImpalaServicePool>, boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #16 0x2148c26 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14
    #17 0x2a8c116 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/util/thread.cc:360:3
    #18 0x2a97998 in void boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9
    #19 0x2a977eb in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()() /data/jenkins/workspace/impala-asf-master-core-asan/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16
    #20 0x41a8939 in thread_proxy (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a8939)

Thread T73 (rpc reactor-552) created by T0 here:
    #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x2143d2c in kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.cc:619:15
    #2 0x32b8755 in kudu::Status kudu::Thread::Create<void (kudu::rpc::ReactorThread::*)(), kudu::rpc::ReactorThread*>(std::string const&, std::string const&, void (kudu::rpc::ReactorThread::* const&)(), kudu::rpc::ReactorThread* const&, scoped_refptr<kudu::Thread>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/util/thread.h:164:12
    #3 0x32ae5a4 in kudu::rpc::ReactorThread::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:188:10
    #4 0x32b6602 in kudu::rpc::Reactor::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/reactor.cc:762:18
    #5 0x329bffb in kudu::rpc::Messenger::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:447:5
    #6 0x329b6ab in kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/kudu/rpc/messenger.cc:203:3
    #7 0x22dae9f in impala::RpcMgr::Init(impala::TNetworkAddress const&) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/rpc/rpc-mgr.cc:151:3
    #8 0x2343545 in impala::ExecEnv::Init() /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/runtime/exec-env.cc:385:3
    #9 0x26a8422 in ImpaladMain(int, char**) /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/impalad-main.cc:73:3
    #10 0x1a7aa9d in main /data/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/src/service/daemon-main.cc:37:12
    #11 0x7f6c08781c04 in __libc_start_main (/lib64/libc.so.6+0x21c04)

Thread T114 created by T0 here:
    #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x41a7cc9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
    #2 0x45e0360d  (<unknown module>)

Thread T109 created by T0 here:
    #0 0x198755d in __interceptor_pthread_create /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/asan_interceptors.cc:317
    #1 0x41a7cc9 in boost::thread::start_thread_noexcept() (/data0/jenkins/workspace/impala-asf-master-core-asan/repos/Impala/be/build/debug/service/impalad+0x41a7cc9)
    #2 0x45e0360d  (<unknown module>)

SUMMARY: AddressSanitizer: heap-use-after-free /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/asan/../sanitizer_common/sanitizer_common_interceptors.inc:904 in read_iovec(void*, __sanitizer::__sanitizer_iovec*, unsigned long, unsigned long)
Shadow bytes around the buggy address:
  0x0fecd118fcb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fecd118fcc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fecd118fcd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fecd118fce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0fecd118fcf0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0fecd118fd00: fd fd fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fecd118fd10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fecd118fd20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fecd118fd30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fecd118fd40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0fecd118fd50: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==4808==ABORTING 

Issue Links

• is duplicated by: IMPALA-9879 ASAN use-after-free with KRPC thread and Coordinator::FilterState::ApplyUpdate()
• is duplicated by: IMPALA-10260 heap-use-after-free AddressSanitizer error in aggregating runtime filters
• is duplicated by: IMPALA-10307 Read over freed space in heap in kudu code
• is duplicated by: IMPALA-10480 heap-use-after-free crash in ASAN build
• is related to: IMPALA-7984 Port UpdateFilter() and PublishFilter() to KRPC
• relates to: IMPALA-9879 ASAN use-after-free with KRPC thread and Coordinator::FilterState::ApplyUpdate()