[MESOS-3589] Mesos tasks with many ports cause significant master performance problems

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.23.0
• Assignee: Unassigned
• Reporter: James Mulcahy
• Created: 06/Oct/15 00:02
• Updated: 16/Oct/15 08:43
• Resolved: 16/Oct/15 08:43

Description

Today, I used a framework to fire off some tasks which each requested a lot of ports. When doing so, the meson-master performance drops heavily.

With 70 tasks each requesting 100 ports, the master seems to spend the majority (~90%) of its time dealing with merging and coalescing Value_Range objects.

I'll attach some text views of the output of 'perf' record/report from the meson-master during this. The call-graph trace for the most frequently sampled call is:

-   4.42%  mesos-master  mesos-master         [.] google::protobuf::internal::RepeatedPtrFieldBase::size() const                                                                                                                                                                            
   - google::protobuf::internal::RepeatedPtrFieldBase::size() const                                                                                                                                                                                                                         
      - 37.05% mesos::Value_Ranges::range_size() const                                                                                                                                                                                                                                      
         - 91.07% mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Range const&)                                                                                                                                                                                                          
            - mesos::coalesce(mesos::Value_Ranges*, mesos::Value_Ranges const&)                                                                                                                                                                                                             
               - 68.35% mesos::operator+=(mesos::Value_Ranges&, mesos::Value_Ranges const&)                                                                                                                                                                                                 
                  - 99.46% Option<mesos::Value_Ranges> mesos::Resources::get<mesos::Value_Ranges>(std::string const&) const                                                                                                                                                                 
                     - mesos::internal::model(mesos::Resources const&)                                                                                                                                                                                                                      
                        - 97.58% mesos::internal::model(mesos::internal::Task const&)                                                                                                                                                                                                       
                             mesos::internal::master::model(mesos::internal::master::Framework const&)                                                                                                                                                                                      
                             mesos::internal::master::Master::Http::state(process::http::Request const&) const                                                                                                                                                                              
                             mesos::internal::master::Master::initialize()::{lambda(process::http::Request const&)#9}::operator()(process::http::Request const&) const                                                                                                                      
                             std::_Function_handler<process::Future<process::http::Response> (process::http::Request const&), mesos::internal::master::Master::initialize()::{lambda(process::http::Request const&)#9}>::_M_invoke(std::_Any_data const&, process::http::Request const&)    
                             std::function<process::Future<process::http::Response> (process::http::Request const&)>::operator()(process::http::Request const&) const                                                                                                                       
                             process::ProcessBase::visit(process::HttpEvent const&)                                                                                                                                                                                                         
                             process::HttpEvent::visit(process::EventVisitor*) const                                                                                                                                                                                                        
                             process::ProcessBase::serve(process::Event const&)                                                                                                                                                                                                             
                             process::ProcessManager::resume(process::ProcessBase*)                                                                                                                                                                                                         
                             process::schedule(void*)                                                                                                                                                                                                                                       
                             start_thread                                                                                                                                                                                                                                                   

The top ~80 call sides pretty much all relate to Value_Range handling, and account for 90% of the runtime of the mesos-master.

Issue Links

• duplicates: MESOS-3051 performance issues with port ranges comparison
• is related to: MESOS-3051 performance issues with port ranges comparison