Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-3052

Allocator performance issue when using a large number of filters.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.24.2, 0.25.0
    • Component/s: allocation
    • Labels:
      None

      Description

      With a large number of frameworks that all refuse offers permanently, the filter lists can get long. Comparing the slave IDs on the long filter lists ends up taking noticeable time.

      Since each filter list traversal matches on on a single slave ID, we can optimize this by hashing the slave ID and checking the hash before comparing the full slave ID. This is a 5% - 10% performance win.

      1. MESOS-3052.tiff
        105 kB
        James Peach

        Activity

        Hide
        bmahler Benjamin Mahler added a comment -
        commit fdc4536c2125e7502ca6b37c14afccbd7917d974
        Author: James Peach <jpeach@apache.org>
        Date:   Tue Sep 1 18:29:29 2015 -0700
        
            Improved performance of filters in the allocator.
        
            When frameworks refuse a lot of resources, the list of filters gets
            long. Since the filters are per-slave,
            HierarchicalAllocatorProcess::isFiltered spends a lot of time just
            comparing SlaveID (which tend to be long strings). Eliminate this
            whole problem by organizing the filters by SlaveID in the first
            place.
        
            Review: https://reviews.apache.org/r/37913
        
        Show
        bmahler Benjamin Mahler added a comment - commit fdc4536c2125e7502ca6b37c14afccbd7917d974 Author: James Peach <jpeach@apache.org> Date: Tue Sep 1 18:29:29 2015 -0700 Improved performance of filters in the allocator. When frameworks refuse a lot of resources, the list of filters gets long. Since the filters are per-slave, HierarchicalAllocatorProcess::isFiltered spends a lot of time just comparing SlaveID (which tend to be long strings). Eliminate this whole problem by organizing the filters by SlaveID in the first place. Review: https://reviews.apache.org/r/37913
        Hide
        jamespeach James Peach added a comment -
        Show
        jamespeach James Peach added a comment - Benjamin Mahler can you cast an eye over https://reviews.apache.org/r/37913/ ?
        Hide
        jamespeach James Peach added a comment -

        For posterity, the patch is https://reviews.apache.org/r/36951/. I'll knock up a new one that organizes the filters by slave.

        Show
        jamespeach James Peach added a comment - For posterity, the patch is https://reviews.apache.org/r/36951/ . I'll knock up a new one that organizes the filters by slave.
        Hide
        jamespeach James Peach added a comment -

        Yup it looks reasonable to organize the filters by slaveId on each framework. I did the change this way because it was less intrusive and SUPPRESS was the better fix anyway.

        Show
        jamespeach James Peach added a comment - Yup it looks reasonable to organize the filters by slaveId on each framework. I did the change this way because it was less intrusive and SUPPRESS was the better fix anyway.
        Hide
        bmahler Benjamin Mahler added a comment -

        Hm.. why bother keeping the long lists in the first place? How about doing (1) from here? That should give a bigger win.

        Show
        bmahler Benjamin Mahler added a comment - Hm.. why bother keeping the long lists in the first place? How about doing (1) from here ? That should give a bigger win.
        Hide
        jamespeach James Peach added a comment -

        Screenshot of %user CPU on mesos masters before and after this change,

        Show
        jamespeach James Peach added a comment - Screenshot of %user CPU on mesos masters before and after this change,
        Hide
        jamespeach James Peach added a comment -

        I have a patch proposal for this, pending internal release approval.

        Show
        jamespeach James Peach added a comment - I have a patch proposal for this, pending internal release approval.

          People

          • Assignee:
            jamespeach James Peach
            Reporter:
            jamespeach James Peach
            Shepherd:
            Benjamin Mahler
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development