Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-13115

BE: Call to filter 100 hosts by alerts takes 25s on 800 node cluster

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: 2.1.2
    • Component/s: ambari-server
    • Labels:
      None

      Description

      UI has a filter to show hosts having alerts. This call takes 25 to complete.

      POST http://gsperfb-1:8080/api/v1/clusters/MuonBlue/hosts?fields=Hosts/rack_info,Hosts/host_name,Hosts/maintenance_state,Hosts/public_host_name,Hosts/cpu_count,Hosts/ph_cpu_count,alerts_summary,Hosts/host_status,Hosts/last_heartbeat_time,Hosts/ip,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/service_name,host_components/HostRoles/desired_admin_state,Hosts/total_mem,stack_versions/HostStackVersions,stack_versions/repository_versions/RepositoryVersions/repository_version,stack_versions/repository_versions/RepositoryVersions/id,stack_versions/repository_versions/RepositoryVersions/display_name&minimal_response=true&page_size=100&from=0&sortBy=Hosts/host_name.asc
      
      {
        "RequestInfo": {
          "query": "page_size=100&from=0&alerts_summary/CRITICAL>0|alerts_summary/WARNING>0"
        }
      }
      

      Or the following GET call has same result:

      http://c6406:8080/api/v1/clusters/c1k/hosts?fields=Hosts/rack_info,Hosts/host_name,Hosts/maintenance_state,Hosts/public_host_name,Hosts/cpu_count,Hosts/ph_cpu_count,alerts_summary,Hosts/host_status,Hosts/last_heartbeat_time,Hosts/ip,host_components/HostRoles/state,host_components/HostRoles/maintenance_state,host_components/HostRoles/stale_configs,host_components/HostRoles/service_name,host_components/HostRoles/desired_admin_state,Hosts/total_mem,stack_versions/HostStackVersions,stack_versions/repository_versions/RepositoryVersions/repository_version,stack_versions/repository_versions/RepositoryVersions/id,stack_versions/repository_versions/RepositoryVersions/display_name&minimal_response=true&page_size=100&from=0&sortBy=Hosts/cpu_count.desc
      
      1. AMBARI-13115.patch
        11 kB
        Srimanth Gunturi

        Issue Links

          Activity

          Hide
          srimanth.gunturi Srimanth Gunturi added a comment -

          The reason this call takes so much time is because the 'host' resource has sub-resources, and there is a paging request on it. When a resource with sub-resources is requested, AND it has paging request, all top-level are got, and their sub-resources are gotten before the paging/predicate filters are applied. This is very expensive when there are 1000 hosts with sub-resources and only 100 hosts are needed. Bulk of the time is spent in populating the 900 top-level resources which are then thrown away.

          There is an optimization we can make here where we can reduce the number of top-level resources by applying the paging & predicate request on top-level resources IF the predicate applies only to the top-level resource and not to the sub-resources. Then we can safely reduce the number of top-level resources by applying the paging/predicate filtering.

          Show
          srimanth.gunturi Srimanth Gunturi added a comment - The reason this call takes so much time is because the 'host' resource has sub-resources, and there is a paging request on it. When a resource with sub-resources is requested, AND it has paging request, all top-level are got, and their sub-resources are gotten before the paging/predicate filters are applied. This is very expensive when there are 1000 hosts with sub-resources and only 100 hosts are needed. Bulk of the time is spent in populating the 900 top-level resources which are then thrown away. There is an optimization we can make here where we can reduce the number of top-level resources by applying the paging & predicate request on top-level resources IF the predicate applies only to the top-level resource and not to the sub-resources. Then we can safely reduce the number of top-level resources by applying the paging/predicate filtering.
          Hide
          srimanth.gunturi Srimanth Gunturi added a comment -

          Below are the times for the request
          Before Patch (ms)

          5452
          4932
          5162
          3984
          6004
          4853
          4089
          3958
          4329

          After Patch (ms)
          1599
          2100
          1297
          1275
          1175
          1185
          1079
          903
          1742
          1074

          Show
          srimanth.gunturi Srimanth Gunturi added a comment - Below are the times for the request Before Patch (ms) 5452 4932 5162 3984 6004 4853 4089 3958 4329 After Patch (ms) 1599 2100 1297 1275 1175 1185 1079 903 1742 1074
          Hide
          srimanth.gunturi Srimanth Gunturi added a comment -

          Committed to trunk and branch-2.1

          Show
          srimanth.gunturi Srimanth Gunturi added a comment - Committed to trunk and branch-2.1
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Ambari-branch-2.1 #541 (See https://builds.apache.org/job/Ambari-branch-2.1/541/)
          AMBARI-13115. BE: Call to filter 100 hosts by alerts takes 25s on 800 node cluster (srimanth) (sgunturi: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=4f2079fa420ba9cff002a2a93939e3a54eddc0ab)

          • ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java
          • ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java
          • ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Ambari-branch-2.1 #541 (See https://builds.apache.org/job/Ambari-branch-2.1/541/ ) AMBARI-13115 . BE: Call to filter 100 hosts by alerts takes 25s on 800 node cluster (srimanth) (sgunturi: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=4f2079fa420ba9cff002a2a93939e3a54eddc0ab ) ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Ambari-trunk-Commit #3452 (See https://builds.apache.org/job/Ambari-trunk-Commit/3452/)
          AMBARI-13115. BE: Call to filter 100 hosts by alerts takes 25s on 800 node cluster (srimanth) (sgunturi: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=44effc56526e7afe8f9ca8fc2a80c65c844f8d30)

          • ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java
          • ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java
          • ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Ambari-trunk-Commit #3452 (See https://builds.apache.org/job/Ambari-trunk-Commit/3452/ ) AMBARI-13115 . BE: Call to filter 100 hosts by alerts takes 25s on 800 node cluster (srimanth) (sgunturi: http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=44effc56526e7afe8f9ca8fc2a80c65c844f8d30 ) ambari-server/src/test/java/org/apache/ambari/server/api/query/QueryImplTest.java ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClusterControllerImplTest.java ambari-server/src/main/java/org/apache/ambari/server/api/query/QueryImpl.java

            People

            • Assignee:
              srimanth.gunturi Srimanth Gunturi
              Reporter:
              srimanth.gunturi Srimanth Gunturi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development