[ATLAS-1818] Performance of Basic Search that Uses indexQuery Takes Long Time to Fetch Results

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8-incubating, trunk
• Fix Version/s: 0.8.1, 1.0.0
• Component/s: atlas-core, atlas-webui
• Assignee: Ashutosh Mestry
• Reporter: Ashutosh Mestry
• Due: 19/May/17
• Created: 18/May/17 21:01
• Updated: 19/May/17 23:15
• Resolved: 19/May/17 19:17

Description

Background

An environment that is setup with 100K hive_tables each with 84 columns.

The basic search with query parameter specified is executed. Results take 75 secs to appear.

Analysis & Findings

Similar test was performed with smaller data set (200 hive_tables each with 81 columns) resulted in less than ideal performance.

Atlas Basic Search API uses graph.indexQuery for performing search. This uses Solr for doing the search.

There are 2 aspects that affect performance:

• Solr's default for returning max query set when no limit is specified is 100K. In the test scenario, this is returning entire result set.
• Once result set is returned, EntityDiscoveryService.searchUsingBasicQuery does a sequential scan to filter data relevant to the query. This operation is proportional to size of the result set.

Solution

Following changes will improve performance:

• Solr's max result set property is governed by atlas.graph.index.search.max-result-set-size. It will make sense to set this to a lower number.
• Modify Solr's configuration solrconfig.xml to use FastLRUCache.
• Modify EntityDiscoveryService.searchUsingBasicQuery to form a query that takes additional parameters.

Issue Links

• links to: Code Review