[ANY23-336] Parsing json-ld content takes prohibitively long time

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.2
• Fix Version/s: 2.3
• Component/s: core, extractors
• Assignee: Hans Brende
• Reporter: Hans Brende
• Created: 27/Mar/18 18:05
• Updated: 02/Apr/18 19:41
• Resolved: 02/Apr/18 17:26

Description

Using the page https://www.guthriegreen.com as a benchmark, a page fetch took about 100 ms, while simply parsing the json-ld content on that page took a staggering 27400 ms. For reference, I'm using Java 8, build 162, on a Macbook Pro (early 2015).

The bad news is that this is not our fault.

I've profiled this behavior down to the com.github.jsonldjava.utils.JsonUtils.fromURL(URL, CloseableHttpClient) function. 94% of the parsing time is spent there. This function is called when trying to load remote json-ld contexts. 

In order to avoid loading remote contexts repeatedly, this function tries to cache them by using a CachingHttpClient from the httpclient-osgi library.

Unfortunately, that strategy is not working, as I have recorded exactly zero cache hits, meaning that every retrieval is a cache miss and a remote context is re-fetched via http every single time it's accessed.

 

Issue Links

• links to: GitHub Pull Request #69