Cache results from the Locationmap

• Key: FOR-711
• Type: Sub-task
• Status: Resolved
• Resolution: Fixed
• Priority: Critical
• Assignee: Tim Williams
• Reporter: Ross Gardler
• Component/s: Locationmap
• Fix Version/s: 0.8

Description

Now that we are using the locationmap extensively it is showing up just how innefficient it is. The problem is that for the majority of requests there are multiple reqeuests to the locationmap. We can make things much faster (especially on the first page request) by caching results in the locationmap.

I think a simple cache will sufice, lets just provide a static hashmap using the hint as a key and, of course, the location as the value.

If we test all locationmaps and find no result we should record that tere is no result in this hashmap. This will also be a good place to throw an exception so that Cocoon can better report such errors (see FOR-701)

Comments

[Ross Gardler]

Tim has implemented a basic caching mechanism using the Hash technique in the description.

This improves performance considerably, however, there are problems...

Because there are a whole chain of locationmaps it is impossible for the cache to know track which lm files have changed and which have not.

We may find some inspiration from the XMLFileModule (which is cacheable, see http://cocoon.apache.org/2.1/apidocs/org/apache/cocoon/components/modules/input/XMLFileModule.html). In this case we would have to aggregate all LMs together into a single doc and use XPath to find our match. I think this would actually be faster than iterating over all entries in the locationmap anyway. We are still left with the problem of knowing when the LM is invalidated, but if we do the aggregation of the sub-LMs in the sitemap we can rely on Cocoons caching to manage this for us.

A second approach would be to store the location of the LM when we store the source that matches a pattern. Then, when we retrieve from the cache we will be able to check the source LM has not been changed and reload if necessary.

This second approach may be faster since we we will not reload the whole LM when a single sub-LM has changed, but it reuses less of the core Cocoon code. It would also be easier to implement. Any comments?

[Tim Williams]

See also http://issues.apache.org/jira/browse/FOR-732

The XMLFileModule is the inspiration for the current LM caching. It doesn't have to deal with Source validity in the same way we do mainly because of the sub-LMs. The XMLFileModule, from what I can gather, was not a good choice to use for inspiration as it turns out anyway since it's caching is done on its own instead of in the cocoon store. I've been trying to figure out how we might be able to put arbitrary data in the cocoon store but hadn't thought of just aggregating the LMs and just implement cachaeblecomponent -- that's worth looking into though as mentioned, we've still got the considerable sourcevalidity question.

The second approach you mentioned is easy enough but just when I had that basically implemented, I started off on this cocoon store quest. then the holidays, then travel...etc:(

If we can't figure out how to store arbitrary data in the cocoon store (a mix of 1 and 2) then I think we should look into 1. I think the XMLFileModule is proving that storing data outside the store is not a good idea because it is not "managed" and ultimately leads to OutOfMemory exceptions -- though this is admittedly less likely in the LM.

[Tim Williams]

For more information on the XMLFileModule's caching problems:
http://issues.apache.org/jira/browse/FOR-591

[Thorsten Scherler]

El jue, 09-02-2006 a las 07:08 -0500, Tim Williams escribió:
...
> To both of you... Are these the documented issues or new ones? If
> it's for FOR-732, for development purposes, just turn lm caching off
> in forrest-core.xconf for now.

Well that is the problem, the caching should be configurable in the
sitemap and not the xconf since IMO we should be able to use caching and
non-caching locationmap lookups. *Without* restarting forrest. ;-)

>
> Actually, if I hear no complaints, I'll turn it off by default until
> FOR-732 is resolved.

No, please let it on, but maybe give us some instructions how we can
configure it on a sitemap level (best as <map:pipe/>).

[Ross Gardler]

With respect to adding data to the cocoon store, maybe this will help:

"Some components like the DirectoryGenerator and the TraxTransformer rely on the validity of other factors than just a template or a set of files. These components often can't determine the validity at setup time. The solution is to use the AggregateValidity and more specifically the DelayedAggregateValidity. The aggregated validity object provides an interface for you to add additional validity components inside and returns the result of the set (typically if one validity object is undertermined or invalid the whole set is)."

(from http://cocoon.zones.apache.org/daisy/documentation/writing/690.html)

Not looked at the atual code yet though

[Ross Gardler]

Following on the AggregatedValidity trial (no idea how Cocoon caching works, so I'm just dumping ideas at present)...

org.apache.cocoon.components.source.impl.MultiSourceValidity describes itself as:

<p>An aggregated {@link SourceValidity} for multiple sources.</p>

Now the question is, can an InputModule use these sitemap component cache tools - I'm guessing there is a reason the XMLFileInputModule doesn't use these classes...

[David Crossley]

Bringing a comment from the last time that we reviewed scheduled issues:
> Last time Ross said:
> > OK to go into 0.9, but we will need a FAQ entry explaining how to turn
> > caching off for development since with it on we are forced to reboot
> > Forrest each time the LM is changed. Tim mentioned in the archives how
> > to turn it off.

[Tim Williams]

I believe this one to also be resolved by the simple TTL fix.