Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.2
-
None
Description
At present, the BPEL server stashes the metadata of deployed processes in an in-memory, unbounded cache. Given the unbounded nature of the cache, the server is bound to eventually run out of memory when presented with a sufficiently large process set. The graceful way to handle this situation would be to place bounds on the cache, either in terms of the total size or number of processes that it may contain at any given point in time. While throttling the server this way may effectively reduce its throughput, it is certainly the far lesser evil compared to the alternative.
To that end, we define the following configurable properties for the benefit of the administrator:
a) "process.hydration.throttled.maximum.size": The value of this property specifies the maximum size of the metadata of all processes that are currently in-use and may be cached in-memory. If the process metadata was hydrated at least once by the server instance, then its size is calculated once by traversing its object model. If not, then we estimate its in-memory size based on the size of the (.cbp) file where its serialized bytes are persisted.
b) "process.hydration.throttled.maximum.count": The value of this property specifies the maximum number of processes that are currently in-use and whose metadata may be cached in-memory.
A process that is stored on disk but is not loaded in-memory is said to be "unhydrated". When the server receives a message that is targeted at an unhydrated process, we must decide whether or not there is sufficient free memory in the cache for it. If not, then we select the least recently used process as our victim and evict (or dehydrate) it. This check and balance is performed until there is sufficient memory, in which case we may hydrate the process. On the other hand, if the cache capacity is exceeded, then we process the message based on its exchange pattern, as described below:
i) If the message is one-way (asynchronous), then we queue it back with the job scheduler so that it can be retried later. The hope is that, given a sufficient amount of delay and number of retries, the cache will have enough room for the targeted process.
ii) If the message is two-way (synchronous), then we cannot simply re-schedule it back, because the client will no doubt timeout. The logical thing to do here is to just return a SOAP fault message that forces the client to handle retries itself, if it so desires.
Furthermore, the administrator may use the following properties to enable and control lazy-loading:
c) "process.hydration.lazy": The value of this property specifies whether or not a process metadata is to be lazy-loaded. If so, then the server will not immediately load process metadata upon startup or deployment. In fact, the process is not hydrated until the server receives a message that demands that that process be loaded in-memory. This is a global property that may be overriden on a process-by-process basis by specifying a value for its namesake in the process' deployment descriptor (deploy.xml) file.
d) "process.hydration.lazy.minimum.size": The value of this property specifies the approximate minimum size of the process metadata for which lazy-loading should be enabled. If the server calculates the approximate size of the process metadata (based on its serialized file size) to be less than the value specified herein, then it will load the process eagerly. This way, you don't suffer the time delay on hydration for messages that don't deal with large-sized processes.
Lastly, we introduce the following property to throttle the number of instances that a process may handle:
e) "process.instance.throttled.maximum.count": The value of this property specifies the maximum number of instances that may be simultaneously active for any given process. This is a global property that may be overriden on a process-by-process basis by specifying a value for its namesake in the process' deployment descriptor (deploy.xml) file. Note that we assume that the process is already hydrated at the time we perform this check and as such this property is not directly related to caching. However, it may have an indirect effect on the cache by virtue of the fact that the it will most likely cause the process to dehydrate sooner rather than later (through dehydration policies).
PS: Note that the the in-memory representation of process metadata is made up of WSDL and BPEL object models. The cache that we refer to here deals specifically with BPEL objects. For the purposes of this issue, caching of WSDL objects is considered to be out of scope.
