Details
-
Sub-task
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.8.0
-
None
-
None
Description
The ExternalSortBatch operator uses spill-to-disk to keep memory needs within a defined limit. However, the "copier" (really, the merge operation) can use an amount of memory determined not by the operator configuration, but by the width of each record.
The copier memory limit appears to be set by the COPIER_BATCH_MEM_LIMIT value.
However, the actual memory use is determined by the number of records that the copier is asked to copy. That record comes from an estimate of row width based on the type of each column. Note that the row width is not based on the actual data in each row. Varchar fields, for example, are assumed to be 40 characters wide. If the sorter is asked to sort records with Varchar fields of, say, 1000 characters, then the row width estimate will be a poor estimator of actual width.
Memory use is based on a
target record count = memory limit / estimate row width
Actual memory use is:
memory use = target row count * actual row width
Which is
memory use = memory limit * actual row width / estimate row width
That is, memory use depends on the ratio of actual to estimated width. If the estimate is off by 2, then we use twice as much memory as expected.
Not that the memory used for the copier defaults to 20 MB, so even an error of 4x still means only 80 MB of memory used; small in comparison to the many GB typically allocated to ESB storage.