Large page cache kills initial performance

• Key: DERBY-704
• Type: Bug
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Knut Anders Hatlen
• Reporter: Knut Anders Hatlen
• Component/s: Services
• Affects Version/s: 10.1.1.0, 10.1.2.1, 10.1.3.1, 10.2.1.6
• Fix Version/s: 10.1.3.1, 10.2.1.6

Description

When the page cache is large the performance gets lower as the page
cache is being filled. As soon as the page cache is filled, the
throughput increases. In the period with low performance, the CPU
usage is high, and when the performance increases the CPU usage is
lower.

This behaviour is caused by the algorithm for finding free slots in
the page cache. If there are invalid pages in the page cache, it will
be scanned to find one of those pages. However, when multiple clients
access the database, the invalid pages are often already taken. This
means that the entire page cache will be scanned, but no free invalid
page is found. Since the scan of the page cache is synchronized on the
cache manager, all other threads that want to access the page cache
have to wait. When the page cache is large, this will kill the
performance.

When the page cache is full, this is not a problem, as there will be
no invalid pages.

Comments

[Knut Anders Hatlen]

Attached patch and report from running derbyall (no tests failed).

The patch modifies the search for a free slot in the page cache
(org.apache.derby.impl.services.cache.Clock.findFreeItem()) in the
following way:

  1) find out how many invalid pages there are in the page cache
  2) check whether pages that are skipped in the search are invalid
  3) if the number of skipped invalid pages equals the number of
     invalid pages, stop the search for invalid pages

Could someone please review the patch? Thanks.

[Knut Anders Hatlen]

Attached graphs showing the throughput and CPU usage running with and
without the patch. The graphs show the average of seven runs,
throughput and CPU usage were reported every 30 seconds.

12 clients were running an update-intensive load on the database. The
database had 10 GB of user data, and the page cache size was 512
MB. The CPU usage is relatively low because the tests were run on an
8-CPU machine.

The first 30 minutes the CPU usage was higher and the throughput
significantly lower without the patch than with the patch.

[Knut Anders Hatlen]

Sorry, I forgot the output from svn stat.

M java/engine/org/apache/derby/impl/services/cache/Clock.java

[David Van Couvering]

Hi, Knut. Normally I wouldn't review a kernel change, but this is a very simple change, it seems to have a lot of value, and it makes sense to me. I'll do a sanity build and test and then I will go ahead and check in. If a kernel developer would like to review, please do so.

[David Van Couvering]

Checked in patch, revision number 344270

bash-2.05$ svn commit
Sending java/engine/org/apache/derby/impl/services/cache/Clock.java
Transmitting file data .
Committed revision 344270.

[Knut Anders Hatlen]

Fixed in revision 344270.

[Knut Anders Hatlen]

Issue is reopened in order to add more comments to the code.

[Knut Anders Hatlen]

Patch DERBY-704-extra-comments.diff adds the following comments
to the code:

  // 1) find out how many invalid items there are in the
  // cache
  // 2) search for a free invalid item
  // 3) stop searching when there are no more invalid
  // items to find

and

  // Invalid items might occur in the cache when
  // a) a new item is created in growCache(), but it
  // is not in use yet, or
  // b) an item is deleted (usually when a table is
  // dropped)

  // It is critical to break out of the loop as soon as
  // possible since we are blocking others trying to
  // access the page cache. New items are added to the
  // end of the page cache, so the search for invalid
  // items should start from the end.

If the comments are OK, could a committer please commit this change?

Thanks.

svn stat:

M java/engine/org/apache/derby/impl/services/cache/Clock.java

[Knut Anders Hatlen]

Fixed in revision 344270. Extra comments put into the code in revision 345215.

[Knut Anders Hatlen]

Reopening to port to 10.1.

[Knut Anders Hatlen]

Fix merged cleanly. Derbyall ran with two failures on Solaris 10 x86/Sun JVM 1.5 (derbynet/testSecMec.java and unit/T_Diagnosticable.unit), but these are also seen in Ole's regression tests.

Committed to 10.1 with revision 409374.