Major Description
When HostDB syncs to disk (by default every two minutes), active transactions will block when they reach HttpSM::do_hostdb_lookup. This is because do_hostdb_lookup calls hostDBProcessor.getbyname_imm which attempts to get the bucket locks. The delays generally last for 500-1200ms. This blocks the event loop so no other actions will be performed by the net handler until the lock is dropped.
I'm assuming that the bucket locks are grabbed by the sync logic. When I increased proxy.config.cache.hostdb.sync_frequency to 1200, the every two minute slow down went away. Fortunately proxy.config.cache.hostdb.sync_frequency set to 0 seems to completely eliminate the sync, which will be my suggested solution internally.
I tried reducing the size of the hostdb table, but that didn't seem to affect the delay time.
The delay only reliably exhibited on loaded system. Running my httperf test case on a machine with no other activity did not show the delays.