Major Description
When installing a new build of our ATS, we noticed that the CPU utilization was higher than a non-upgraded peer in the same pod. Looking at perf top for that process we saw that 10-17% of CPU was spent in IOBufferReader::read_avail. In the older system, that function didn't show up in the top couple screens of perf top.
I tracked it down to an "ink_assert(read_avail() > 0)" call in IOBufferReader::consume. We didn't have the debug asserts compiled in, but the parameter arguments are still being executed. Commented out the ink_assert call and the CPU utilization went back to normal. We later found a similar growth in IOBufferReader::read() due to the use of IOBufferReader::read_avail.
It looks like the issue is compiling with openssl 1.0.2 instead of openssl 1.0.1. We have been running against the openssl 1.0.2 library but were still compiling against openssl 1.0.1. We recently upgraded our builds to compile against the new openssl as well.
I built a version of the older source against openssl 1.0.2 and it had very similar performance to what I was seeing with the newer source.
Still don't know why compiling against the newer openssl would make such a difference. But it is easy enough to rework these use cases to take read_avail (which walks the block chain) out of the fast path. This should be a good optimization in any case.
Attachments
Issue Links
- links to
