Uploaded image for project: 'Traffic Server'
  1. Traffic Server
  2. TS-5095

IOBufferReader::read_avail adds considerably to CPU utilization

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.1.0
    • Component/s: Core
    • Labels:
      None

      Description

      When installing a new build of our ATS, we noticed that the CPU utilization was higher than a non-upgraded peer in the same pod. Looking at perf top for that process we saw that 10-17% of CPU was spent in IOBufferReader::read_avail. In the older system, that function didn't show up in the top couple screens of perf top.

      I tracked it down to an "ink_assert(read_avail() > 0)" call in IOBufferReader::consume. We didn't have the debug asserts compiled in, but the parameter arguments are still being executed. Commented out the ink_assert call and the CPU utilization went back to normal. We later found a similar growth in IOBufferReader::read() due to the use of IOBufferReader::read_avail.

      It looks like the issue is compiling with openssl 1.0.2 instead of openssl 1.0.1. We have been running against the openssl 1.0.2 library but were still compiling against openssl 1.0.1. We recently upgraded our builds to compile against the new openssl as well.

      I built a version of the older source against openssl 1.0.2 and it had very similar performance to what I was seeing with the newer source.

      Still don't know why compiling against the newer openssl would make such a difference. But it is easy enough to rework these use cases to take read_avail (which walks the block chain) out of the fast path. This should be a good optimization in any case.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shinrich Susan Hinrichs
                Reporter:
                shinrich Susan Hinrichs
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h