Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: ---
    • Fix Version/s: 1.8.0
    • Component/s: libsvn_ra_serf
    • Labels:
      None

      Description

      Serf increases the CPU load on the server.  Using Subversion trunk as my
      testcase, and mod_dav_svn 1.7 on the server I see the following CPU used by
      apache during a checkout:
      
      http/neon:   8.5s
      http/serf:  13s
      https/neon: 10s
      https/serf: 17s
      
      So serf uses more server CPU and the problem is worse with HTTPS.
      

        Issue Links

          Activity

          Hide
          philipm Philip Martin added a comment -

          If I increase the logging the the difference between serf and neon increases.
          Since serf uses many more request that makes logging more expensive for serf and
          increases the server load.
          
          Since the difference is greater with https I assume that some of the TLS
          processing happens on a per-request basis.  Since serf makes many more requests
          the TLS overhead is greater for serf.
          

          Show
          philipm Philip Martin added a comment - If I increase the logging the the difference between serf and neon increases. Since serf uses many more request that makes logging more expensive for serf and increases the server load. Since the difference is greater with https I assume that some of the TLS processing happens on a per-request basis. Since serf makes many more requests the TLS overhead is greater for serf.
          Hide
          philipm Philip Martin added a comment -

          Using mod_status to monitor Apache I see a large increase in traffic with serf
          as opposed to neon.  I'm using mod_dav_svn trunk and I'm using Subversion trunk
          as my dataset.
          
          A single checkout over ra_neon generates 20.7MB of traffic, the same checkout
          over ra_serf generates 44.3MB of traffic.
          

          Show
          philipm Philip Martin added a comment - Using mod_status to monitor Apache I see a large increase in traffic with serf as opposed to neon. I'm using mod_dav_svn trunk and I'm using Subversion trunk as my dataset. A single checkout over ra_neon generates 20.7MB of traffic, the same checkout over ra_serf generates 44.3MB of traffic.
          Hide
          philipm Philip Martin added a comment -

          Those numbers are without mod_deflate.  With mod_deflate enabled the numbers are
          much closer: neon 14.7MB, serf 15.4MB.
          

          Show
          philipm Philip Martin added a comment - Those numbers are without mod_deflate. With mod_deflate enabled the numbers are much closer: neon 14.7MB, serf 15.4MB.
          Hide
          philipm Philip Martin added a comment -

          mod_deflate increases the CPU load.  With neon the CPU for a checkout with
          mod_deflate is 10s, for serf it is 20s.
          

          Show
          philipm Philip Martin added a comment - mod_deflate increases the CPU load. With neon the CPU for a checkout with mod_deflate is 10s, for serf it is 20s.
          Hide
          philipm Philip Martin added a comment -

          Doubling the traffic, or doubling the server CPU with mod_deflate, is enough to
          make this a concern for 1.7.
          

          Show
          philipm Philip Martin added a comment - Doubling the traffic, or doubling the server CPU with mod_deflate, is enough to make this a concern for 1.7.
          Hide
          philipm Philip Martin added a comment -

          Across the lan I can use the "ip -s link show eth0" to monitor the ethernet usage.
          
          A typical neon checkout uses:
          
          143469532-120760643=22708889  rx bytes
              6381216-6097084=  284132  tx bytes
                              22993021     bytes
          
                    106975-91636=15339  rx packets
                     53191-48960= 4231  tx packets
                                 19570     packets
          
          A typical serf checkout uses:
          
          317361273-266967630=50393643  rx bytes
            19707398-15319584= 4387814  tx bytes
                              54781457     bytes
          
                   247128-205531=41597  rx packets
                   151575-120108=31467  tx packets
                                 73064     packets
          
          
          I did the checkout 3 times for each library and the results were similar.
          

          Show
          philipm Philip Martin added a comment - Across the lan I can use the "ip -s link show eth0" to monitor the ethernet usage. A typical neon checkout uses: 143469532-120760643=22708889 rx bytes 6381216-6097084= 284132 tx bytes 22993021 bytes 106975-91636=15339 rx packets 53191-48960= 4231 tx packets 19570 packets A typical serf checkout uses: 317361273-266967630=50393643 rx bytes 19707398-15319584= 4387814 tx bytes 54781457 bytes 247128-205531=41597 rx packets 151575-120108=31467 tx packets 73064 packets I did the checkout 3 times for each library and the results were similar.
          Hide
          gstein Greg Stein added a comment -

          Shift to 1.8.0, after ungating per r1158522.
          

          Show
          gstein Greg Stein added a comment - Shift to 1.8.0, after ungating per r1158522.
          Hide
          gstein Greg Stein added a comment -

          My intent is to set up a testing sandbox, verify your results, enable mod_cache in the server, and see how 
          that turns out. It *should* greatly reduce the server load. If not, then something on the server needs to be 
          tuned.
          
          (and the underlying statement is that ra_serf reduces server load, if the server is configured to take 
          advantage of the caching hints that mod_dav_svn provides it)
          
          

          Show
          gstein Greg Stein added a comment - My intent is to set up a testing sandbox, verify your results, enable mod_cache in the server, and see how that turns out. It *should* greatly reduce the server load. If not, then something on the server needs to be tuned. (and the underlying statement is that ra_serf reduces server load, if the server is configured to take advantage of the caching hints that mod_dav_svn provides it)
          Hide
          philipm Philip Martin added a comment -

          mod_cache might reduce the CPU load caused by repository access, it might also
          reduce the CPU load caused by mod_deflate if caching happens after compression.
           I don't see how it can reduce the traffic or the CPU cause by SSL/TLS.
          

          Show
          philipm Philip Martin added a comment - mod_cache might reduce the CPU load caused by repository access, it might also reduce the CPU load caused by mod_deflate if caching happens after compression. I don't see how it can reduce the traffic or the CPU cause by SSL/TLS.
          Hide
          philipm Philip Martin added a comment -

          This is still a problem using trunk@1333080. The traffic reported by mod_status
          for a neon checkout is 21.1MB while for a serf checkout it is 46.7MB.
          

          Show
          philipm Philip Martin added a comment - This is still a problem using trunk@1333080. The traffic reported by mod_status for a neon checkout is 21.1MB while for a serf checkout it is 46.7MB.
          Hide
          gstein Greg Stein added a comment -

          My comment on Aug 16 is unrelated to this. mod_cache cannot help on a checkout.
          

          Show
          gstein Greg Stein added a comment - My comment on Aug 16 is unrelated to this. mod_cache cannot help on a checkout.
          Hide
          gstein Greg Stein added a comment -

          In my test just now, I exported (via http) a copy of subversion/trunk/ from a local repository.
          
          neon used 3.82 cpu secs of a single process
          serf used 4.68 cpu secs across all httpd processes
          
          22% increase. (the original description had 55% and 70% increases)
          
          This is using svn trunk, neon 0.29.6, and serf trunk (the same as 1.0 basically).
          
          I'm not concerned about the amount of traffic since that is fixable using mod_deflate. The administrator 
          can choose that if traffic is a concern (eg. not a problem on a LAN, but maybe useful in a WAN).
          
          Moving to 1.8-consider pending further discussion/investigation.
          

          Show
          gstein Greg Stein added a comment - In my test just now, I exported (via http) a copy of subversion/trunk/ from a local repository. neon used 3.82 cpu secs of a single process serf used 4.68 cpu secs across all httpd processes 22% increase. (the original description had 55% and 70% increases) This is using svn trunk, neon 0.29.6, and serf trunk (the same as 1.0 basically). I'm not concerned about the amount of traffic since that is fixable using mod_deflate. The administrator can choose that if traffic is a concern (eg. not a problem on a LAN, but maybe useful in a WAN). Moving to 1.8-consider pending further discussion/investigation.
          Hide
          gstein Greg Stein added a comment -

          Ouch. ra_serf sends individual PROPFIND requests for each node, rather than one per directory.
          
          Opening a new issue. That should be fixed. It will have an impact on this issue, but will keep this as the 
          "master" issue.
          
          

          Show
          gstein Greg Stein added a comment - Ouch. ra_serf sends individual PROPFIND requests for each node, rather than one per directory. Opening a new issue. That should be fixed. It will have an impact on this issue, but will keep this as the "master" issue.
          Hide
          philipm Philip Martin added a comment -

          I've already tried mod_deflate, see comments on 5 Aug 2011.  With mod_deflate
          the serf transfer is still greater than with neon, but the numbers are much
          closer. However mod_deflate increases the CPU difference between serf and neon.
          

          Show
          philipm Philip Martin added a comment - I've already tried mod_deflate, see comments on 5 Aug 2011. With mod_deflate the serf transfer is still greater than with neon, but the numbers are much closer. However mod_deflate increases the CPU difference between serf and neon.
          Hide
          philipm Philip Martin added a comment -

          Using trunk@1339233 mod_dav_svn/serf an a checkout of subversion trunk from a
          local mirror as the testcase:
          
          neon: cpu: 3.9s traffic: 21.2M
          serf: cpu: 3.6s traffic: 46.8M
          
          So the huge traffic increase is still present but serf does reduce server CPU usage.
          
          If I import subversion trunk into a fresh repository I get a different testcase,
          fewer deltas and properties in the repository.  For this I see:
          
          neon: cpu: 3.4s traffic: 22.3M
          serf: cpu: 2.7s traffic: 50.6M
          
          Not sure why the traffic goes as with fewer properties I expected it to be
          marginally lower. Serf makes more requests than for the previous dataset, does
          even better on CPU but even worse on traffic.
          

          Show
          philipm Philip Martin added a comment - Using trunk@1339233 mod_dav_svn/serf an a checkout of subversion trunk from a local mirror as the testcase: neon: cpu: 3.9s traffic: 21.2M serf: cpu: 3.6s traffic: 46.8M So the huge traffic increase is still present but serf does reduce server CPU usage. If I import subversion trunk into a fresh repository I get a different testcase, fewer deltas and properties in the repository. For this I see: neon: cpu: 3.4s traffic: 22.3M serf: cpu: 2.7s traffic: 50.6M Not sure why the traffic goes as with fewer properties I expected it to be marginally lower. Serf makes more requests than for the previous dataset, does even better on CPU but even worse on traffic.
          Hide
          cmpilato C. Michael Pilato added a comment -

          I'd be interested to see how my changes to embed all the prop-changes in the
          REPORT response for checkout/update/switch/diff/merge/status affects ra_serf's
          server-side load.  Philip, are you able to test this scenario?
          

          Show
          cmpilato C. Michael Pilato added a comment - I'd be interested to see how my changes to embed all the prop-changes in the REPORT response for checkout/update/switch/diff/merge/status affects ra_serf's server-side load. Philip, are you able to test this scenario?
          Hide
          philipm Philip Martin added a comment -

          Using serf 1.1.x@1691 and subversion trunk@1408335 as 1.8.
          Using 1.7.7 with neon as 1.7.
          Using subversion trunk as my dataset.
          
          The server CPU and bandwidth to service one checkout:
          
          1.7 neon client, 1.7 server
          3.6s, 21.4MB
          
          1.8 serf client, 1.7 server
          2.9s, 46.9MB
          
          1.7 neon client, 1.7 server, mod_deflate
          5.3s, 15.4MB
          
          1.8 serf client, 1.7 server, mod_deflate
          5.7s, 15.9MB
          
          1.7 neon client, 1.8 server
          3.3s, 21.4MB
          
          1.8 serf client, 1.8 server
          1.6s, 44.7MB
          
          1.7 neon client, 1.8 server, mod_deflate
          4.8s, 15.4MB
          
          1.8 serf client, 1.8 server, mod_deflate
          4.1s. 14.7MB
          
          Serf greatly increases bandwidth unless mod_deflate is in use (see
          http://svn.haxx.se/dev/archive-2012-05/0343.shtml for why). Using mod_deflate
          solves the bandwidth problem to some extent although CPU use is increased.
          

          Show
          philipm Philip Martin added a comment - Using serf 1.1.x@1691 and subversion trunk@1408335 as 1.8. Using 1.7.7 with neon as 1.7. Using subversion trunk as my dataset. The server CPU and bandwidth to service one checkout: 1.7 neon client, 1.7 server 3.6s, 21.4MB 1.8 serf client, 1.7 server 2.9s, 46.9MB 1.7 neon client, 1.7 server, mod_deflate 5.3s, 15.4MB 1.8 serf client, 1.7 server, mod_deflate 5.7s, 15.9MB 1.7 neon client, 1.8 server 3.3s, 21.4MB 1.8 serf client, 1.8 server 1.6s, 44.7MB 1.7 neon client, 1.8 server, mod_deflate 4.8s, 15.4MB 1.8 serf client, 1.8 server, mod_deflate 4.1s. 14.7MB Serf greatly increases bandwidth unless mod_deflate is in use (see http://svn.haxx.se/dev/archive-2012-05/0343.shtml for why). Using mod_deflate solves the bandwidth problem to some extent although CPU use is increased.
          Hide
          philipm Philip Martin added a comment -

          The 1.8 server is using less CPU than the 1.7 server. I think this is partly
          because I ran each checkout 10 times without restarting the server and FSFS
          caching means 1.8 uses less CPU. So 1.8 without mod_deflate uses much less CPU
          for serf than neon because it is simply reading data from the cache; with
          mod_deflate the serf CPU advantage is less because the caching of FSFS access
          becomes less significant.
          

          Show
          philipm Philip Martin added a comment - The 1.8 server is using less CPU than the 1.7 server. I think this is partly because I ran each checkout 10 times without restarting the server and FSFS caching means 1.8 uses less CPU. So 1.8 without mod_deflate uses much less CPU for serf than neon because it is simply reading data from the cache; with mod_deflate the serf CPU advantage is less because the caching of FSFS access becomes less significant.
          Hide
          philipm Philip Martin added a comment -

          I keep having to look this up so I'll record it here. There was a serious memory
          leak that prevented some people using mod_deflate with mod_dav.  This was fixed
          by r1103315 in httpd, see discussion:
          http://mail-archives.apache.org/mod_mbox/httpd-dev/201105.mbox/%3CBANLkTimeAz9pw_7zRnf+vVMMaGtXvcfqnA@mail.gmail.com%3E
          
          This fix is present in Apache 2.4 but not Apache 2.2.
          

          Show
          philipm Philip Martin added a comment - I keep having to look this up so I'll record it here. There was a serious memory leak that prevented some people using mod_deflate with mod_dav. This was fixed by r1103315 in httpd, see discussion: http://mail-archives.apache.org/mod_mbox/httpd-dev/201105.mbox/%3CBANLkTimeAz9pw_7zRnf+vVMMaGtXvcfqnA@mail.gmail.com%3E This fix is present in Apache 2.4 but not Apache 2.2.
          Hide
          markphip Mark Phippard added a comment -

          > So 1.8 without mod_deflate uses much less CPU
          > for serf than neon because it is simply reading data from the cache
          
          I get why you say 1.8 server would use less CPU than 1.7 due to the cache, but I fail to see the Serf/Neon 
          connection here and why these changes would help Serf more than Neon.
          
          The CPU usage for a Neon client also dropped with 1.8 server.  It seems like the reason Serf likely dropped 
          by more than Neon is that the 1.8 server response also includes the SVN properties so the Serf client does 
          not have to make as many requests as it does with a 1.7 server.
          
          Given that we cannot recommend mod_deflate with Apache 2.2, perhaps you should do tests using SSL 
          and using SSL compression?  I do not recall that it has the same problems as mod_deflate and with the 
          changes made to not double-compress maybe that will give good results that we can recommend.
          
          Overall, I think your test results look promising.  Serf clients are using less CPU load now than Neon 
          clients.  That seems like a tremendous amount of progress.  I would guess that if there were some large 
          binary files then Serf would get even better.  Neon would have to Base64 encode the data as well as waste 
          time trying to compress it, where as Serf will just stream the content.
          

          Show
          markphip Mark Phippard added a comment - > So 1.8 without mod_deflate uses much less CPU > for serf than neon because it is simply reading data from the cache I get why you say 1.8 server would use less CPU than 1.7 due to the cache, but I fail to see the Serf/Neon connection here and why these changes would help Serf more than Neon. The CPU usage for a Neon client also dropped with 1.8 server. It seems like the reason Serf likely dropped by more than Neon is that the 1.8 server response also includes the SVN properties so the Serf client does not have to make as many requests as it does with a 1.7 server. Given that we cannot recommend mod_deflate with Apache 2.2, perhaps you should do tests using SSL and using SSL compression? I do not recall that it has the same problems as mod_deflate and with the changes made to not double-compress maybe that will give good results that we can recommend. Overall, I think your test results look promising. Serf clients are using less CPU load now than Neon clients. That seems like a tremendous amount of progress. I would guess that if there were some large binary files then Serf would get even better. Neon would have to Base64 encode the data as well as waste time trying to compress it, where as Serf will just stream the content.
          Hide
          markphip Mark Phippard added a comment -

          Sounds like we should NOT recommend SSL compression:
          
          https://issues.apache.org/bugzilla/show_bug.cgi?id=53219
          
          So guess we should disregard that suggestion.
          

          Show
          markphip Mark Phippard added a comment - Sounds like we should NOT recommend SSL compression: https://issues.apache.org/bugzilla/show_bug.cgi?id=53219 So guess we should disregard that suggestion.
          Hide
          rhuijben Bert Huijben added a comment -

          mod_deflate and ssl use the same zlib compression library, so they will have +- 
          the same cpu impact.
          
          
          
          mod_ssl's compression has the 'advantage' that the same compression dictionary 
          can be used for multiple files, which can really help Subversion which usually 
          handles multiple similar files.
          
          
          
          I think we should try to configure just one compression library instead of two 
          to avoid cpu overhead, and to optimize compression. (See lgo's measurements)
          
          
          
          I don't think the ssl compression is to cpu heavy arguments apply here, as they 
          would for sites as commerce sites.
          

          Show
          rhuijben Bert Huijben added a comment - mod_deflate and ssl use the same zlib compression library, so they will have +- the same cpu impact. mod_ssl's compression has the 'advantage' that the same compression dictionary can be used for multiple files, which can really help Subversion which usually handles multiple similar files. I think we should try to configure just one compression library instead of two to avoid cpu overhead, and to optimize compression. (See lgo's measurements) I don't think the ssl compression is to cpu heavy arguments apply here, as they would for sites as commerce sites.
          Hide
          ivan.z Ivan Zhakov added a comment -

          Bert,
          
          From my testing I found that OpenSSL compression is very slow for some reason.
          
          Concerning double compression in mod_deflate and OpenSSL: I think the best
          solution is to add code to mod_deflate to disable it if OpenSSL already uses
          compression.
          

          Show
          ivan.z Ivan Zhakov added a comment - Bert, From my testing I found that OpenSSL compression is very slow for some reason. Concerning double compression in mod_deflate and OpenSSL: I think the best solution is to add code to mod_deflate to disable it if OpenSSL already uses compression.
          Hide
          cmpilato C. Michael Pilato added a comment -

          As regards the original concern, I think this issue can be closed as FIXED in 1.8.
          
          Regarding the other tangentially related stuffs discussed throughout, I confess
          I'm struggling to find something really actionable here.  We have open TODOs
          elsewhere that cover what server configuration recommendations we should be
          making to admins.
          
          So.  Closing as FIXED.
          

          Show
          cmpilato C. Michael Pilato added a comment - As regards the original concern, I think this issue can be closed as FIXED in 1.8. Regarding the other tangentially related stuffs discussed throughout, I confess I'm struggling to find something really actionable here. We have open TODOs elsewhere that cover what server configuration recommendations we should be making to admins. So. Closing as FIXED.

            People

            • Assignee:
              Unassigned
              Reporter:
              philipm Philip Martin
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development