Bug 43308 - Persistent backend connections not supported for RewriteRule [P]
Persistent backend connections not supported for RewriteRule [P]
Status: NEW
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy
2.2.6
All All
: P3 normal (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
: PatchAvailable
Depends on: 43472
Blocks:
  Show dependency tree
 
Reported: 2007-09-04 23:27 UTC by Axel-Stephane Smorgrav
Modified: 2011-11-25 12:14 UTC (History)
3 users (show)



Attachments
Patch against httpd 2.2.4 (3.30 KB, patch)
2007-09-04 23:34 UTC, Axel-Stephane Smorgrav
Details | Diff
Configuration and debug log illustrating the creation of new workers (91.94 KB, text/plain)
2007-09-05 01:20 UTC, Axel-Stephane Smorgrav
Details
Patch in Unified format (2.50 KB, patch)
2007-09-13 01:54 UTC, Axel-Stephane Smorgrav
Details | Diff
Patch against httpd 2.2.4 (5.88 KB, patch)
2007-09-17 02:09 UTC, Axel-Stephane Smorgrav
Details | Diff
Patch against (5.92 KB, patch)
2007-09-17 02:29 UTC, Axel-Stephane Smorgrav
Details | Diff
Patch against 2.2.8 (5.93 KB, text/plain)
2008-03-28 02:51 UTC, Axel-Stephane Smorgrav
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Axel-Stephane Smorgrav 2007-09-04 23:27:45 UTC
In Apache 2.2.x a worker, and the associated backend connection pool, is created
for each ProxyPass directive. In that case the backend connections will by
default be persistent (reusable).

Another mechanism exists for proxying requests to a backend: RewriteRule with
the [P] flag. Unless the URL resulting from the rewriting process matches an
existing worker (created because of the presence of a RewriteRule), the
rewritten request will not be able to benefit from a persistent connection to
the backend. In that case the default worker (*) will be used. The connections
in the pool of the default worker are not persistent.

I therefore suggest to modify ap_proxy_pre_request() in order to create new
workers as needed in the event of requests that do not match an existing worker.
Comment 1 Axel-Stephane Smorgrav 2007-09-04 23:34:28 UTC
Created attachment 20771 [details]
Patch against httpd 2.2.4

The attached patch adds new workers on the fly as new backends for which no
worker exists are identified. This can happen if a RewriteRule proxies a
request (with the [P] flag), and the backend URL does not match any of the
ProxyPass' second argument (URL).

1. New workers are created for scheme://address/. The URL path is ignored. One
might want to add a new configuration directive for specifying the number of
path elements to include in the worker name.

2. The request for which the worker is created is served using the default
worker. The connection pool for the new worker is then only populated on the
first request matching the new worker (the 2nd request), and an existing
connection may only be reused from the following request on.

3. There is no way to tune the connection pooling of the workers created
on-the-fly.

4. One might want to add a new configuration directive for (de-)activating this
mechanism


Regarding the implementation details, being a novice to Apache development, I
look forward to your comments. However

i) PROXY_COPY_CONF_PARAMS should probably be moved to mod_proxy.h to avoid
duplication of code

ii) I allocate a copy of the URL on the stack rather than out of the pool since
as far as I could see, it is duplicated in ap_proxy_add_worker() anyway

iii) Someone skilled in the art of Apache httpd development could probably come
up with a way that the newly created worker could be used immediately instead
of having to wait for the next request.
Comment 2 Axel-Stephane Smorgrav 2007-09-05 01:20:44 UTC
Created attachment 20772 [details]
Configuration and debug log illustrating the creation of new workers

The attached file contains the debug log tracing a few requests made against a
patched server.
Comment 3 Paul Querna 2007-09-10 18:52:30 UTC
Can you provide the patch in 'diff -u' format?
Comment 4 Axel-Stephane Smorgrav 2007-09-13 01:54:01 UTC
Created attachment 20808 [details]
Patch in Unified format

Here's the same patch again, this time in diff -u format
Comment 5 Axel-Stephane Smorgrav 2007-09-16 23:52:21 UTC
In order to address certain concerns about possible DoS attacks or potential
resource consumption, rather than a configuration directive for de-activating
this mechanism, I suggest adding a new configuration directive ProxyMaxAddtlWorkers.

It will take a single numeric argument which is the maximum number of workers
that can be implicitly created. The default value would be 0 - zero -
effectively disabling the creation of additional workers.

Apart from limiting the number of workers created by the above described
mechanism, an equivalent number of additional scoreboard entries should also be
allocated during the configuration phase in order to avoid starving client
connections.
Comment 6 Axel-Stephane Smorgrav 2007-09-17 02:09:13 UTC
Created attachment 20830 [details]
Patch against httpd 2.2.4

In addition to creating workers implicitly, this new patch implements a new
configuration directive ProxyMaxAddtlWorkers which defaults to 0. This
directive limits the number of workers that can be implicitly created during
request processing and an equivalent number of additional scoreboard entries
are allocated at startup.

Another advantage over the previous patch is that the newly created worker is
used immediately - no need to wait for the next request.

The macro PROXY_COPY_CONF_PARAMS has been moved from mod_proxy.c to mod_proxy.h
because it is also used in proxy_util.c.
Comment 7 Axel-Stephane Smorgrav 2007-09-17 02:29:10 UTC
Created attachment 20831 [details]
Patch against 

The scoreboard entries are not really allocated, but proxy_lb_workers is
incremented by the value of ProxyMaxAddtlWorkers.
Comment 8 Takashi Sato 2007-09-28 23:45:03 UTC
I've read mod_proxy.c and proxy_util.c and found a hackful workaround.
Using ProxyPass to create dummy workers.
Rewriting applied, then uses the dummy ProxyPass's worker.

-----example-----

my server conf:
RewriteEngine On
RewriteRule ^/([^.]+)\.php$ http://localhost:20000/test/$1.php [P]
ProxyPassReverse / http://localhost:20000/test/

added:
ProxyPass /keepalive-dummy/ http://localhost:20000/
Comment 9 Axel-Stephane Smorgrav 2007-10-01 10:26:16 UTC
(In reply to comment #8)
> I've read mod_proxy.c and proxy_util.c and found a hackful workaround.
> Using ProxyPass to create dummy workers.
> Rewriting applied, then uses the dummy ProxyPass's worker.

Yes indeed, but it supposes that you code the exhaustive list of workers into
the configuration. If your backend server list is in a RewriteMap where you can
add and delete workers without restarting Apache, or if you use any other means
to maintain the list of backend servers, this approach does not work.
Comment 10 Axel-Stephane Smorgrav 2008-03-28 02:51:33 UTC
Created attachment 21731 [details]
Patch against 2.2.8
Comment 11 Axel-Stephane Smorgrav 2010-04-28 03:13:22 UTC
We have had this patch in production with several versions of Apache 2.2 including 2.2.14 without having encountered any problems.