Bug 44736 - mod_proxy_balancer looses it's mind on reloads.
mod_proxy_balancer looses it's mind on reloads.
Status: NEEDINFO
Product: Apache httpd-2
Classification: Unclassified
Component: mod_proxy_balancer
2.2.8
PC Linux
: P2 critical with 52 votes (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
:
: 42621 45950 (view as bug list)
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2008-04-01 17:22 UTC by Tautvydas
Modified: 2014-03-13 13:47 UTC (History)
13 users (show)



Attachments
keepalive.py - Python script to keep an apache process alive indefinitely by using the keepalive issue (544 bytes, text/x-python)
2011-01-26 20:03 UTC, Andrew
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Tautvydas 2008-04-01 17:22:50 UTC
I have in my config:

<Virtualhost 10.10.11.108:443>
        ServerName test
        DocumentRoot "/var/www/test/"
        ErrorDocument 500 "/error/50x.php"
        ErrorDocument 502 "/error/50x.php"
        ErrorDocument 503 "/error/50x.php"
        ErrorDocument 403 "/error/403.html"
        ErrorDocument 404 "/error/404.html"
        ProxyPass /error/ !
        ProxyPass /lb/ !
        RewriteEngine On
        RewriteRule ^/$ /test/ [R]
<Location /lb>
        SetHandler balancer-manager
</Location>
        ProxyTimeout 3000
        KeepAlive Off
        ProxyPass / balancer://test/ stickysession=JSESSIONID nofailover=On
        ProxyPassReverse / http://10.10.20.51:8080/
        ProxyPassReverse / http://10.10.21.51:8080/
        <Proxy balancer://dashboard/>
                BalancerMember http://10.10.20.51:8080 route=node1 retry=0
                BalancerMember http://10.10.21.51:8080 route=node2 retry=0
        </Proxy>
        ProxyPreserveHost On
        ProxyErrorOverride On
        SSLEngine on
        SSLCertificateFile /etc/httpd/certs/_.localhost.com.crt
        SSLCertificateKeyFile /etc/httpd/certs/_.localhost.com.key
        SSLCertificateChainFile /etc/httpd/certs/some_intermediate.crt
#       LogLevel debug
        ErrorLog /var/log/httpd/test.error.log
        CustomLog /var/log/httpd/test.access.log combined
</Virtualhost>


When apache 2.2.8 starts everything works fine and if I go to https://10.10.11.108/lb/

I see something like this:

StickySession	Timeout	FailoverAttempts	Method
JSESSIONID	0	1 	byrequests

Worker URL	Route	RouteRedir	Factor	Set	Status	Elected	To	From
http://10.10.20.51:8080	node1		1	0	Ok	19013	19M	143M
http://10.10.21.51:8080	node2		1	0	Ok	1602	752K	5.1M

However sometimes if i modify other virtualhosts and do:

# apachectl graceful

Load balancer looses stickyness, and if i go to https://10.10.11.108/lb/

I see something like this:

StickySession	Timeout	FailoverAttempts	Method
JSESSIONID	0	1 	byrequests

Worker URL	Route	RouteRedir	Factor	Set	Status	Elected	To	From
http://10.10.20.51:8080			0	0	Ok	19013	19M	143M
http://10.10.21.51:8080	node1		1	0	Ok	1602	752K	5.1M

If you do: /etc/init.d/httpd restart it works fine.
Comment 1 Jim Jagielski 2008-08-18 09:12:47 UTC
Please try with 2.2.9
Comment 2 Tautvydas 2009-03-09 12:10:27 UTC
Tried with 2.2.9 still not fixed. I believe my report is a duplicate of this:
https://issues.apache.org/bugzilla/show_bug.cgi?id=42621

For know my "simple solution" is perl wrapper around apache reload. Basically do apachectl graceful, read configs from a perl script and go to balance-manager web page to fix the routes.
Comment 3 William A. Rowe Jr. 2009-05-19 10:49:49 UTC
*** Bug 42621 has been marked as a duplicate of this bug. ***
Comment 4 Andrew 2011-01-26 20:03:50 UTC
Created attachment 26555 [details]
keepalive.py - Python script to keep an apache process alive indefinitely by using the keepalive issue
Comment 5 Glenn Nielsen 2011-03-28 10:49:56 UTC
I have seen the exact same problem with mod_proxy_balancer losing its routes when you do an apachectl graceful. Here is my relevant config:

ProxyPass /balancer-manager !

<Proxy balancer://webmail>
  BalancerMember http://boreas.sp:80 route=boreas loadfactor=1
  BalancerMember http://chinook.sp:80 route=chinook loadfactor=1
  BalancerMember http://zephyrus.sp:80 route=zephyrus loadfactor=1
  ProxySet lbmethod=byrequests
</Proxy>
ProxyPass / balancer://webmail/ stickysession=WEBMAILID
Comment 6 Glenn Nielsen 2011-03-28 10:53:24 UTC
I have seen the exact same problem with mod_proxy_balancer losing its routes when you do an apachectl graceful. Here is my relevant config:

ProxyPass /balancer-manager !

<Proxy balancer://webmail>
  BalancerMember http://boreas.sp:80 route=boreas loadfactor=1
  BalancerMember http://chinook.sp:80 route=chinook loadfactor=1
  BalancerMember http://zephyrus.sp:80 route=zephyrus loadfactor=1
  ProxySet lbmethod=byrequests
</Proxy>
ProxyPass / balancer://webmail/ stickysession=WEBMAILID

Oh, here is the server info (Server runs as a VM in ESX):

FreeBSD kottke.kinetic.more.net 7.3-RELEASE-p2 FreeBSD 7.3-RELEASE-p2 #0: Mon Jul 12 19:23:19 UTC 2010     root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC  amd64 

Server version: Apache/2.2.16 (FreeBSD)
Server built:   Aug 16 2010 15:38:53
Comment 7 Jim Jagielski 2011-08-04 14:49:40 UTC
WONTFIX - 2.2.x does not guarantee local changes via balancer-manager are kept
Comment 8 Robert Egglestone 2011-08-04 20:57:04 UTC
The comment on the WONTFIX above suggests that the issue has been misinterpreted.

The issue is not that local changes via balancer-manager are being lost, this behaviour is understood.

The issue is that by making change to the configuration file (no changes to balancer manager), and then doing a graceful restart, the routes can become offset in a way that no longer matches the configured servers.

For example, if I have this:

<Proxy balancer://balancer1/>
  BalancerMember http://host1 route=host1
  BalancerMember http://host2 route=host2
  BalancerMember http://host3 route=host3
</Proxy>

... and I comment out host1, then do a graceful restart, I may end up with Apache behaving like this ...

http://host2 route=host1
http://host3 route=host2


Some observed notes:

+ Problems appear when new balancer blocks are added or removed, or balancer 
members are added or removed.

+ The route names can even cross from one balancer configuration block to another!

+ It doesn't always happen, we see it on active systems.

+ A subsequent graceful restart usually fixes the routes.

+ When it does happen, it happens consistently - we have multiple httpd servers polling Subversion for configuration changes, then checking them out and doing a graceful restart. In this situation, all of our httpd servers tend to develop the same route problems at the same time.


This issue has various impacts on applications. In the worst case we saw it cause all traffic for a balancer to be directed to a single balancer member, which couldn't handle the load.
Comment 9 bugzilla 2012-04-02 20:15:09 UTC
Same here on a busy system changing between one active+hot spare backend servers configuration to a sticky load balanced setup activated by a reload causes node1 and node2 sessions to be routed to node1 first, but after a restart correctly to seperate nodes, which means half of the users loose their session info. Observed workaround is always to do a restart.
Comment 10 dario_uni 2012-04-13 22:53:22 UTC
Reproduced on OEL 5 with apache httpd 2.2.17.
I'll try to reproduce on 2.2.22.
Comment 11 tarun 2012-05-04 23:21:18 UTC
*** Bug 45950 has been marked as a duplicate of this bug. ***
Comment 12 krauseed 2012-10-18 01:00:10 UTC
I have been able to reproduce this on 2.2.22
Comment 13 Jim Jagielski 2012-11-01 14:34:12 UTC
Can you check w/ 2.4.x and/or trunk?
Comment 14 Frank 2014-03-13 13:47:20 UTC
Problem still exists with 2.2.24.

I have no idea about 2.4, though. That would be major change in our environment, which we don't intent to do.