[CLOUDSTACK-6473] Debian 7 Virtual Router ip_conntrack_max not set at boot

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 4.3.0
• Fix Version/s: 4.5.0
• Component/s: Virtual Router
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: XenServer 6.2; CloudStack 4.3.0; Debian 7 SystemVM/Virtual Router
• Assignee: Sheng Yang
• Reporter: Logan B
• Created: 22/Apr/14 17:31
• Updated: 31/Jan/15 09:37
• Resolved: 14/Nov/14 22:40

Description

The Problem:
The Debian 7 Virtual Router VMs for XenServer experiences intermittent connectivity problems. This affects all VMs behind the virtual router in various ways: SSH failures, Apache connections fail, etc.

This issue also affects various function within CloudStack that attempt to connect to the Virtual Router (updating firewall rules, NAT, etc.)

The Cause:
It appears that the issues is being caused by a low default limit for the net.ipv4.netfilter.ip_conntrack_max sysctl. The issue can be easily diagnosed in /var/log/messages:
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.117498] nf_conntrack: table full, dropping packet.
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.133095] nf_conntrack: table full, dropping packet.
Apr 22 15:45:34 r-5602-VM kernel: [ 1085.145440] nf_conntrack: table full, dropping packet.

The default setting for ip_conntrack_max is '3796':

1. sysctl net.ipv4.netfilter.ip_conntrack_max
   net.ipv4.netfilter.ip_conntrack_max = 3796

As per /etc/sysctl.conf this setting should be '1000000':
net.ipv4.netfilter.ip_conntrack_max=1000000

It would appear that this setting is not being correctly applied when the virtual router boots.

The Solution:

• A temporary workaround is to manually set the ip_conntrack_max sysctl to the correct value:

1. sysctl -w net.ipv4.netfilter.ip_conntrack_max=1000000

It's likely that this sysctl is being run at boot before the module is loaded, so it doesn't take effect. There are various solutions suggested around the web, any of which should work fine.

To resolve this problem a new System VM template should be created. I'm assuming this can be done in between CloudStack releases. I know there is supposed to be a new template released to fix the HeartBleed vulnerability, so this would be a good fix to include with that updated template.

Issue Links

• duplicates: CLOUDSTACK-6997 conntrack set to a very low value on router