[VCL-879] Add semaphore to iptables commands

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.2
• Fix Version/s: 2.5
• Component/s: vcld (backend)
• Assignee: Andrew Kurth
• Reporter: Andrew Kurth
• Created: 26/Jun/15 15:55
• Updated: 20/Jul/16 17:12
• Resolved: 20/Jul/16 17:12

Description

A command to configure iptables on a particular host may fail if another iptables command is being executed at the same time.

This is a problem for the iptables commands executed on NAT hosts. Multiple vcld processes may be running at the same time which attempt to modify a NAT host's iptables firewall. The following warning will appear in vcld.log when this problem occurs:

2015-06-25 12:45:09|13308|5183|5206|timeout|iptables.pm:flush_chain|588|flushed 'vcld-5206' chain in 'nat' table on nat-01

|13288|5182|5205|timeout|iptables.pm:flush_chain|584| ---- WARNING ----
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| 2015-06-25 12:45:09|13288|5182|5205|timeout|iptables.pm:flush_chain|584|failed to flush 'vcld-5205' chain in 'nat' table on nat-01, exit status: 4, command:
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| /sbin/iptables --flush vcld-5205 --table nat
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| output:
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| iptables: Resource temporarily unavailable.
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| ( 0) iptables.pm, flush_chain (line: 584)
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| (-1) iptables.pm, delete_chain (line: 433)
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| (-2) iptables.pm, sanitize_reservation (line: 485)
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| (-3) reclaim.pm, process (line: 136)
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| (-4) vcld, make_new_child (line: 587)
|13288|5182|5205|timeout|iptables.pm:flush_chain|584| (-5) vcld, main (line: 348)

This is not normally a problem for commands which configure iptables on a compute node because only a single vcld process is acting on the node at a time, which means a single iptables command is executed at a time.

A semaphore should be optained before executing iptables commands which may encounter this problem. There are several iptables commands which are executed on compute nodes and NAT hosts. Care should be taken to limit the use of a semaphore to only the commands which require it, otherwise performance will suffer.