[CLOUDSTACK-4599] GRE isolation - createandConfigureTunnelNetwork failing on XenServer 6.1

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.1.1
• Component/s: Network Controller, Network Devices, XenServer
• Security Level: Public (Anyone can view this level - this is the default.)

Environment

Management server: CentOS 6.4 x86_64, CentOS Plus repo enabled, also pri/sec storage NFS server
Hosts: XenServer 6.1, included OVS (1.4.2?)
All the above running on ESX 5.1 (but I don't think it's a factor), with recommended CPU and RAM configurations.

• Assignee: Murali Reddy
• Reporter: David Black
• Created: 03/Sep/13 12:39
• Updated: 28/Oct/13 12:45

Description

Each hypervisor has two NICs, one on a public/management network, the other dedicated to private/isolated traffic. Both NICs (as xenbrX) have IP addresses and each host can reach the other over the respective bridge IP addresses. The bridge networks are labeled Public (xenbr0) and Private (xenbr1). The xenbr1 bridge IP subnet has no default gateway as it's there only for the tunnel endpoints on each host.

Guest networking is configured to be on the Private label, isolation type GRE. In addition, in Global Settings,
sdn.ovs.controller = true and sdn.ovs.controller.default.label = Private. I tried VLAN (vNet) ranges of 1-4094 and 0-2147483647. In the below example it was set to the latter and picked 10960.

BTW, it took quite a bit of scouring the net to find out how to config and what hypervisor (XenServer only) GRE tunnels should work with, in 4.1.1. Also, I'd prefer to use KVM but understand that support is also in the process of being implemented.

I had initially tried using XCP 1.6 and received a different error, then realized XCP support isn't yet finished. I thought XenServer 6.1 should work though. The xapiX bridges are set up on the host and a vifX.Y appears to be added for the tunnel endpoints, then the next step - I guess creating the tunnel itself, fails. In this example, both the virtual router and single isolated VM were on the same host... but I also could not migrate either one, receiving a "VM requires network" error. I won't provide details of that second error right now, but can if desired. Below is an excerpt of the management log

Please let me know what other details to provide. If desired, I can arrange access to the vCloud Director system on which it's running, preferably via IPv6 and/or of course collect more logs, run tests, etc.

2013-09-02 22:11:38,508 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-407:null) Xen Server network for tunnels found:OVSTunnel10960
2013-09-02 22:11:38,579 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-407:null) Create a vif on dom0 for tunnel network for account 10960
2013-09-02 22:11:38,873 WARN [xen.resource.CitrixResourceBase] (DirectAgent-407:null) createandConfigureTunnelNetwork failed
The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
at com.xensource.xenapi.Types.checkResponse(Types.java:1694)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VIF.plug(VIF.java:846)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.enableXenServerNetwork(CitrixResourceBase.java:655)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configureTunnelNetwork(CitrixResourceBase.java:745)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:5135)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:541)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-09-02 22:11:38,873 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-407:null) OVS Bridge configured
2013-09-02 22:11:38,873 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-407:null) Seq 7-390594607: Response Received:
2013-09-02 22:11:38,873 DEBUG [agent.transport.Request] (DirectAgent-407:null) Seq 7-390594607: Processing: { Ans: , MgmtId: 345040289813, via: 7, Ver: v1, Flags: 110, [{"Answer":{"result":true,"wait":0}}] }
2013-09-02 22:11:38,873 DEBUG [agent.transport.Request] (Job-Executor-16:job-42) Seq 7-390594607: Received: { Ans: , MgmtId: 345040289813, via: 7, Ver: v1, Flags: 110,

{ Answer }

}
2013-09-02 22:11:38,874 DEBUG [network.ovs.OvsTunnelManagerImpl] (Job-Executor-16:job-42) Placeholder for something more meanginful to come