[CLOUDSTACK-4621] Changing the management server's ethernet interface / mac address leaves the system in unstable state. - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Critical
Resolution: Unresolved
Affects Version/s: 4.2.0
Fix Version/s: Future
Component/s: Management Server
Security Level: Public (Anyone can view this level - this is the default.)
Labels:
None

Description

Steps to reproduce:

1. Have latest CloudStack setup with 4.2 build.
2. Have at least 1 advanced zone using Xen Cluster
3. deploy VMs and make sure everything works fine.

Note : In my case, the management server deployed on VMware.

Before MAC changes on mgmt server :

[root@Rhel63-Sanjeev ~]# cat ifconfig.output
eth1 Link encap:Ethernet HWaddr 06:04:5A:00:00:66
inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
inet6 addr: fe80::404:5aff:fe00:66/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:35035311 errors:0 dropped:0 overruns:0 frame:0
TX packets:31941744 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:31951991629 (29.7 GiB) TX bytes:17754778160 (16.5 GiB)

mysql> select * from mshost;
----------------------------------------------------------------------------------------------------------------------

msid

runid

name

state

version

service_ip

service_port

last_update

removed

alert_count

----------------------------------------------------------------------------------------------------------------------

6615759585382

1378110990284

Rhel63-Sanjeev

4.2.0

10.147.59.126

9090

2013-09-06 04:44:45

NULL

4. I have manually logged into my vmware host and change the above MAC address to "06:04:5A:00:00:68" and that resulted in a new interface.

[root@Rhel63-Sanjeev ~]# ifconfig
eth2 Link encap:Ethernet HWaddr 06:04:5A:00:00:68
inet addr:10.147.59.126 Bcast:10.147.59.255 Mask:255.255.255.0
inet6 addr: fe80::404:5aff:fe00:68/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:294927 errors:0 dropped:0 overruns:0 frame:0
TX packets:475806 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:22230231 (21.2 MiB) TX bytes:392491450 (374.3 MiB)

5. Restart the management server and verify the cloud.mshost table

mysql> select * from mshost;
----------------------------------------------------------------------------------------------------------------------

msid

runid

name

state

version

service_ip

service_port

last_update

removed

alert_count

----------------------------------------------------------------------------------------------------------------------

1	6615759585382	1378110990284	Rhel63-Sanjeev	Up	4.2.0	10.147.59.126	9090	2013-09-06 04:44:45	NULL	0
2	6615759585384	1378462772622	Rhel63-Sanjeev	Up	4.2.0	10.147.59.126	9090	2013-09-06 10:20:35	NULL	0

----------------------------------------------------------------------------------------------------------------------

Observations:

i. Now it created a new entry for the mshost due to mac / interface changes
ii. Both the above entries shows the status as UP and it assumes there are two mgmt servers
iii. All the system VMs have the mgmt_server_id set to the old which is not infact up

mysql> select * from host where name like '%-VM%';
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

name

uuid

status

type

private_ip_address

private_netmask

private_mac_address

storage_ip_address

storage_netmask

storage_mac_address

storage_ip_address_2

storage_mac_address_2

storage_netmask_2

cluster_id

public_ip_address

public_netmask

public_mac_address

proxy_port

data_center_id

pod_id

cpus

speed

url

fs_type

hypervisor_type

hypervisor_version

ram

resource

version

parent

total_size

capabilities

guid

available

setup

dom0_memory

last_ping

mgmt_server_id

disconnected

created

removed

update_count

resource_state

owner

lastUpdated

engine_state

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

s-1-VM

ab73e26c-7c52-4d7b-9389-97135c399ea6

SecondaryStorageVM

10.147.40.228

255.255.254.0

06:51:08:00:00:08

10.147.40.228

255.255.254.0

06:51:08:00:00:08

NULL

10.147.48.3

255.255.255.0

06:c8:d6:00:00:0b

NULL

NoIqn

NULL

4.2.0

NULL

s-1-VM-PremiumSecondaryStorageResource

1346155088

6615759585384

NULL

2013-09-02 09:15:00

NULL

Enabled

NULL

Disabled

v-2-VM

c6e9f30a-b920-40f7-b320-1c6375ef287a

ConsoleProxy

10.147.40.226

255.255.254.0

06:59:6a:00:00:06

10.147.40.226

255.255.254.0

06:59:6a:00:00:06

NULL

10.147.48.4

255.255.255.0

06:35:aa:00:00:0c

NULL

NoIqn

NULL

4.2.0

NULL

Proxy.2-ConsoleProxyResource

1346155089

6615759585384

NULL

2013-09-02 09:17:02

NULL

Enabled

NULL

Disabled

s-3-VM

12a0b450-afe3-45b5-a88b-d528f244b016

SecondaryStorageVM

10.147.40.240

255.255.254.0

06:04:dc:00:00:0a

10.147.40.240

255.255.254.0

06:04:dc:00:00:0a

NULL

10.147.48.11

255.255.255.0

06:78:00:00:00:0b

NULL

NoIqn

NULL

4.2.0

NULL

s-3-VM-PremiumSecondaryStorageResource

1346155089

6615759585384

NULL

2013-09-02 09:44:02

NULL

Enabled

NULL

Disabled

v-4-VM

ffb8865a-d917-431b-9a61-feac7ce49b69

ConsoleProxy

10.147.40.231

255.255.254.0

06:68:88:00:00:01

10.147.40.231

255.255.254.0

06:68:88:00:00:01

NULL

10.147.48.12

255.255.255.0

06:4b:76:00:00:0c

NULL

NoIqn

NULL

4.2.0

NULL

Proxy.4-ConsoleProxyResource

1346155089

6615759585384

NULL

2013-09-02 09:46:14

NULL

Enabled

NULL

Disabled

iv. This resulted in issues like unable to deploy user VMs and also unable restart/destroy system VMs

2013-09-06 01:11:31,295 DEBUG [agent.manager.ClusteredAgentAttache] (AgentManager-Handler-8:null) Seq 1-873922573: Forwarding Seq 1-873922573: { Cmd , MgmtId: 6615759585384, via: 1, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"fabe8c72-74a8-4816-80f5-64fb7a4de583","origUrl":"http://10.147.28.7/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2","uuid":"0bf2c102-13aa-11e3-a816-06045a000066","id":1,"format":"VHD","accountId":1,"checksum":"74b92f031cc5c2089ee89efb81344dcf","hvm":false,"displayText":"SystemVM Template (XenServer)","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"routing-1","hypervisorType":"XenServer"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"1f5d583d-ef0b-4689-a01d-2848dfabdead","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c65a038a-750c-3b4f-bf26-7ce3b74e1c85","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_xen_os","port":2049}},"name":"ROOT-20","size":223003018,"volumeId":27,"vmName":"r-20-VM","accountId":2,"format":"VHD","id":27,"hypervisorType":"XenServer"}},"executeInSequence":true,"wait":0}}] } to 6615759585382

Attaching all the relevant logs along with db dump.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

logs.tgz
06/Sep/13 07:26
522 kB
Venkata Swamy

Activity

People

Assignee:: Unassigned

Reporter:: Venkata Swamy

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 06/Sep/13 06:56

Updated:: 09/Oct/13 10:15