[CLOUDSTACK-5446] KVM-Secondary Store down-Even after secondary store is brought back up after being down for few hours,snapshot jobs do not get triggered with reason "there is other active snapshot tasks on the instance to which the volume is attached"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.3.0
• Fix Version/s: 4.4.0, 4.5.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: Build from 4.3
• Assignee: Disheng Su
• Reporter: Sangeetha Hariharan
• Created: 10/Dec/13 19:47
• Updated: 11/Dec/14 16:53
• Resolved: 19/Nov/14 23:54

Description

KVM - Secondary Store down - Even after secondary store is brought back up after being down for few hours , snapshot jobs do not get triggered with reason "here is other active snapshot tasks on the instance to which the volume is attached, please try again later"

Set up:
Advanced Zone set up with 2 KVM (RHEL 6.3) hosts.

Steps to reproduce the problem:

1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms.
2. Start concurrent snapshots for ROOT volumes of all the Vms.
3. Shutdown the Secondary storage server when the snapshots are in the progress.
4. Bring the Secondary storage server up after ~ 12 hours.

When the secondary storage was down:

The snapshot jobs that were in progress timed out after 6 hours.

Even after this , I do not see the hourly snapshots being scheduled due to the following reason:

2013-12-10 13:07:49,285 WARN [c.c.s.s.SnapshotSchedulerImpl] (SnapshotPollTask:ctx-cf0775f7) Scheduling snapshot failed due to com.cloud.utils.exception.CloudRuntimeException: There is other active snapshot tasks on the instance to which the volume is attached, please try again later

Even after the secondary storage was brought up , there is no hourly snapshots being scheduled due to the same reason - "com.cloud.utils.exception.CloudRuntimeException: There is other active snapshot tasks on the instance to which the volume is attached, please try again later"

mysql> select * FROM snapshots;
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

id

data_center_id

account_id

domain_id

volume_id

disk_offering_id

status

path

name

uuid

snapshot_type

type_description

size

created

removed

backup_snap_id

swift_id

sechost_id

prev_snap_id

hypervisor_type

version

s3_id

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

1	1	6	1	45	18	BackedUp	NULL	TestVM-tiny-host-0ps-0-0_ROOT-45_20131209234410	ee2c46b7-7623-439a-9a30-63eec1d95c56	3	HOURLY	21474836480	2013-12-09 23:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
2	1	6	1	43	18	BackedUp	NULL	TestVM-1_ROOT-43_20131209234410	62c01389-49df-475b-b225-617d3903d25a	3	HOURLY	21474836480	2013-12-09 23:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
3	1	6	1	54	19	BackedUp	NULL	TestVM-tiny-host-1ps-0-4_ROOT-54_20131209234410	8c7621f0-5d94-4f22-b3f8-7954c810e1be	3	HOURLY	21474836480	2013-12-09 23:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
4	1	6	1	53	19	BackedUp	NULL	TestVM-tiny-host-1ps-0-3_ROOT-53_20131209234410	30c72de6-2a3e-4bf2-88da-b2a3b6ae9c66	3	HOURLY	21474836480	2013-12-09 23:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
5	1	6	1	52	19	BackedUp	NULL	TestVM-tiny-host-1ps-0-2_ROOT-52_20131209234411	e8e9f99a-5a8b-4cfd-b743-327925955980	3	HOURLY	21474836480	2013-12-09 23:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
6	1	6	1	51	19	BackedUp	NULL	TestVM-tiny-host-1ps-0-1_ROOT-51_20131209234411	07e4edc8-b289-4f94-8696-ccf9db60e3ab	3	HOURLY	21474836480	2013-12-09 23:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
7	1	6	1	50	19	BackedUp	NULL	TestVM-tiny-host-1ps-0-0_ROOT-50_20131209234411	b285aa1a-6650-4b90-9c2c-e8bf0c507bfb	3	HOURLY	21474836480	2013-12-09 23:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
8	1	6	1	49	18	BackedUp	NULL	TestVM-tiny-host-0ps-0-4_ROOT-49_20131209234412	31e37c8d-aab2-4ee6-9c43-c23501b08b02	3	HOURLY	21474836480	2013-12-09 23:44:12	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
9	1	6	1	48	18	BackedUp	NULL	TestVM-tiny-host-0ps-0-3_ROOT-48_20131209234412	60b6023e-138f-42c2-9867-f559e1d95352	3	HOURLY	21474836480	2013-12-09 23:44:12	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
10	1	6	1	47	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-2_ROOT-47_20131209234412	fa37ebe5-f36c-4f19-b99f-249ab95c4e61	3	HOURLY	21474836480	2013-12-09 23:44:12	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
11	1	6	1	46	18	BackedUp	NULL	TestVM-tiny-host-0ps-0-1_ROOT-46_20131209234412	bd571765-b28e-433e-b480-ccdc3bc0450b	3	HOURLY	21474836480	2013-12-09 23:44:12	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
12	1	6	1	45	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-0_ROOT-45_20131210004410	4c6f50c2-661e-4bad-9f07-357b2ca4e7fc	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
13	1	6	1	43	18	CreatedOnPrimary	NULL	TestVM-1_ROOT-43_20131210004410	bd3744ae-182f-4a0b-9685-5bef0c747365	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
14	1	6	1	54	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-4_ROOT-54_20131210004410	14f53a2f-bb4b-40e5-aa90-e7b4be4a1076	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
15	1	6	1	53	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-3_ROOT-53_20131210004410	88dd82f5-ade5-4dd5-a147-e0a35a15be85	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
16	1	6	1	52	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-2_ROOT-52_20131210004410	1830af7d-97e5-488c-bc37-39c11961e509	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
17	1	6	1	51	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-1_ROOT-51_20131210004410	3d452970-f042-4af6-a8cc-3003482da9a8	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
18	1	6	1	50	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-0_ROOT-50_20131210004410	0cb1b176-bcfe-4bc2-9b5e-07ec8e9e97f5	3	HOURLY	21474836480	2013-12-10 00:44:10	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
19	1	6	1	49	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-4_ROOT-49_20131210004411	87337d8b-e698-47b1-961a-32479d44a6e5	3	HOURLY	21474836480	2013-12-10 00:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
20	1	6	1	48	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-3_ROOT-48_20131210004411	c7b321bd-31b4-4c9c-be8f-c9231cb08530	3	HOURLY	21474836480	2013-12-10 00:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
21	1	6	1	47	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-2_ROOT-47_20131210004411	ea1a71b0-9b52-42e5-a252-7a3e8938bc0f	3	HOURLY	21474836480	2013-12-10 00:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
22	1	6	1	46	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-1_ROOT-46_20131210004411	f1fbc899-5f55-472c-b798-639350b2facd	3	HOURLY	21474836480	2013-12-10 00:44:11	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
33	1	6	1	45	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-0_ROOT-45_20131210022749	396c80a8-299c-42b2-8c3d-32da5af9536c	3	HOURLY	21474836480	2013-12-10 02:27:49	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
34	1	6	1	43	18	CreatedOnPrimary	NULL	TestVM-1_ROOT-43_20131210022749	9444a355-08a6-4c7d-98f2-74729a0b28b5	3	HOURLY	21474836480	2013-12-10 02:27:49	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
35	1	6	1	54	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-4_ROOT-54_20131210022749	09cbfc03-7f9f-470f-bc34-e2bbad72b9ba	3	HOURLY	21474836480	2013-12-10 02:27:49	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
36	1	6	1	53	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-3_ROOT-53_20131210022749	2e7259f0-82d2-4910-9047-617d54bda035	3	HOURLY	21474836480	2013-12-10 02:27:49	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
37	1	6	1	52	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-2_ROOT-52_20131210022749	420230ed-a037-4ed3-a019-031d24fd0bfb	3	HOURLY	21474836480	2013-12-10 02:27:49	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
38	1	6	1	51	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-1_ROOT-51_20131210022750	6bf87e88-3a14-467f-ac5b-ab620fd03206	3	HOURLY	21474836480	2013-12-10 02:27:50	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
39	1	6	1	50	19	CreatedOnPrimary	NULL	TestVM-tiny-host-1ps-0-0_ROOT-50_20131210022750	93021350-9df1-4234-9df2-869d9e9a912f	3	HOURLY	21474836480	2013-12-10 02:27:50	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
40	1	6	1	49	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-4_ROOT-49_20131210022750	4699ebdf-3093-4f2f-86b7-ae5c9195cc6a	3	HOURLY	21474836480	2013-12-10 02:27:50	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
41	1	6	1	48	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-3_ROOT-48_20131210022750	8d663979-31a2-48ad-884b-f1ef85a1f4d5	3	HOURLY	21474836480	2013-12-10 02:27:50	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL
42	1	6	1	46	18	CreatedOnPrimary	NULL	TestVM-tiny-host-0ps-0-1_ROOT-46_20131210022751	f80c2994-b612-4234-add4-27d323cf3dd3	3	HOURLY	21474836480	2013-12-10 02:27:51	NULL	NULL	NULL	NULL	NULL	KVM	2.2	NULL

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
32 rows in set (0.00 sec)

mysql>

Issue Links

• is duplicated by: CLOUDSTACK-5395 When backup snapshot fails becasue of backup.snapshot.wait time exceeding , the vhd entries form the primary store is not getting cleared.
• is duplicated by: CLOUDSTACK-5370 Xenserver - Snapshots - vhd entries get accumulated on the primary store when snapshot creation fails becasue of not being able to reach the secondary store.