[FALCON-1580] Feed Retention corner case : does not clean up feed instances

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8
• Component/s: feed
• Assignee: Unassigned
• Reporter: Balu Vellanki
• Created: 02/Nov/15 23:50
• Updated: 02/Nov/15 23:50

Description

If a Feed entity is defined as follows,

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="rawEmailFeed" description="Raw customer email feed" xmlns="uri:falcon:feed:0.1">
    <tags>externalSystem=USWestEmailServers</tags>
    <groups>churnAnalysisDataPipeline</groups>
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(1)"/>
    <clusters>
        <cluster name="primaryCluster" type="source">
            <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
            <retention limit="days(30)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/user/ambari-qa/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
        <location type="stats" path="/"/>
        <location type="meta" path="/"/>
    </locations>
    <ACL owner="ambari-qa" group="users" permission="0x755"/>
    <schema location="/none" provider="/none"/>
</feed>

The feed is valid for one day, hence the feed bundle is valid for a day. The retention coordinator completes after a day with status SUCCEEDED. During this one day, there are no feed instances older than 30 days. So no feed instances are deleted.

After 30 days, the user expects feed instances to be deleted. But there is no retention coordinator running to delete the feed instances. The solution is to set the "endtime" for retention coordinator to "feed validityEndTime + retentionAgeLimit".