[FALCON-1372] Retention does not work in corner cases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.1
• Fix Version/s: 0.9
• Assignee: Balu Vellanki
• Reporter: Balu Vellanki
• Created: 06/Aug/15 00:08
• Updated: 01/Dec/15 05:57
• Resolved: 15/Nov/15 21:06

Description

Originally reported by rostafiychuk.

RetentionTest#testRetention (24, DAYS, true, MONTHLY, true)
Uploaded feed:

<?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="A852708b0-3057547e" description="clicks log">
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(6)"/>
    <clusters>
        <cluster name="A852708b0-cd30589f" type="source">
            <validity start="2012-01-30T00:00Z" end="2099-03-31T23:59Z"/>
            <retention limit="days(24)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/${YEAR}/${MONTH}"/>
        <location type="stats" path="/projects/falcon/clicksStats"/>
        <location type="meta" path="/projects/falcon/clicksMetaData"/>
    </locations>
    <ACL owner="hrt_qa" group="users" permission="*"/>
    <schema location="/schema/clicks" provider="protobuf"/>
    <properties/>
</feed>

Created dirs:

dataDates = [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06, 2015/07, 2015/08, 2015/09, 2015/10, 2015/11, 2015/12, 2016/01, 2016/02, 2016/03, 2016/04, 2016/05, 2016/06, 2016/07, 2016/08, 2016/09, 2016/10, 2016/11, 2016/12, 2017/01, 2017/02, 2017/03, 2017/04, 2017/05, 2017/06, 2017/07, 2017/08, 2017/09, 2017/10, 2017/11, 2017/12, 2018/01, 2018/02, 2018/03, 2018/04, 2018/05, 2018/06, 2018/07] 

expected for deletion:

 [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05]

deleted:

[2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06]

24 days ago was 31th of May, but 2015/06 was deleted too.

Another case:
RetentionTest#testRetention (10080, HOURS, false, MONTHLY, true)

<retention limit="hours(10080)" action="delete"/>

10080 hours ago was 30th of April, but dir "2014/05" was deleted.

Issue Links

• is related to: FALCON-1603 FeedHelperTest::testGetDateFromPath fails in some environments
• links to: Review Board