[FALCON-321] Feed evictor deleting more stuff than it should

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.5
• Fix Version/s: 0.5
• Labels: system-tests
• Assignee: Shaik Idris Ali
• Reporter: Raghav Kumar Gautam
• Created: 25/Feb/14 00:53
• Updated: 05/Mar/14 23:54
• Resolved: 03/Mar/14 13:10

Description

In FeedEvictor.java we have:

private void deleteParentIfEmpty(FileSystem fs, Path parent, Path feedBasePath) throws IOException {
        if (feedBasePath.equals(parent)) {
            LOG.info("Not deleting feed base path:" + parent);
        } else {
            if (fs.getContentSummary(parent).getFileCount() == 0) {
                LOG.info("Parent path: " + parent + " is empty, deleting path");
                if (fs.delete(parent, true)) {
                    LOG.info("Deleted empty dir: " + parent);
                } else {
                    throw new IOException("Unable to delete parent path:" + parent);
                }
                deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
            }
        }
    }

In the fs.getContentSummary(parent).getFileCount() call if the parent has no files but has directories then we delete the parent directory. Which is incorrect.

Here is log from falcon-regression's RetentionTest.testRetention(parameters: hours, 24, true, daily) :

2014-02-24 15:09:45,034 INFO [main] org.apache.falcon.retention.FeedEvictor: Applying retention on DATA=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}#META=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksMetaData#STATS=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksStats#TMP=/tmp type: instance, Limit: hours(24), timezone: UTC, frequency: hours, storageFILESYSTEM
2014-02-24 15:09:45,051 INFO [main] org.apache.falcon.retention.FeedEvictor: Normalized path : /retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}
2014-02-24 15:09:45,123 INFO [main] org.apache.falcon.retention.FeedEvictor: Searching for /retention/testFolders/*/*/*/*
2014-02-24 15:09:45,486 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted instance :/retention/testFolders/2014/01/21/00
2014-02-24 15:09:45,500 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path: /retention/testFolders/2014/01/21 is empty, deleting path
2014-02-24 15:09:45,509 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted empty dir: /retention/testFolders/2014/01/21
2014-02-24 15:09:45,511 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path: /retention/testFolders/2014/01 is empty, deleting path
2014-02-24 15:09:45,517 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted empty dir: /retention/testFolders/2014/01
2014-02-24 15:09:45,518 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path: /retention/testFolders/2014 is empty, deleting path
2014-02-24 15:09:45,525 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted empty dir: /retention/testFolders/2014
2014-02-24 15:09:45,526 INFO [main] org.apache.falcon.retention.FeedEvictor: Not deleting feed base path:/retention/testFolders

Stacktrace:

Failing Oozie Launcher, Main class [org.apache.falcon.retention.FeedEvictor], main() threw exception, Unable to delete instance: /retention/testFolders/2014/01/21/03
java.io.IOException: Unable to delete instance: /retention/testFolders/2014/01/21/03
	at org.apache.falcon.retention.FeedEvictor.deleteInstance(FeedEvictor.java:321)
	at org.apache.falcon.retention.FeedEvictor.fileSystemEvictor(FeedEvictor.java:174)
	at org.apache.falcon.retention.FeedEvictor.evictFS(FeedEvictor.java:149)
	at org.apache.falcon.retention.FeedEvictor.evict(FeedEvictor.java:139)
	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:121)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:93)