[SAMZA-1034] Support LATEST path and single file as the input of HDFSSystemConsumer

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Assignee: Hai Lu
• Reporter: Hai Lu
• Created: 07/Oct/16 22:17
• Updated: 10/Oct/16 19:51
• Resolved: 10/Oct/16 19:51

Description

Right now users have to specify a directory of the HDFS to be consumed. Many other systems support the idea of "LATEST" file/directory. For example, if we have:

/data/database/db1/snapshot-2016-09-10
/data/database/db1/snapshot-2016-09-11

Then "/data/database/db1/#LATEST" will automatically point to "/data/database/db1/snapshot-2016-09-11"

We want to implement such a feature.

Plus, we also want to support the consumption of single file.