[OOZIE-2216] Aperiodic Data handling in oozie

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: coordinator
• Assignee: Jaydeep Vishwakarma
• Reporter: Jaydeep Vishwakarma
• Created: 24/Apr/15 10:28
• Updated: 03/Jun/15 13:45

Description

Currently Oozie scheduling works on periodic datasets. It does not have any mechanism to handle aperiodic datasets, which doesn’t follow a fixed schedule/frequency.

Use cases
When incoming dataset arrives with no fixed schedule.
Need to trigger the job based all data available since last run with a possible cap on the max size to process in one run.
Try to avoid creating so many instances when you know input instances will be very few.

Issue Links

• is blocked by: OOZIE-2179 Use HDFS INotify to track HDFS data dependencies instead of polling