[OOZIE-2179] Use HDFS INotify to track HDFS data dependencies instead of polling

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: coordinator
• Assignee: Unassigned
• Reporter: Robert Kanter
• Created: 20/Mar/15 20:06
• Updated: 09/Jun/15 23:38

Description

Instead of polling the NN every minute for Coordinators, we should look into using the new INotify feature in HDFS-6634. It allows you to get a stream of events from HDFS. Internally, it still uses a polling mechanism for now, but even so, it would likely be more efficient and less heavy-handed than what we're doing.

We'd probably still have to check if the directory exists when a coordinator action starts in case we missed the event, but while waiting for an HDFS dependency to be available, we can use INotify.

For HCat dependencies we still have a backup polling of 10 minutes in case a JMS message is missed or lost. I don't think we'll need to do this for INotify because you can view past events as long as you keep track of the event ID. For example, if you restart Oozie and we kept track of the last ID Oozie looked at, we could resume from there without losing anything.

The INotify stream is asynchronous, so we won't receive a notification immediately. We should look into the guarantees of how long it can take for the notification to show up.

Issue Links

• blocks: OOZIE-2216 Aperiodic Data handling in oozie
• depends upon: HDFS-6634 inotify in HDFS
• relates to: OOZIE-2265 Use hdfs inotify to update sharelib symlink