[OOZIE-699] Support for EL function to append pathFilter to all the paths materialized for coord:dataIn('inPath-event')

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: el functions

Environment

Oozie 3.0.2, CDH3U0.

• Assignee: Unassigned
• Reporter: Shaik Idris Ali
• Created: 16/Feb/12 06:32
• Updated: 16/Feb/12 06:32

Description

Hi currently, the coordinator does not support gating/waiting on URI with patterns/filters.

My dataset in coordinator.xml is:
<datasets><dataset name="inPath" frequency="${coord:hours(1)}" initial-instance="2012-01-30T00:00Z" timezone="UTC"><uri-template>${nameNode}/projects/bi/ioout/${YEAR}${MONTH}${DAY}-${HOUR}</uri-template><done-flag></done-flag></dataset>

My input-event is:
<input-events><data-in name="inPath-event" dataset="inPath">
<start-instance>${ivory:now(-25,0)}</start-instance>
<end-instance>${ivory:now(0,0)}
</end-instance></data-in></input-events>

However I would like to pass, the materialized events with appended filter (/clickon///OK//) to workflow:
<property><name>inPath</name><value>${coord:dataIn('inPath-event')}/clickon///OK//</value>
workflow can have actions actions as pig script or M/R Job.

The resolved URIs are:
hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-07,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-06,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-05,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-04,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-03,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-02,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-01,hdfs://shaik-idris:8020/projects/bi/ioout/2012-01-30-00/clickon///OK//

i.e. the pathFilter or pattern is appended to only last URI.

It would be good if we have an coordinator EL function:
coord:appendPathFilter(coord:dataIn('inPath-event'),/clickon///OK//)

This is a very common usecase while working with pig scripts or M/R jobs.

Thanks,
-Idris