[FALCON-1260] Instance dependency API produces incorrect results - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 0.7
Component/s: feed, process
Labels:
None
Environment:

QA

Description

I have the following definitions :
Process :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<process name="InstanceDependencyTest--agregator-coord16-4adb02c6" xmlns="uri:falcon:process:0.1">
<clusters>
<cluster name="InstanceDependencyTest--corp-a5b30313">
<validity start="2015-06-06T09:37Z" end="2015-06-06T10:37Z"/>
</cluster>
</clusters>
<parallel>5</parallel>
<order>FIFO</order>
<frequency>minutes(10)</frequency>
<timezone>UTC</timezone>
<inputs>
<input name="inputData" feed="InstanceDependencyTest--raaw-logs16-38bc46f1" start="now(0,-20)" end="now(0,0)"/>
</inputs>
<outputs>
<output name="outputData" feed="InstanceDependencyTest--agregated-logs16-e6cfefc5" instance="now(0,0)"/>
</outputs>
<properties>
<property name="queueName" value="default"/>
<property name="fileTime" value="${formatTime(dateOffset(instanceTime(), 1, 'DAY'), 'yyyy-MMM-dd')}"/>
<property name="user" value="${user()}"/>
</properties>
<workflow path="/tmp/falcon-regression/InstanceDependencyTest/aggregator"/>
<retry policy="periodic" delay="minutes(3)" attempts="3"/>
<ACL owner="pragya" group="dataqa" permission="*"/>
</process>

Input feed :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="InstanceDependencyTest--raaw-logs16-38bc46f1" description="clicks log" xmlns="uri:falcon:feed:0.1">
<partitions>
<partition name="country"/>
<partition name="colo"/>
</partitions>
<frequency>minutes(5)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(6)"/>
<clusters>
<cluster name="InstanceDependencyTest--corp-a5b30313" type="source">
<validity start="2009-02-01T00:00Z" end="2099-05-01T00:00Z"/>
<retention limit="months(9000)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data" path="/tmp/falcon-regression/InstanceDependencyTest/input/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
<location type="stats" path="/projects/falcon/clicksStats"/>
<location type="meta" path="/projects/falcon/clicksMetaData"/>
</locations>
<ACL owner="pragya" group="dataqa" permission="*"/>
<schema location="/schema/clicks" provider="protobuf"/>
<properties>
<property name="field1" value="value1"/>
<property name="field2" value="value2"/>
</properties>
</feed>

Output feed:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="InstanceDependencyTest--agregated-logs16-e6cfefc5" description="clicks log" xmlns="uri:falcon:feed:0.1">
<frequency>minutes(5)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(6)"/>
<clusters>
<cluster name="InstanceDependencyTest--corp-a5b30313" type="source">
<validity start="2009-02-01T01:00Z" end="2099-05-01T00:00Z"/>
<retention limit="hours(6)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data" path="/tmp/falcon-regression/InstanceDependencyTest/output-data/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
<location type="stats" path="/projects/falcon/clicksStats"/>
<location type="meta" path="/projects/falcon/clicksMetaData"/>
</locations>
<ACL owner="pragya" group="dataqa" permission="*"/>
<schema location="/schema/clicks" provider="protobuf"/>
<properties>
<property name="field1" value="value1"/>
<property name="field2" value="value2"/>
</properties>
</feed>

Cluster :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cluster name="InstanceDependencyTest--corp-a5b30313" description="" colo="ua1" xmlns="uri:falcon:cluster:0.1">
<interfaces>
<interface type="readonly" endpoint="hdfs://192.168.138.200:8020" version="0.20.2"/>
<interface type="write" endpoint="hdfs://192.168.138.200:8020" version="0.20.2"/>
<interface type="execute" endpoint="192.168.138.200:8032" version="0.20.2"/>
<interface type="workflow" endpoint="http://192.168.138.200:11000/oozie/" version="3.1.0"/>
<interface type="messaging" endpoint="tcp://192.168.138.200:61616?daemon=true" version="5.1.6"/>
</interfaces>
<locations>
<location name="staging" path="/tmp/falcon-regression-staging"/>
<location name="working" path="/tmp/falcon-regression-working"/>
<location name="temp" path="/tmp"/>
</locations>
<ACL owner="pragya" group="dataqa" permission="*"/>
<properties>
<property name="field1" value="value1"/>
<property name="field2" value="value2"/>
</properties>
</cluster>

Usage :
1. Output feed :

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2015-06-06T09:40Z
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:40Z, tag: Output


Response: ua1/Success!

Request Id: ua1/1610525991@qtp-1766505436-0 - deabb3d7-fc7e-474a-b315-50da006ebf7a

Although process instance doesnot exist, it says it exists

2.InstanceTime in process validaity range but instance doesnt exist
Input feed:

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2015-06-06T09:40Z
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:56Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:45Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:42Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:49Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:52Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:50Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:57Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:54Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:55Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:53Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:43Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:41Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:51Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:48Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:47Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:46Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:44Z, tag: Input


Response: ua1/Success!

Request Id: ua1/1610525991@qtp-1766505436-0 - 8b49d24a-1654-423c-a591-68de039a41f0

Output feed :

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2015-06-06T09:40Z
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2015-06-06T09:40Z, tag: Output


Response: ua1/Success!

Request Id: ua1/1610525991@qtp-1766505436-0 - c88476ad-0bff-44f4-b43d-82b846e9b1cf

3. InstanceTime < process start time

Input feed:

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2014-06-06T08:40Z
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:42Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:43Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:52Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:41Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:45Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:57Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:54Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:44Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:47Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:49Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:46Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:56Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:55Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:48Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:53Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:50Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2014-06-06T08:51Z, tag: Input


Response: ua1/Success!

Request Id: ua1/1610525991@qtp-1766505436-0 - 7e39a727-651a-4efb-925b-2a65aa8b6b

Output feed:

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2014-06-06T08:40Z
ERROR: Bad Request;<?xml version="1.0" encoding="UTF-8" standalone="yes"?><dependents><status>FAILED</status><message>ua1/org.apache.falcon.FalconException::javax.ws.rs.WebApplicationException: javax.xml.bind.UnmarshalException: unexpected element (uri:&quot;&quot;, local:&quot;instancesResult&quot;). Expected elements are &lt;{}dependents&gt;,&lt;{}result&gt;
</message><requestId>ua1/339141107@qtp-212921632-26 - 998c331e-188a-4ca3-88da-bd1a83bc40bb
</requestId></dependents>

4. InstanceTime > process end time

Input feed:

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--raaw-logs16-38bc46f1 -instanceTime 2016-06-06T08:40Z
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:56Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:49Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:51Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:48Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:55Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:46Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:50Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:53Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:45Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:43Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:42Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:41Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:52Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:57Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:54Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:47Z, tag: Input
name: InstanceDependencyTest--agregator-coord16-4adb02c6, type: PROCESS, cluster: InstanceDependencyTest--corp-a5b30313, instanceTime: 2016-06-06T08:44Z, tag: Input


Response: ua1/Success!

Request Id: ua1/1610525991@qtp-1766505436-0 - 7cc314ab-9b3a-413d-9024-eeb5804a88db

Output feed :

dataqa@ip-192-168-138-200:/mnt/git/falcon/prism$ bin/falcon instance -type feed -dependency -name InstanceDependencyTest--agregated-logs16-e6cfefc5 -instanceTime 2016-06-06T08:40Z
ERROR: Bad Request;<?xml version="1.0" encoding="UTF-8" standalone="yes"?><dependents><status>FAILED</status><message>ua1/org.apache.falcon.FalconException::javax.ws.rs.WebApplicationException: javax.xml.bind.UnmarshalException: unexpected element (uri:&quot;&quot;, local:&quot;instancesResult&quot;). Expected elements are &lt;{}dependents&gt;,&lt;{}result&gt;
</message><requestId>ua1/339141107@qtp-212921632-26 - c98b92b3-a34a-4f57-97c7-418727c338ab
</requestId></dependents>

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

FALCON-1260-v1.patch
29/Jun/15 06:58
49 kB
Ajay Yadav
FALCON-1260.patch
15/Jun/15 08:44
44 kB
Ajay Yadav

Issue Links

is related to

FALCON-1039 Add instance dependency API in falcon

Closed

relates to

FALCON-1268 Instance Dependency API failure message is not intuitive in distributed mode

Resolved

links to

Review Board Link

Activity

People

Assignee:: Ajay Yadav

Reporter:: Pragya Mittal

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 08/Jun/15 07:00

Updated:: 29/Jun/15 09:10

Resolved:: 29/Jun/15 09:10