[FALCON-494] update feature broken

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.6
• Fix Version/s: 0.4, 0.5
• Component/s: general

Environment

QA

• Assignee: Shwetha GS
• Reporter: Samarth Gupta
• Created: 25/Jun/14 07:50
• Updated: 08/Jul/14 08:59
• Resolved: 08/Jul/14 08:59

Description

when we update an process / feed , the update take place from current time, and all the future instance of the old coordinator are moved to new coord with new definition.

However with the latest falcon, the coord which is getting created is having all instances which have have been already processed by old coodinator. Resulting in rerunning on those instances which are already in RUNNING / READY / WAITING state in old coord.

in the logs, following lines can be seen which says the new start time :

2014-06-25 06:36:40,819 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Submitted 0000328-140624044727425-oozie-oozi-B on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e with properties: {nameNode=hdfs://192.168.138.182:8020, oozie.use.system.libpath=true, field2=value2, jobTracker=192.168.138.182:8021, field1=value1, falcon.libpath=/projectsTest/ivory/working/lib, colo.name=ua3, oozie.bundle.application.path=${nameNode}/projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513, user.name=dataqa} (OozieWorkflowEngine:1316)
2014-06-25 06:36:40,827 INFO  - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ New bundle 0000328-140624044727425-oozie-oozi-B scheduled successfully with start time 2014-06-25T06:39Z (OozieWorkflowEngine:1131)
2014-06-25 06:36:40,828 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Updating endtime of coord 0000323-140624044727425-oozie-oozi-C to 2014-06-25T06:39Z on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e (OozieWorkflowEngine:1055)

but in the coord xml the start time is till the original process time :

2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ Writing definition to /projects/ivory/staging/falcon/workflows/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395/1403678197513/DEFAULT/coordinator.xml on cluster corp-2babc92c-129f-422e-b836-17e286eeaf0111-9b497e34-e892-4e9b-a5ee-f9a417e55f1e (OozieWorkflowBuilder:338)
2014-06-25 06:36:39,925 DEBUG - [1871593525@qtp-1222207508-0:dataqa:POST//sync/update/process/agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395 1118a58a-4a4e-4052-afea-40d4960470f3] ~ <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<coordinator-app name="FALCON_PROCESS_DEFAULT_agregator-coord16-bf1e8076-16fe-4fba-bb76-b780ec3a05b0-e92d8b59-42ef-41ab-9625-596df66d2395" frequency="${coord:minutes(1)}" start="2014-06-25T06:34Z" end="2014-06-25T06:44Z" timezone="UTC" xmlns="uri:oozie:coordinator:0.3">
    <controls>
        <timeout>30</timeout>
        <concurrency>1</concurrency>
        <execution>FIFO</execution>
        <throttle>60</throttle>
    </controls>
    <datasets>
        <dataset name="inputData" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z" timezone="UTC">
            <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
            <done-flag></done-flag>
        </dataset>
        <dataset name="inputData2" frequency="${coord:minutes(1)}" initial-instance="2009-02-01T00:00Z" timezone="UTC">
            <uri-template>hdfs://192.168.138.182:8020/tmp/falcon-regression/NewPrismProcessUpdateTest/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}</uri-template>
            <done-flag></done-flag>
        </dataset>

most of the regression NewPrismProcessUpdateTest is failing coz of this.

Issue Links

• is broken by: FALCON-430 Process update with user