Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
Description
We saw a weird instance where multiple coords were created for same bundle id when the bundle was supposed to have just 1 coordinator. Here are the oozie logs:
2013-11-19 09:09:46,473 INFO BundleStartXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING 2013-11-19 09:09:46,473 WARN BundleStartXCommand:542 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] E1100: Command precondition does not hold before execution, [Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING], Error Code: E1100 2013-11-19 09:09:46,473 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit 2013-11-19 09:09:46,483 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist 2013-11-19 09:09:46,515 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C 2013-11-19 09:09:46,529 INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [0] 2013-11-19 09:09:46,535 INFO CoordMaterializeTransitionXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for tz=Coordinated Universal Time, 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING' 2013-11-19 09:09:54,590 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to FALSE 2013-11-19 09:10:16,326 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'SUSPENDED' from 'SUSPENDED' 2013-11-19 09:12:57,246 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE 2013-11-19 09:13:16,410 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:16:16,446 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:17:00,913 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING' 2013-11-19 09:17:00,914 INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE 2013-11-19 09:19:16,490 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:22:16,907 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:25:17,086 INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C numWaitingActions : 0 MatThrottle : 60 2013-11-19 09:26:49,373 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit 2013-11-19 09:26:49,383 INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist 2013-11-19 09:26:49,438 INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C 2013-11-19 09:26:49,445 INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [1]