Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1622

Multiple CoordSubmit for same bundle

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.1.0
    • Component/s: None
    • Labels:
      None

      Description

      We saw a weird instance where multiple coords were created for same bundle id when the bundle was supposed to have just 1 coordinator. Here are the oozie logs:

      2013-11-19 09:09:46,473  INFO BundleStartXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING
      2013-11-19 09:09:46,473  WARN BundleStartXCommand:542 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] E1100: Command precondition does not hold before execution, [Bundle 0484436-131016085136608-oozie-oozi-B is not in PREP status. It is in : RUNNING], Error Code: E1100
      2013-11-19 09:09:46,473  INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit
      2013-11-19 09:09:46,483  INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist 
      2013-11-19 09:09:46,515  INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484437-131016085136608-oozie-oozi-C
      2013-11-19 09:09:46,529  INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [0]
      2013-11-19 09:09:46,535  INFO CoordMaterializeTransitionXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484437-131016085136608-oozie-oozi-C] ACTION[-] materialize actions for tz=Coordinated Universal Time,
      2013-11-19 09:09:54,590  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING'
      2013-11-19 09:09:54,590  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to FALSE
      2013-11-19 09:10:16,326  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:12:57,246  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'SUSPENDED' from 'SUSPENDED'
      2013-11-19 09:12:57,246  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE
      2013-11-19 09:13:16,410  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:16:16,446  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:17:00,913  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Set bundle job [0484436-131016085136608-oozie-oozi-B] status to 'RUNNING' from 'RUNNING'
      2013-11-19 09:17:00,914  INFO StatusTransitService$StatusTransitRunnable:539 - USER[-] GROUP[-] Bundle job [0484436-131016085136608-oozie-oozi-B] Pending set to TRUE
      2013-11-19 09:19:16,490  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:22:16,907  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:25:17,086  INFO CoordMaterializeTriggerService$CoordMaterializeTriggerRunnable:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] Job :0484437-131016085136608-oozie-oozi-C  numWaitingActions : 0 MatThrottle : 60
      2013-11-19 09:26:49,373  INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] STARTED Coordinator Submit
      2013-11-19 09:26:49,383  INFO CoordSubmitXCommand:539 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0484436-131016085136608-oozie-oozi-B] ACTION[-] configDefault Doesn't exist
      2013-11-19 09:26:49,438  INFO CoordSubmitXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] ENDED Coordinator Submit jobId=0484598-131016085136608-oozie-oozi-C
      2013-11-19 09:26:49,445  INFO BundleStatusUpdateXCommand:539 - USER[fetl] GROUP[-] TOKEN[] APP[<app name>] JOB[0484598-131016085136608-oozie-oozi-C] ACTION[-] Updated bundle action [0484436-131016085136608-oozie-oozi-B_<app name>] from prev status [PREP] to current coord status [PREP], and new bundle action pending [1]
      

        Attachments

        1. OOZIE-1622.patch
          19 kB
          Shwetha GS

          Activity

            People

            • Assignee:
              shwethags Shwetha GS
              Reporter:
              shwethags Shwetha GS
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: