Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3612

CompositeCallable became empty after requeue and cause IndexOutOfBoundException in getType() method, which cause counters for this type in activeCallables never be descreased and exceed concurrency

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 4.2.0
    • Fix Version/s: None
    • Component/s: action, workflow
    • Labels:
      None
    • Environment:

      We use oozie 4.2.0. but in the newest release this problems seems also exists

      Description

      This should rarely happens but it just happened in out environment twice. The process is listed below.

      1. we have long hang actions which last serveral hours (This is and error sitution, task execute quickly in normal time). the action type is developed by ourself by extend the Executor.
      2. These actions's ActionCheckXCommand is put into the callable queue by composited into an array with max ten element
      3. Before really put into the thread pool for execute, they will be filtered if element is already existed ,  the existence is identified by the uniqueCallables. So the real CompositeCallable in queue may actually have element less then 10.
      4. When this CompositeCallable is poll form queue. Before execute it will check the concurreny for the action_check type. if the concurrency is reached, it will be requeued. 
      5. In the requeue procedure ,  the ActionCheckService happens to already put these ActionCheckXCommand in queue, so the CompositeCallable is filtered with 0 element.
      6. In the finally procedure, the CallableEnd method will need to descrease the counter of this type . But IndexOutOfBounds happens when called on the empty CompositeCallable's getType method, which cause the counter will never be descreased
      7. If it happens for the maxComcurrency time, the counter itself will exceed the maxConcurrency and this type can never be taked outside of the queue,  they will lived in queue forever, which cause the workflow hang. 

       

      The pic  where the exception happened is attached. Hopes it describe clearly. 

        Attachments

        1. eception_in_composite_callable.png
          93 kB
          yuanyimeng
        2. counternotdescreaedifgettypeexceptionhappen.png
          79 kB
          yuanyimeng
        3. exceptionlog.png
          101 kB
          yuanyimeng
        4. callableRun.png
          133 kB
          yuanyimeng

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              wineternity yuanyimeng
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: