Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2397

LAST_ONLY and NONE don't properly handle READY actions

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 4.2.0
    • Fix Version/s: 4.3.0
    • Component/s: core
    • Labels:
      None

      Description

      When using LAST_ONLY or NONE, actions are supposed to be able to transition from READY to SKIPPED if the right criteria are met, but they don't. This is in contrast to the timeout feature, which does not.

      Here's a more detailed technical description of the problem:
      We handle LAST_ONLY in CoordMaterializeTransitionXCommand and CoordActionInputCheckXCommand. The former deals with materializing the actions and the behavior to set "old" actions to SKIPPED when materializing them. The latter deals with checking the input datasets for actions and the behavior to determine if a WAITING action is ready to transition to READY (deps are met) and all that entails, including changing status to READY and queuing a CoordActionReadyXCommand. If the deps are not met and the dataset is not there yet, it will queue itself at some delay. So, these only handle the materialization and WAITING states. However, LAST_ONLY is supposed to also do READY --> SKIPPED if it's condition is met (unlike TIMEDOUT, which can only come from WAITING; this additional difference should probably be called out in the docs).

      CoordActionReadyXCommand needs to be updated to handle LAST_ONLY. It currently treats LAST_ONLY the same as LIFO (via CoordJobGetReadyActionsJPAExecutor), where the order is the only difference from FIFO. After retrieving all READY actions, it should check if any meet their LAST_ONLY condition, and if so, queue a CoordActionSkipXCommand for them (maybe make a bulk version?) instead of a CoordActionStartXCommand.

      We have the same issue with NONE, which has similar behavior.

        Attachments

        1. OOZIE-2397.003.patch
          70 kB
          Robert Kanter
        2. OOZIE-2397.002.patch
          67 kB
          Robert Kanter
        3. OOZIE-2397.001.patch
          43 kB
          Robert Kanter

          Issue Links

            Activity

              People

              • Assignee:
                rkanter Robert Kanter
                Reporter:
                rkanter Robert Kanter
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: