Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2348

Recovery service keeps on recovering coord action of suspended jobs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.3.0
    • Component/s: None
    • Labels:
      None

      Description

               @NamedQuery(name = "GET_READY_ACTIONS_GROUP_BY_JOBID", query = "select a.jobId, min(a.lastModifiedTimestamp) from CoordinatorActionBean a where a.statusStr = 'READY' group by a.jobId having min(a.lastModifiedTimestamp) < :lastModifiedTime"),
      
              @NamedQuery(name = "GET_COORD_ACTIONS_WAITING_SUBMITTED_OLDER_THAN", query = "select a.id, a.jobId, a.statusStr, a.externalId, a.pushMissingDependencies from CoordinatorActionBean a where (a.statusStr = 'WAITING' OR a.statusStr = 'SUBMITTED') AND a.lastModifiedTimestamp <= :lastModifiedTime"),
      
      

      Even if job is suspended, Recovery service will keep on queuing ready, waiting, ready command.

      1. amend-OOZIE-2348-V1.patch
        2 kB
        Purshotam Shah
      2. OOZIE-2348-V1.patch
        17 kB
        Purshotam Shah
      3. OOZIE-2348-V2.patch
        26 kB
        Purshotam Shah
      4. OOZIE-2348-V2.patch
        18 kB
        Purshotam Shah

        Activity

        Hide
        rohini Rohini Palaniswamy added a comment -
        +                        else if (caction.getStatus() == CoordinatorActionBean.Status.READY) {
        +                            if (caction.getExternalId() != null) {
        +                                readyJobs.add(caction.getJobId());
        +                            }
        +                        }
        

        is not correct. Need to remove the externalId != null check. Can you also delete CoordActionsGetReadyGroupbyJobIDJPAExecutor and CoordActionsGetForRecoveryJPAExecutor classes?

        Show
        rohini Rohini Palaniswamy added a comment - + else if (caction.getStatus() == CoordinatorActionBean.Status.READY) { + if (caction.getExternalId() != null ) { + readyJobs.add(caction.getJobId()); + } + } is not correct. Need to remove the externalId != null check. Can you also delete CoordActionsGetReadyGroupbyJobIDJPAExecutor and CoordActionsGetForRecoveryJPAExecutor classes?
        Hide
        rohini Rohini Palaniswamy added a comment -

        Also remove GET_READY_ACTIONS_GROUP_BY_JOBID from CoordinatorActionBean.java

        Show
        rohini Rohini Palaniswamy added a comment - Also remove GET_READY_ACTIONS_GROUP_BY_JOBID from CoordinatorActionBean.java
        Hide
        puru Purshotam Shah added a comment - - edited

        Can you also delete CoordActionsGetReadyGroupbyJobIDJPAExecutor and CoordActionsGetForRecoveryJPAExecutor classes?

        I did. looks like git diff, didn't include delete file.. will upload new patch.

        Show
        puru Purshotam Shah added a comment - - edited Can you also delete CoordActionsGetReadyGroupbyJobIDJPAExecutor and CoordActionsGetForRecoveryJPAExecutor classes? I did. looks like git diff, didn't include delete file.. will upload new patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        Testing JIRA OOZIE-2348

        Cleaning local git workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        -1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . -1 the patch contains 1 line(s) longer than 132 characters
        . +1 the patch does adds/modifies 1 testcase(s)
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        +1 TESTS
        . Tests run: 1690
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        -1 Overall result, please check the reported -1(s)

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/2525/

        Show
        hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2348 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . -1 the patch contains 1 line(s) longer than 132 characters . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1690 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2525/
        Hide
        rohini Rohini Palaniswamy added a comment -

        -1 the patch contains 1 line(s) longer than 132 characters

        This is named query

        +1. Thanks for the cleanup of the JPA classes.

        Show
        rohini Rohini Palaniswamy added a comment - -1 the patch contains 1 line(s) longer than 132 characters This is named query +1. Thanks for the cleanup of the JPA classes.
        Hide
        puru Purshotam Shah added a comment -
                    queue(new CoordPushDependencyCheckXCommand(coordAction.getId()), nominalTime.getTime() - currentTime
                            .getTime()), getCoordPushCheckRequeueInterval());
        

        Requeue logic is not correct. if current time is 23:55:00 and nominal time is 23:55:01 and coord push reueue interval is 10 min. Then command will get requeued for 10 min which is not correct. Ditto for CoordActionInputCheckXCommand.

        Show
        puru Purshotam Shah added a comment - queue( new CoordPushDependencyCheckXCommand(coordAction.getId()), nominalTime.getTime() - currentTime .getTime()), getCoordPushCheckRequeueInterval()); Requeue logic is not correct. if current time is 23:55:00 and nominal time is 23:55:01 and coord push reueue interval is 10 min. Then command will get requeued for 10 min which is not correct. Ditto for CoordActionInputCheckXCommand.
        Hide
        rohini Rohini Palaniswamy added a comment -

        +1 for +1 for amend-OOZIE-2348-V1.patch

        Show
        rohini Rohini Palaniswamy added a comment - +1 for +1 for amend- OOZIE-2348 -V1.patch
        Hide
        hadoopqa Hadoop QA added a comment -

        Testing JIRA OOZIE-2348

        Cleaning local git workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        -1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 132
        . -1 the patch does not add/modify any testcase
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        -1 TESTS - patch does not compile, cannot run testcases
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        -1 Overall result, please check the reported -1(s)

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/2541/

        Show
        hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2348 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2541/
        Hide
        hadoopqa Hadoop QA added a comment -

        Testing JIRA OOZIE-2348

        Cleaning local git workspace

        ----------------------------

        +1 PATCH_APPLIES
        +1 CLEAN
        -1 RAW_PATCH_ANALYSIS
        . +1 the patch does not introduce any @author tags
        . +1 the patch does not introduce any tabs
        . +1 the patch does not introduce any trailing spaces
        . +1 the patch does not introduce any line longer than 132
        . -1 the patch does not add/modify any testcase
        +1 RAT
        . +1 the patch does not seem to introduce new RAT warnings
        +1 JAVADOC
        . +1 the patch does not seem to introduce new Javadoc warnings
        +1 COMPILE
        . +1 HEAD compiles
        . +1 patch compiles
        . +1 the patch does not seem to introduce new javac warnings
        +1 BACKWARDS_COMPATIBILITY
        . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
        . +1 the patch does not modify JPA files
        +1 TESTS
        . Tests run: 1691
        +1 DISTRO
        . +1 distro tarball builds with the patch

        ----------------------------
        -1 Overall result, please check the reported -1(s)

        The full output of the test-patch run is available at

        . https://builds.apache.org/job/oozie-trunk-precommit-build/2543/

        Show
        hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2348 Cleaning local git workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1691 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/2543/
        Hide
        hadoopqa Hadoop QA added a comment -

        Testing JIRA OOZIE-2348

        Cleaning local git workspace

        ----------------------------

        -1 Patch failed to apply to head of branch

        ----------------------------

        Show
        hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-2348 Cleaning local git workspace ---------------------------- -1 Patch failed to apply to head of branch ----------------------------
        Hide
        rohini Rohini Palaniswamy added a comment -

        Purshotam Shah,
        Is this jira or any of the amend patches not checked in? Please verify and close the jira.

        Show
        rohini Rohini Palaniswamy added a comment - Purshotam Shah , Is this jira or any of the amend patches not checked in? Please verify and close the jira.
        Hide
        rkanter Robert Kanter added a comment -

        Closing issue; Oozie 4.3.0 is released.

        Show
        rkanter Robert Kanter added a comment - Closing issue; Oozie 4.3.0 is released.

          People

          • Assignee:
            puru Purshotam Shah
            Reporter:
            puru Purshotam Shah
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development