Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-630

QueryMasterTask never finished when Internal error occurs.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: None
    • Labels:
      None

      Description

      Query state is changed to QUERY_ERROR when QueryEventType.INTERNAL_ERROR occurs, and QueryFinishEventHandler invoked.
      However, there is no check logic for QUERY_ERROR. So stopQuery() never called.
      QueryMasterTask.java

        private class QueryFinishEventHandler implements EventHandler<QueryMasterQueryCompletedEvent> {
          @Override
          public void handle(QueryMasterQueryCompletedEvent event) {
            QueryId queryId = event.getQueryId();
            LOG.info("Query completion notified from " + queryId);
      
            while (!isTerminatedState(query.getState())) {
              try {
                synchronized (this) {
                  wait(10);
                }
              } catch (InterruptedException e) {
                LOG.error(e);
              }
            }
            LOG.info("Query final state: " + query.getState());
            queryMasterContext.stopQuery(queryId);
          }
      
          private boolean isTerminatedState(QueryState state) {
            return
                state == QueryState.QUERY_SUCCEEDED ||
                state == QueryState.QUERY_FAILED ||
                state == QueryState.QUERY_KILLED;
          }
        }
      
      1. TAJO-630.patch
        0.8 kB
        Hyunsik Choi

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #4 (See https://builds.apache.org/job/Tajo-0.8.0-build/4/)
        TAJO-630: QueryMasterTask never finished when Internal error occurs. (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=5178ac28bc58e394e8d5d879e7acebc1913bff3a)

        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #4 (See https://builds.apache.org/job/Tajo-0.8.0-build/4/ ) TAJO-630 : QueryMasterTask never finished when Internal error occurs. (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=5178ac28bc58e394e8d5d879e7acebc1913bff3a ) CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #97 (See https://builds.apache.org/job/Tajo-master-build/97/)
        TAJO-630: QueryMasterTask never finished when Internal error occurs. (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=15c4576ef680e05be877e9bb5b4370a03857c446)

        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #97 (See https://builds.apache.org/job/Tajo-master-build/97/ ) TAJO-630 : QueryMasterTask never finished when Internal error occurs. (hyunsik: https://git-wip-us.apache.org/repos/asf/?p=incubator-tajo.git&a=commit&h=15c4576ef680e05be877e9bb5b4370a03857c446 ) CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for the review. committed it to master branch and branch-0.8.0.

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for the review. committed it to master branch and branch-0.8.0.
        Hide
        jhkim Jinho Kim added a comment -

        +1 for the patch.

        Show
        jhkim Jinho Kim added a comment - +1 for the patch.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Could you anyone can review this?

        Show
        hyunsik Hyunsik Choi added a comment - Could you anyone can review this?
        Hide
        hyunsik Hyunsik Choi added a comment -

        This patch only fixes the bug of QueryFinishEventHandler::isTerminatedState.

        From your commented report, I also found wrong exceptional event handling problem in QueryUnitAttempt. I'll create a separate jira issue for that.

        Show
        hyunsik Hyunsik Choi added a comment - This patch only fixes the bug of QueryFinishEventHandler::isTerminatedState. From your commented report, I also found wrong exceptional event handling problem in QueryUnitAttempt. I'll create a separate jira issue for that.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/18514/

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/18514/
        Hide
        hyunsik Hyunsik Choi added a comment -

        Thank you for your report. It's definitely a bug. I'll fix it ASAP.

        Show
        hyunsik Hyunsik Choi added a comment - Thank you for your report. It's definitely a bug. I'll fix it ASAP.
        Hide
        hjkim Hyoungjun Kim added a comment -

        When the failure occurred, the following logged in the QueryMaster.

        2014-02-23 22:25:55,617 ERROR querymaster.QueryUnitAttempt (QueryUnitAttempt.java:handle(425)) - Can't handle this event at current state of ta_1393161894205_0001_000004_000101_00)
        org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_UPDATE at TA_ASSIGNED
                at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388)
                at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
                at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
                at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
                at org.apache.tajo.master.querymaster.QueryUnitAttempt.handle(QueryUnitAttempt.java:423)
                at org.apache.tajo.master.querymaster.QueryMasterTask$TaskAttemptEventDispatcher.handle(QueryMasterTask.java:241)
                at org.apache.tajo.master.querymaster.QueryMasterTask$TaskAttemptEventDispatcher.handle(QueryMasterTask.java:233)
                at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137)
                at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)
        
        Show
        hjkim Hyoungjun Kim added a comment - When the failure occurred, the following logged in the QueryMaster. 2014-02-23 22:25:55,617 ERROR querymaster.QueryUnitAttempt (QueryUnitAttempt.java:handle(425)) - Can't handle this event at current state of ta_1393161894205_0001_000004_000101_00) org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: TA_UPDATE at TA_ASSIGNED at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:388) at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.tajo.master.querymaster.QueryUnitAttempt.handle(QueryUnitAttempt.java:423) at org.apache.tajo.master.querymaster.QueryMasterTask$TaskAttemptEventDispatcher.handle(QueryMasterTask.java:241) at org.apache.tajo.master.querymaster.QueryMasterTask$TaskAttemptEventDispatcher.handle(QueryMasterTask.java:233) at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137) at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hjkim Hyoungjun Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development