Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-587

Query is hanging when OutOfMemoryError occurs in the query master

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: TajoMaster
    • Labels:
      None

      Description

      See the title. When I run a simple sort query against a table of 1TB, the query is hanging and not finished.
      Queries should be terminated immediately when OOME occurs.

      tajo> select l_orderkey from lineitem order by l_orderkey
      
      2014-02-05 17:20:52,339 FATAL master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:dispatch(143)) - Error in dispatcher thread:SUBQUERY_COMPLETED
      java.lang.OutOfMemoryError: GC overhead limit exceeded
              at java.net.URI.create(URI.java:857)
              at org.apache.tajo.master.querymaster.Repartitioner.scheduleRangeShuffledFetches(Repartitioner.java:342)
              at org.apache.tajo.master.querymaster.Repartitioner.scheduleFragmentsForNonLeafTasks(Repartitioner.java:261)
              at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.schedule(SubQuery.java:680)
              at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:523)
              at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:504)
              at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
              at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
              at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
              at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
              at org.apache.tajo.master.querymaster.SubQuery.handle(SubQuery.java:481)
              at org.apache.tajo.master.querymaster.Query$SubQueryCompletedTransition.executeNextBlock(Query.java:311)
              at org.apache.tajo.master.querymaster.Query$SubQueryCompletedTransition.transition(Query.java:357)
              at org.apache.tajo.master.querymaster.Query$SubQueryCompletedTransition.transition(Query.java:297)
              at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
              at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
              at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
              at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
              at org.apache.tajo.master.querymaster.Query.handle(Query.java:584)
              at org.apache.tajo.master.querymaster.Query.handle(Query.java:58)
              at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137)
              at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)
              at java.lang.Thread.run(Thread.java:701)
      2014-02-05 17:20:52,339 WARN  querymaster.QueryMaster (QueryMaster.java:run(459)) - Query q_1391587770871_0001 stopped cause query sesstion timeout: 384113 ms
      2014-02-05 17:20:52,339 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(168)) - Stopping QueryMasterTask:q_1391587770871_0001
      2014-02-05 17:20:52,346 INFO  master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1391587770871_0001
      2014-02-05 17:20:52,351 INFO  querymaster.QueryMasterTask (QueryMasterTask.java:stop(198)) - Stopped QueryMasterTask:q_1391587770871_0001
      2014-02-05 17:23:28,614 ERROR worker.TajoWorker (SignalLogger.java:handle(60)) - RECEIVED SIGNAL 15: SIGTERM
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jihoonson Jihoon Son
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: