Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-655

QueryMaster sent "Select query" command to TajoWorker,but the TajoWorker don't working

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: QueryMaster, Worker
    • Labels:
      None
    • Environment:

      CentOS 6.3 32bit,jdk 1.7,hadoop 2.2.0

      Description

      Hi

      Yesterday ,You provide tajo branch-0.8.0 already installed. In the http://xxx:26080 web ui,the TajaWorker is live .In the TSQL interface input "select * from test" command,but Prompt "Internal error!" error.

      TajoWorker error log content:
      =======================================================
      Block Id: eb_1393922377833_0005_000001 [ROOT]
      =======================================================

      SCAN(0) on test
      => target list: test.id (INT4), test.interface (TEXT), test.start_time (TEXT), test.byte (TEXT)
      => out schema:

      {(4) test.id (INT4),test.interface (TEXT),test.start_time (TEXT),test.byte (TEXT)}
      => in schema: {(4) test.id (INT4),test.interface (TEXT),test.start_time (TEXT),test.byte (TEXT)}

      =======================================================
      Block Id: eb_1393922377833_0005_000002 [TERMINAL]
      =======================================================

      2014-03-05 10:48:18,219 INFO master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:start(101)) - AsyncDispatcher started:q_1393922377833_0005
      2014-03-05 10:48:18,222 INFO querymaster.Query (Query.java:handle(641)) - Processing q_1393922377833_0005 of type START
      2014-03-05 10:48:18,223 INFO querymaster.SubQuery (SubQuery.java:initTaskScheduler(661)) - org.apache.tajo.master.DefaultTaskScheduler is chosen for the task scheduling
      2014-03-05 10:48:18,230 INFO storage.AbstractStorageManager (AbstractStorageManager.java:listStatus(382)) - Total input paths to process : 1
      2014-03-05 10:48:18,236 ERROR querymaster.SubQuery (SubQuery.java:transition(645)) - SubQuery (eb_1393922377833_0005_000001) ERROR:
      java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
      at org.apache.tajo.storage.AbstractStorageManager.newScannerInstance(AbstractStorageManager.java:661)
      at org.apache.tajo.storage.StorageManager.getScanner(StorageManager.java:60)
      at org.apache.tajo.storage.AbstractStorageManager.getScanner(AbstractStorageManager.java:106)
      at org.apache.tajo.storage.AbstractStorageManager.getFileScanner(AbstractStorageManager.java:94)
      at org.apache.tajo.storage.AbstractStorageManager.isSplittable(AbstractStorageManager.java:399)
      at org.apache.tajo.storage.AbstractStorageManager.getSplits(AbstractStorageManager.java:554)
      at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.scheduleFragmentsForLeafQuery(SubQuery.java:872)
      at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.schedule(SubQuery.java:778)
      at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:629)
      at org.apache.tajo.master.querymaster.SubQuery$InitAndRequestContainer.transition(SubQuery.java:610)
      at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
      at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      at org.apache.tajo.master.querymaster.SubQuery.handle(SubQuery.java:587)
      at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:332)
      at org.apache.tajo.master.querymaster.Query$StartTransition.transition(Query.java:321)
      at org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
      at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
      at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
      at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
      at org.apache.tajo.master.querymaster.Query.handle(Query.java:646)
      at org.apache.tajo.master.querymaster.Query.handle(Query.java:58)
      at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137)
      at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      at org.apache.tajo.storage.AbstractStorageManager.newScannerInstance(AbstractStorageManager.java:659)
      ... 25 more
      Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
      at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
      at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
      at org.apache.tajo.storage.CSVFile$CSVScanner.<init>(CSVFile.java:245)
      ... 30 more
      Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1626)
      at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
      ... 32 more
      2014-03-05 10:48:18,237 INFO querymaster.Query (Query.java:handle(654)) - q_1393922377833_0005 Query Transitioned from QUERY_NEW to QUERY_RUNNING
      2014-03-05 10:48:18,238 INFO querymaster.Query (Query.java:handle(641)) - Processing q_1393922377833_0005 of type SUBQUERY_COMPLETED
      2014-03-05 10:48:18,238 INFO querymaster.Query (Query.java:handle(641)) - Processing q_1393922377833_0005 of type QUERY_COMPLETED
      2014-03-05 10:48:18,238 INFO querymaster.Query (Query.java:handle(654)) - q_1393922377833_0005 Query Transitioned from QUERY_RUNNING to QUERY_ERROR
      2014-03-05 10:48:18,238 INFO querymaster.QueryMasterTask (QueryMasterTask.java:handle(266)) - Query completion notified from q_1393922377833_0005
      2014-03-05 10:48:18,238 INFO querymaster.QueryMasterTask (QueryMasterTask.java:handle(277)) - Query final state: QUERY_ERROR
      2014-03-05 10:48:18,239 INFO querymaster.QueryMasterTask (QueryMasterTask.java:stop(172)) - Stopping QueryMasterTask:q_1393922377833_0005
      2014-03-05 10:48:18,246 WARN master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(115)) - Interrupted Exception while stopping
      2014-03-05 10:48:18,247 INFO master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:q_1393922377833_0005
      2014-03-05 10:48:18,248 INFO querymaster.QueryMasterTask (QueryMasterTask.java:stop(200)) - Stopped QueryMasterTask:q_1393922377833_0005
      2014-03-05 10:48:18,248 INFO querymaster.QueryMaster (QueryMaster.java:cleanup(167)) - cleanup query resources : q_1393922377833_0005
      2014-03-05 10:48:18,419 INFO worker.TajoWorkerClientService (TajoWorkerClientService.java:closeQuery(211)) - Stop Query:q_1393922377833_0005
      2014-03-05 10:48:18,844 INFO querymaster.QueryMasterManagerService (QueryMasterManagerService.java:executeQuery(224)) - Receive executeQuery request:q_1393922377833_0006
      2014-03-05 10:48:18,845 INFO querymaster.QueryMaster (QueryMaster.java:handle(374)) - Start QueryStartEventHandler:q_1393922377833_0006
      2014-03-05 10:48:18,853 INFO querymaster.QueryMasterTask (QueryMasterTask.java:initStagingDir(403)) - The staging dir 'null' is created.
      2014-03-05 10:48:18,943 INFO rewrite.BasicQueryRewriteEngine (BasicQueryRewriteEngine.java:rewrite(64)) - The rule "ProjectionPushDown " rewrites the query.
      2014-03-05 10:48:18,944 INFO global.GlobalPlanner (GlobalPlanner.java:build(103)) -
      -----------------------------
      Query Block Graph

      The QueryMaster log content:
      -----------------------------
      Query Block Graph
      -----------------------------

      -#ROOT
      -----------------------------
      Optimization Log:
      -----------------------------

      SCAN(0) on test
      => target list: test.id (INT4), test.interface (TEXT), test.start_time (TEXT), test.byte (TEXT)
      => out schema:

      {(4) test.id (INT4),test.interface (TEXT),test.start_time (TEXT),test.byte (TEXT)}
      => in schema: {(4) test.id (INT4),test.interface (TEXT),test.start_time (TEXT),test.byte (TEXT)}

      2014-03-05 10:48:11,760 INFO master.GlobalEngine (GlobalEngine.java:createLogicalPlan(252)) - =============================================
      2014-03-05 10:48:11,761 INFO master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:start(101)) - AsyncDispatcher started:QueryInProgress:q_1393922377833_0003
      2014-03-05 10:48:11,761 INFO querymaster.QueryInProgress (QueryInProgress.java:startQueryMaster(151)) - Initializing QueryInProgress for QueryID=q_1393922377833_0003
      2014-03-05 10:48:11,761 INFO rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:allocateQueryMaster(201)) - q_1393922377833_0003's QueryMaster is host:slave2, port=28093,28091,28092,53265, slots=m:1024,d:1.0,c:1, used=m:512,d:0.0,c:0
      2014-03-05 10:48:11,762 INFO querymaster.QueryInProgress (QueryInProgress.java:connectQueryMaster(194)) - Connect to QueryMaster:slave2/192.168.1.79:28093
      2014-03-05 10:48:11,767 INFO querymaster.QueryInProgress (QueryInProgress.java:submmitQueryToMaster(215)) - Call executeQuery to :slave2:28093,q_1393922377833_0003
      2014-03-05 10:48:12,013 INFO querymaster.QueryInProgress (QueryInProgress.java:heartbeat(246)) - Received QueryMaster heartbeat:q_1393922377833_0003state=QUERY_RUNNING,progress=NaN, queryMaster=host:slave2, port=28093,28091,28092,0, slots=m:0,d:0.0,c:0, used=m:0,d:0.0,c:0
      2014-03-05 10:48:12,015 INFO querymaster.QueryInProgress (QueryInProgress.java:heartbeat(246)) - Received QueryMaster heartbeat:q_1393922377833_0003state=QUERY_ERROR,progress=NaN, queryMaster=host:slave2, port=28093,28091,28092,0, slots=m:0,d:0.0,c:0, used=m:0,d:0.0,c:0
      2014-03-05 10:48:12,015 INFO querymaster.QueryJobManager (QueryJobManager.java:stopQuery(139)) - Stop QueryInProgress:q_1393922377833_0003
      2014-03-05 10:48:12,016 INFO querymaster.QueryInProgress (QueryInProgress.java:stop(104)) - =========================================================
      2014-03-05 10:48:12,016 INFO querymaster.QueryInProgress (QueryInProgress.java:stop(105)) - Stop query:q_1393922377833_0003
      2014-03-05 10:48:12,016 INFO rm.TajoWorkerResourceManager (TajoWorkerResourceManager.java:stopQueryMaster(634)) - release QueryMaster resource:q_1393922377833_0003,host:slave2, port=28093,28091,28092,53265, slots=m:1024,d:1.0,c:1, used=m:0,d:0.0,c:0
      2014-03-05 10:48:12,016 INFO querymaster.QueryInProgress (QueryInProgress.java:stop(113)) - q_1393922377833_0003 QueryMaster stopped
      2014-03-05 10:48:12,019 INFO master.TajoAsyncDispatcher (TajoAsyncDispatcher.java:stop(122)) - AsyncDispatcher stopped:QueryInProgress:q_1393922377833_0003
      2014-03-05 10:48:16,253 INFO master.GlobalEngine (GlobalEngine.java:executeQuery(107)) - SQL: select * from test
      2014-03-05 10:48:16,253 INFO master.GlobalEngine (GlobalEngine.java:executeQuery(126)) - hive.query.mode:false
      2014-03-05 10:48:16,342 INFO master.GlobalEngine (GlobalEngine.java:createLogicalPlan(246)) - =============================================
      2014-03-05 10:48:16,342 INFO master.GlobalEngine (GlobalEngine.java:createLogicalPlan(247)) - Non Optimized Query:

      -----------------------------
      Query Block Graph
      -----------------------------

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        Hi Guo,

        According to the error message, the worker cannot load LzoCodec library as follows:

        at org.apache.tajo.storage.AbstractStorageManager.newScannerInstance(AbstractStorageManager.java:659)
        ... 25 more
        Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
        at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
        at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
        at org.apache.tajo.storage.CSVFile$CSVScanner.<init>(CSVFile.java:245)
        

        Since Tajo uses the configs of an existing hadoop cluster, the tajo cluster will find LzoCodec if your hadoop has the configs for LzoCodec. Could you check the Lzo config in your hadoop cluster?

        Show
        hyunsik Hyunsik Choi added a comment - Hi Guo, According to the error message, the worker cannot load LzoCodec library as follows: at org.apache.tajo.storage.AbstractStorageManager.newScannerInstance(AbstractStorageManager.java:659) ... 25 more Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found. at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135) at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175) at org.apache.tajo.storage.CSVFile$CSVScanner.<init>(CSVFile.java:245) Since Tajo uses the configs of an existing hadoop cluster, the tajo cluster will find LzoCodec if your hadoop has the configs for LzoCodec. Could you check the Lzo config in your hadoop cluster?
        Hide
        guorui7@gmail.com Andrew Guo added a comment -

        Hi Hyunsik

        tajo-size.xml config content:
        <configuration>
        <property>
        <name>tajo.rootdir</name>
        <value>hdfs://master:9000/tajo</value>
        </property>

        <property>
        <name>tajo.worker.tmpdir.locations</name>
        <value>/home/hadoop/clouderaCDH/tajoTmp</value>
        </property>

        <property>
        <name>tajo.master.umbilical-rpc.address</name>
        <value>master:26001</value>
        </property>

        <property>
        <name>tajo.catalog.client-rpc.address</name>
        <value>master:26005</value>
        </property>

        <property>
        <name>tajo.master.client-rpc.address</name>
        <value>master:26002</value>
        </property>

        <property>
        <name>tajo.worker.parallel-execution.max-num</name>
        <value>12</value>
        </property>

        </configuration>

        "catalog-site.xml config content":
        <configuration>
        <property>
        <name>tajo.catalog.store.class</name>
        <value>org.apache.tajo.catalog.store.MySQLStore</value>
        </property>

        <property>
        <name>tajo.catalog.jdbc.connection.id</name>
        <value>esight</value>
        </property>

        <property>
        <name>tajo.catalog.jdbc.connection.password</name>
        <value>123456</value>
        </property>

        <property>
        <name>tajo.catalog.jdbc.uri</name>
        <value>jdbc:mysql://192.168.1.133:3306/tajo?createDatabaseIfNotExist=true</value>
        </property>
        </configuration>

        Show
        guorui7@gmail.com Andrew Guo added a comment - Hi Hyunsik tajo-size.xml config content: <configuration> <property> <name>tajo.rootdir</name> <value>hdfs://master:9000/tajo</value> </property> <property> <name>tajo.worker.tmpdir.locations</name> <value>/home/hadoop/clouderaCDH/tajoTmp</value> </property> <property> <name>tajo.master.umbilical-rpc.address</name> <value>master:26001</value> </property> <property> <name>tajo.catalog.client-rpc.address</name> <value>master:26005</value> </property> <property> <name>tajo.master.client-rpc.address</name> <value>master:26002</value> </property> <property> <name>tajo.worker.parallel-execution.max-num</name> <value>12</value> </property> </configuration> "catalog-site.xml config content": <configuration> <property> <name>tajo.catalog.store.class</name> <value>org.apache.tajo.catalog.store.MySQLStore</value> </property> <property> <name>tajo.catalog.jdbc.connection.id</name> <value>esight</value> </property> <property> <name>tajo.catalog.jdbc.connection.password</name> <value>123456</value> </property> <property> <name>tajo.catalog.jdbc.uri</name> <value>jdbc:mysql://192.168.1.133:3306/tajo?createDatabaseIfNotExist=true</value> </property> </configuration>
        Hide
        hyunsik Hyunsik Choi added a comment -

        In order to use lzo codec, you should firstly install lzo codec and add configs to your hadoop cluster. There is nothing todo in Tajo because hadoop's configs are shared with Tajo. You seem to use CDH. The following articles would be helpful for you.

        https://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_topic_4_4.html

        If you can find the keyword 'lzo' in the page, you would get useful information for you.

        Thanks,
        Hyunsik

        Show
        hyunsik Hyunsik Choi added a comment - In order to use lzo codec, you should firstly install lzo codec and add configs to your hadoop cluster. There is nothing todo in Tajo because hadoop's configs are shared with Tajo. You seem to use CDH. The following articles would be helpful for you. https://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH5/latest/CDH5-Installation-Guide/cdh5ig_topic_4_4.html If you can find the keyword 'lzo' in the page, you would get useful information for you. Thanks, Hyunsik
        Hide
        guorui7@gmail.com Andrew Guo added a comment -

        Hi Hyunsik

        I am not using CDH, I am using hadoop 2.2.0, is the core-site.xml configuration errors.
        Thank you for reminding me.

        Show
        guorui7@gmail.com Andrew Guo added a comment - Hi Hyunsik I am not using CDH, I am using hadoop 2.2.0, is the core-site.xml configuration errors. Thank you for reminding me.
        Hide
        hyunsik Hyunsik Choi added a comment -

        I'm sorry for misunderstanding. I guessed your hadoop distribution from the directory in your configs. But, it wasn't

        Hadoop wiki also provides LZO codec documentation, but it looks stale. I'm not sure that the documentation is still valid.
        http://wiki.apache.org/hadoop/UsingLzoCompression

        Actually, there are rarely documentations which address LZO codec in Hadoop due to its license problems.

        Show
        hyunsik Hyunsik Choi added a comment - I'm sorry for misunderstanding. I guessed your hadoop distribution from the directory in your configs. But, it wasn't Hadoop wiki also provides LZO codec documentation, but it looks stale. I'm not sure that the documentation is still valid. http://wiki.apache.org/hadoop/UsingLzoCompression Actually, there are rarely documentations which address LZO codec in Hadoop due to its license problems.

          People

          • Assignee:
            Unassigned
            Reporter:
            guorui7@gmail.com Andrew Guo
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development