Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2032

Tez install does not set up tez jars on hdfs causing Pig to fail

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1, 1.1.0
    • Component/s: general
    • Labels:
      None

      Description

      Instructions on how to configure Tez for Pig http://pig.apache.org/docs/r0.14.0/perf.html

      Stack trace does not seem to be obivious. But the same Pig operations work (load and then dump) then Tez is not installed.

      2015-09-08 15:31:51,571 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!

      2015-09-08 15:31:51,589 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias a. Backend error : java.io.IOException: org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException: Application with id 'application_1441732819845_0003' doesn't exist in RM.

      at org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplicationReport(ClientRMService.java:324)

      at org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplicationReport(ApplicationClientProtocolPBServiceImpl.java:170)

      at org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:401)

      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)

      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:415)

      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)

      Details at logfile: /root/pig_1441740673067.log

      grunt>

        Activity

        Hide
        oflebbe Olaf Flebbe added a comment -

        Installation of a package will never install anything in HDFS.

        The deployment scripts handle this

        Show
        oflebbe Olaf Flebbe added a comment - Installation of a package will never install anything in HDFS. The deployment scripts handle this
        Hide
        tomzeng Tom Zeng added a comment - - edited

        Look like some sample deployment templates would be helpful, otherwise Pig will be broken when Tez is installed. Same applies to others app. Some sensible defaults in deployment scripts to have all the apps working would be nice.

        Show
        tomzeng Tom Zeng added a comment - - edited Look like some sample deployment templates would be helpful, otherwise Pig will be broken when Tez is installed. Same applies to others app. Some sensible defaults in deployment scripts to have all the apps working would be nice.
        Hide
        oflebbe Olaf Flebbe added a comment - - edited

        tez will be deployed by the bigtop-puppet/puppet/ framework, if configured.i.e. it is already there

        Show
        oflebbe Olaf Flebbe added a comment - - edited tez will be deployed by the bigtop-puppet/puppet/ framework, if configured.i.e. it is already there
        Hide
        jonathak Jonathan Kelly added a comment -

        Tom Zeng, this is probably because we have temporarily disabled the copying of jars for Oozie in the init-hcfs.groovy script in our EMR fork of Bigtop. I think those jars would be there if you were running vanilla Bigtop. See https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/init-hcfs.groovy#L329

        Show
        jonathak Jonathan Kelly added a comment - Tom Zeng , this is probably because we have temporarily disabled the copying of jars for Oozie in the init-hcfs.groovy script in our EMR fork of Bigtop. I think those jars would be there if you were running vanilla Bigtop. See https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/init-hcfs.groovy#L329
        Hide
        evans_ye Evans Ye added a comment - - edited

        I agree we should have deployment samples. I'm thinking maybe we can give users more specific settings based on what exactly the need is:

        • Hadoop Big Data Stack (hadoop, pig, hive, tez, oozie, etc)
        • Spark Big Data Stack (hadoop, spark, tachyon, kafka, etc)
        • In-memory Big Data Stack (hadoop, ignite, pig, hive, etc)

        Another set of template can be driven by functionality:

        • Kerberos enabled
        • Namenode/RM HA enabled

        I'll do some improvements and refractory works to our provisioners and I'm looking for these being available in near future.

        Show
        evans_ye Evans Ye added a comment - - edited I agree we should have deployment samples. I'm thinking maybe we can give users more specific settings based on what exactly the need is: Hadoop Big Data Stack (hadoop, pig, hive, tez, oozie, etc) Spark Big Data Stack (hadoop, spark, tachyon, kafka, etc) In-memory Big Data Stack (hadoop, ignite, pig, hive, etc) Another set of template can be driven by functionality: Kerberos enabled Namenode/RM HA enabled I'll do some improvements and refractory works to our provisioners and I'm looking for these being available in near future.
        Hide
        apurtell Andrew Purtell added a comment -

        If I read this correctly this should be resolved as Not a problem? We can follow up with issues for the deployment examples.

        Show
        apurtell Andrew Purtell added a comment - If I read this correctly this should be resolved as Not a problem? We can follow up with issues for the deployment examples.
        Hide
        oflebbe Olaf Flebbe added a comment -

        The original topic of this JIRA isn't a problem AFAIK. The other deployment issues are valid but should be handled constructivley in a different JIRA.

        Show
        oflebbe Olaf Flebbe added a comment - The original topic of this JIRA isn't a problem AFAIK. The other deployment issues are valid but should be handled constructivley in a different JIRA.
        Hide
        tomzeng Tom Zeng added a comment -

        sounds good, let's follow up with deployments examples

        Show
        tomzeng Tom Zeng added a comment - sounds good, let's follow up with deployments examples

          People

          • Assignee:
            Unassigned
            Reporter:
            tomzeng Tom Zeng
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development