Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-5042

Pig with Tez requires a staging directory with submitter ownership

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.16.0
    • Fix Version/s: None
    • Component/s: tez
    • Labels:
      None
    • Environment:

      Hadoop 2.7.2, Pig 0.16.0, Tez 0.8.4

      Description

      With Pig 0.15.0, I have been able to work using the Tez execution engine without issues. After upgrading Pig to 0.16.0, I haven't been able to submit my jobs.

      Since the beginning, the tez-site.xml is configured as below:

      <property>
        <description>String value to a file path. The location of the Tez libraries which will be localized for DAGs. This follows the following semantics</description>
        <name>tez.lib.uris</name>
        <value>hdfs:///Products/TEZ/tez-0.8.4.tar.gz</value>
      </property>
      <property>
        <description>String value. Specifies a directory where Tez can create temporary job artifacts.</description>
        <name>tez.staging-dir</name>
        <value>/Products/TEZ/staging/</value>
      </property>
      

      The staging directory was set up with 777 rights, including a sticky bit:

      [root@spark01 ~]# hadoop fs -ls /Products/TEZ/
      Found 3 items
      drwxrwxrwt+  - hdfs hadoop          0 2014-11-27 10:31 /Products/TEZ/staging
      -rw-r--r--   3 hdfs hadoop   46518370 2016-09-02 11:06 /Products/TEZ/tez-0.8.4.tar.gz
      

      Pig 0.16.0 seems to need a staging directory with submitter directory, even if the staging directory has full access to "other" like my previous configuration.
      This issue looks like a regression to me, compared to Pig 0.15.0. Can you check that point?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              BigDataOrange Alexandre Linte
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: