Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-3300

Hadoop 3 Pig (with Tez) action shall not use given distributed cache

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Won't Fix
    • Affects Version/s: 4.3.1
    • Fix Version/s: None
    • Component/s: action
    • Labels:
    • Environment:

      Oozie-4.3.1
      Hadoop-3

      Description

      When I run my Pig action and set -x Tez to use Tez as the execution engine, I got the following error:

      Caused by: java.lang.RuntimeException: org.apache.hadoop.mapred.InvalidJobConfException: cache file (mapreduce.job.cache.files) scheme: "hdfs" host: "mycluster" port: -1 file: "/tmp/temp257132233/hive-exec-3.0.0.3.0.0.0-1469.jar" conflicts with cache file (mapreduce.job.cache.files)
       hdfs://mycluster/user/oozie/share/lib/lib_20180611041054/pig/hive-exec-3.0.0.3.0.0.0-1469.jar
      

      or

      Caused by: java.lang.RuntimeException: org.apache.hadoop.mapred.InvalidJobConfException: cache file (mapreduce.job.cache.files) scheme: "hdfs" host: "ns1" port: -1 file: "/tmp/temp36150863/joda-time-2.9.6.jar" conflicts with cache file (mapreduce.job.cache.files) hdfs://ns1/user/oozie/share/lib/lib_20180612003013/pig/joda-time-2.9.6.jar
      

      When I exclude the conflicting jar, then Pig cannot start due to class not found exception.

      I think a workaround or solution? could be if we exclude the mapreduce.job.cache.files and mapreduce.job.cache.archives from pig.properties file.

       

      It is a bit scary change because I am not sure if the user's job shall use specific driver or implementation which needs to be on distributed cache.

        Attachments

          Activity

            People

            • Assignee:
              dionusos Denes Bodo
              Reporter:
              dionusos Denes Bodo
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: