Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2763

IllegalArgumentException while using the dist cache

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: mrv2
    • Labels:
      None

      Description

      IllegalArgumentException is seen while using distributed cache to cache some files and custom jars in classpath.

      A simple way to reproduce this error is by using a streaming job:
      hadoop jar hadoop-streaming.jar -libjars file://<path to custom jar> -input <path to input file> -output out -mapper "cat" -reducer NONE -cacheFile hdfs://<path to some file>#linkname

      This is a regression introduced and the same command works fine on 0.20.x

        Activity

        Hide
        Ramya Sunil added a comment -

        Below is the complete stacktrace:

        Exception in thread "main" java.lang.IllegalArgumentException: Invalid specification for distributed-cache artifacts of type FILE : #uris=1 #timestamps=2 #visibilities=2
                at org.apache.hadoop.mapred.YARNRunner.parseDistributedCacheArtifacts(YARNRunner.java:411)
                at org.apache.hadoop.mapred.YARNRunner.setupDistributedCache(YARNRunner.java:392)
                at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:234)
                at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:388)
                at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1064)
                at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1061)
                at java.security.AccessController.doPrivileged(Native Method)
                at javax.security.auth.Subject.doAs(Subject.java:396)
                at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094)
                at org.apache.hadoop.mapreduce.Job.submit(Job.java:1061)
                at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534)
                at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1010)
                at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:133)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
                at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
                at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
                at java.lang.reflect.Method.invoke(Method.java:597)
                at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
        
        Show
        Ramya Sunil added a comment - Below is the complete stacktrace: Exception in thread "main" java.lang.IllegalArgumentException: Invalid specification for distributed-cache artifacts of type FILE : #uris=1 #timestamps=2 #visibilities=2 at org.apache.hadoop.mapred.YARNRunner.parseDistributedCacheArtifacts(YARNRunner.java:411) at org.apache.hadoop.mapred.YARNRunner.setupDistributedCache(YARNRunner.java:392) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:234) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:388) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1064) at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1061) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1094) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1061) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:534) at org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1010) at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:133) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
        Hide
        Ramya Sunil added a comment -

        I see this issue being fixed in the latest code base. Hence resolving this.

        Show
        Ramya Sunil added a comment - I see this issue being fixed in the latest code base. Hence resolving this.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ramya Sunil
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development