Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-420

FileNotFoundException when finishing a profiled task that doesn't generate an output file

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Trivial Trivial
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When running a hadoop job with mapred.task.profile=true, hadoop assumes that mapred.task.profile.params contains a "%s" that can be changed for a file name that will be created by the profiler containing its text output. If the profiler doesn't generate such an output file, profiled tasks will raise a FileNotFound exception at the end of their execution:

      09/02/05 15:39:10 INFO mapred.JobClient: map 100% reduce 0%
      09/02/05 15:39:10 INFO mapred.JobClient: Communication problem with server: java.io.FileNotFoundException: http://machine1:50060/tasklog?plaintext=true&taskid=attempt_200901221838_0043_m_000000_0&filter=profile
      at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1239)
      at org.apache.hadoop.mapred.JobClient.downloadProfile(JobClient.java:1089)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1166)
      at JavaCat.main(JavaCat.java:46)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:165)
      at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

      This happens because it's assumed users will use the default hprof profiler, which actually allows you to set the output file name. Not all profilers are like that and some don't have any option to create a text output log at all or don't allow you to change the name of the output file. In my case, I'm a profiler that doesn't generate text output and only allows you to set the output directory for the snapshot files it generates.

        Issue Links

          Activity

          Aaron Kimball made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-5180 MAPREDUCE-420
          Component/s mapred [ 12310690 ]
          Rodrigo Schmidt made changes -
          Link This issue is duplicated by HADOOP-3551 [ HADOOP-3551 ]
          Rodrigo Schmidt made changes -
          Link This issue is related to HADOOP-2367 [ HADOOP-2367 ]
          Rodrigo Schmidt made changes -
          Field Original Value New Value
          Assignee Rodrigo Schmidt [ rschmidt ]
          Rodrigo Schmidt created issue -

            People

            • Assignee:
              Rodrigo Schmidt
              Reporter:
              Rodrigo Schmidt
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development