Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: fs
    • Labels:
      None

      Description

      -bash-3.1$ hadoop jar examples.jar wordcount -Dmapred.job.queue.name=${JOBQ}  har:///user/tsz/t10_4.har/t10_4 t10_4_har_wc
      java.io.IOException: No FileSystem for scheme: namenode_hostname
              at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1375)
              at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
              at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
              at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
              at org.apache.hadoop.fs.HarFileSystem.initialize(HarFileSystem.java:104)
              at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
              at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:193)
              at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
              at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:203)
              ...
      

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          Moved this from MapReduce to Common since this is a fs issue.

          Show
          Tsz Wo Nicholas Sze added a comment - Moved this from MapReduce to Common since this is a fs issue.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The problem is that org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job, Path path) uses the default FileSystem, instead of the FileSystem specified in the path.

          //org.apache.hadoop.mapreduce.lib.input.FileInputFormat
            public static void addInputPath(Job job, 
                                            Path path) throws IOException {
              Configuration conf = job.getConfiguration();
              FileSystem fs = FileSystem.get(conf);
              path = path.makeQualified(fs); // In our case, path scheme is har:
                                             // but fs scheme is hdfs://
              ...
            }
          
          Show
          Tsz Wo Nicholas Sze added a comment - The problem is that org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(Job job, Path path) uses the default FileSystem, instead of the FileSystem specified in the path. //org.apache.hadoop.mapreduce.lib.input.FileInputFormat public static void addInputPath(Job job, Path path) throws IOException { Configuration conf = job.getConfiguration(); FileSystem fs = FileSystem.get(conf); path = path.makeQualified(fs); // In our case , path scheme is har: // but fs scheme is hdfs:// ... }
          Hide
          Tsz Wo Nicholas Sze added a comment -

          c6564_20100222tmp_fix.patch: a tmp fix which slightly changed FileInputFormat and CompressionCodecFactory. Then, WordCound works with hdfs://.

          Show
          Tsz Wo Nicholas Sze added a comment - c6564_20100222tmp_fix.patch: a tmp fix which slightly changed FileInputFormat and CompressionCodecFactory. Then, WordCound works with hdfs://.
          Hide
          Mahadev konar added a comment -

          nicholas, does this jira belong to mapreduce then?

          Show
          Mahadev konar added a comment - nicholas, does this jira belong to mapreduce then?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > nicholas, does this jira belong to mapreduce then?
          Probably yes. Let me do some more tests first.

          I posted the changes on CompressionCodecFactory in HADOOP-6588.

          Show
          Tsz Wo Nicholas Sze added a comment - > nicholas, does this jira belong to mapreduce then? Probably yes. Let me do some more tests first. I posted the changes on CompressionCodecFactory in HADOOP-6588 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Tested har:// with ls, cp and distcp. All worked fine. I am closing this as invalid.

          Will file a jira for FileInputFormat.

          Show
          Tsz Wo Nicholas Sze added a comment - Tested har:// with ls, cp and distcp. All worked fine. I am closing this as invalid. Will file a jira for FileInputFormat.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Filed MAPREDUCE-1522 for the FileInputFormat problem.

          Show
          Tsz Wo Nicholas Sze added a comment - Filed MAPREDUCE-1522 for the FileInputFormat problem.

            People

            • Assignee:
              Mahadev konar
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development