Hadoop Common
  1. Hadoop Common
  2. HADOOP-3162

Map/reduce stops working with comma separated input paths

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16.
      The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat :
      public static void setInputPaths(JobConf job, Path... paths);
      public static void setInputPaths(JobConf job, String commaSeparatedPaths);
      public static void addInputPath(JobConf job, Path path);
      public static void addInputPaths(JobConf job, String commaSeparatedPaths);
      Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively
      Show
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16. The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat : public static void setInputPaths(JobConf job, Path... paths); public static void setInputPaths(JobConf job, String commaSeparatedPaths); public static void addInputPath(JobConf job, Path path); public static void addInputPaths(JobConf job, String commaSeparatedPaths); Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively

      Description

      When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

      org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
      namenode:port/gridmix/data/MonsterQueryBlockCompressed/part-
      00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
      QueryBlockCompressed/part-00002
      at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
      t.java:213)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
      at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
      rator.java:189)

      1. patch-3162.txt
        64 kB
        Amareshwari Sriramadasu
      2. patch-3162.txt
        71 kB
        Amareshwari Sriramadasu
      3. patch-3162.txt
        71 kB
        Amareshwari Sriramadasu
      4. patch-3162.txt
        72 kB
        Amareshwari Sriramadasu
      5. patch-3162.txt
        72 kB
        Amareshwari Sriramadasu
      6. patch-3162.txt
        73 kB
        Amareshwari Sriramadasu

        Activity

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Runping Qi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development