Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3162

Map/reduce stops working with comma separated input paths

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16.
      The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat :
      public static void setInputPaths(JobConf job, Path... paths);
      public static void setInputPaths(JobConf job, String commaSeparatedPaths);
      public static void addInputPath(JobConf job, Path path);
      public static void addInputPaths(JobConf job, String commaSeparatedPaths);
      Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively
      Show
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16. The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat : public static void setInputPaths(JobConf job, Path... paths); public static void setInputPaths(JobConf job, String commaSeparatedPaths); public static void addInputPath(JobConf job, Path path); public static void addInputPaths(JobConf job, String commaSeparatedPaths); Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively

      Description

      When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

      org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
      namenode:port/gridmix/data/MonsterQueryBlockCompressed/part-
      00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
      QueryBlockCompressed/part-00002
      at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
      t.java:213)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
      at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
      rator.java:189)

        Attachments

        1. patch-3162.txt
          64 kB
          Amareshwari Sriramadasu
        2. patch-3162.txt
          71 kB
          Amareshwari Sriramadasu
        3. patch-3162.txt
          71 kB
          Amareshwari Sriramadasu
        4. patch-3162.txt
          72 kB
          Amareshwari Sriramadasu
        5. patch-3162.txt
          72 kB
          Amareshwari Sriramadasu
        6. patch-3162.txt
          73 kB
          Amareshwari Sriramadasu

          Activity

            People

            • Assignee:
              amareshwari Amareshwari Sriramadasu
              Reporter:
              runping Runping Qi
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: