Hadoop Common
  1. Hadoop Common
  2. HADOOP-3162

Map/reduce stops working with comma separated input paths

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.17.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Hide
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16.
      The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat :
      public static void setInputPaths(JobConf job, Path... paths);
      public static void setInputPaths(JobConf job, String commaSeparatedPaths);
      public static void addInputPath(JobConf job, Path path);
      public static void addInputPaths(JobConf job, String commaSeparatedPaths);
      Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively
      Show
      The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16. The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat : public static void setInputPaths(JobConf job, Path... paths); public static void setInputPaths(JobConf job, String commaSeparatedPaths); public static void addInputPath(JobConf job, Path path); public static void addInputPaths(JobConf job, String commaSeparatedPaths); Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively

      Description

      When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

      org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
      namenode:port/gridmix/data/MonsterQueryBlockCompressed/part-
      00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
      QueryBlockCompressed/part-00002
      at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
      t.java:213)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
      at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
      rator.java:189)

      1. patch-3162.txt
        73 kB
        Amareshwari Sriramadasu
      2. patch-3162.txt
        72 kB
        Amareshwari Sriramadasu
      3. patch-3162.txt
        72 kB
        Amareshwari Sriramadasu
      4. patch-3162.txt
        71 kB
        Amareshwari Sriramadasu
      5. patch-3162.txt
        71 kB
        Amareshwari Sriramadasu
      6. patch-3162.txt
        64 kB
        Amareshwari Sriramadasu

        Activity

        Runping Qi created issue -
        Runping Qi made changes -
        Field Original Value New Value
        Component/s mapred [ 12310690 ]
        Description
        When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

        org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
        /gs205234.inktomisearch.com:55638/gridmix/data/MonsterQueryBlockCompressed/part-
        00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
        QueryBlockCompressed/part-00002
                at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
        t.java:213)
                at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
                at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
                at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
        rator.java:189)
        When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

        org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
        /gs205234.inktomisearch.com:55638/gridmix/data/MonsterQueryBlockCompressed/part-
        00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
        QueryBlockCompressed/part-00002
                at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
        t.java:213)
                at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
                at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
                at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
        rator.java:189)
        Nigel Daley made changes -
        Affects Version/s 0.17.0 [ 12312913 ]
        Sameer Paranjpye made changes -
        Fix Version/s 0.17.0 [ 12312913 ]
        Assignee Amar Kamat [ amar_kamat ]
        Runping Qi made changes -
        Description When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

        org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
        /gs205234.inktomisearch.com:55638/gridmix/data/MonsterQueryBlockCompressed/part-
        00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
        QueryBlockCompressed/part-00002
                at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
        t.java:213)
                at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
                at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
                at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
        rator.java:189)
        When a job is given a comma separated input file list, FileInputFormat class throws an exception, complaining the input is invalid:

        org.apache.hadoop.mapred.InvalidInputException: Input path doesnt exist : hdfs:/
        namenode:port/gridmix/data/MonsterQueryBlockCompressed/part-
        00000,/gridmix/data/MonsterQueryBlockCompressed/part-00001,/gridmix/data/Monster
        QueryBlockCompressed/part-00002
                at org.apache.hadoop.mapred.FileInputFormat.validateInput(FileInputForma
        t.java:213)
                at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:705)
                at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
                at org.apache.hadoop.mapred.GenericMRLoadGenerator.run(GenericMRLoadGene
        rator.java:189)
        Amareshwari Sriramadasu made changes -
        Assignee Amar Kamat [ amar_kamat ] Amareshwari Sriramadasu [ amareshwari ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379388 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379389 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379388 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Assignee Amareshwari Sriramadasu [ amareshwari ] Cameron Pope [ camerooni ]
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379539 ]
        Amareshwari Sriramadasu made changes -
        Assignee Cameron Pope [ camerooni ] Amareshwari Sriramadasu [ amareshwari ]
        Amareshwari Sriramadasu made changes -
        Release Note 1 Adds the following APIs in FileInputFormat
        public static void setInputPaths(JobConf job, Path... paths);
        public static void setInputPaths(JobConf job, String commaSepatedPaths);
        public static void addInputPath(JobConf job, Path path);
        public static void addInputPaths(JobConf job, String commaSepatedPaths);
        2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path)
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Release Note 1 Adds the following APIs in FileInputFormat
        public static void setInputPaths(JobConf job, Path... paths);
        public static void setInputPaths(JobConf job, String commaSepatedPaths);
        public static void addInputPath(JobConf job, Path path);
        public static void addInputPaths(JobConf job, String commaSepatedPaths);
        2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path)
        1 Adds the following APIs in FileInputFormat
        public static void setInputPaths(JobConf job, Path... paths);
        public static void setInputPaths(JobConf job, String commaSeparatedPaths);
        public static void addInputPath(JobConf job, Path path);
        public static void addInputPaths(JobConf job, String commaSeparatedPaths);
        2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path)
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379638 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379833 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12379981 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Amareshwari Sriramadasu made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-3162.txt [ 12380174 ]
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Arun C Murthy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Amareshwari Sriramadasu made changes -
        Release Note 1 Adds the following APIs in FileInputFormat
        public static void setInputPaths(JobConf job, Path... paths);
        public static void setInputPaths(JobConf job, String commaSeparatedPaths);
        public static void addInputPath(JobConf job, Path path);
        public static void addInputPaths(JobConf job, String commaSeparatedPaths);
        2. Deprecates JobConf.setInputPath(Path) and JobConf.addInputPath(Path)
        The public methods org.apache.hadoop.mapred.JobConf.setInputPath(Path) and org.apache.hadoop.mapred.JobConf.addInputPath(Path) are deprecated. And the methods have the semantics of branch 0.16.
        The following public APIs are added in org.apache.hadoop.mapred.FileInputFormat :
        public static void setInputPaths(JobConf job, Path... paths);
        public static void setInputPaths(JobConf job, String commaSeparatedPaths);
        public static void addInputPath(JobConf job, Path path);
        public static void addInputPaths(JobConf job, String commaSeparatedPaths);
        Earlier code calling JobConf.setInputPath(Path), JobConf.addInputPath(Path) should now call FileInputFormat.setInputPaths(JobConf, Path...) and FileInputFormat.addInputPath(Path) respectively
        Nigel Daley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s mapred [ 12310690 ]

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Runping Qi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development