Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5782

Sort Use TableOrderPartitioner cannot find the path of _partition.lst int the container cwd.

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.23.10, 2.4.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Release Note:
      Fix the sort example when TotalOrderPartitioner is specified.

      Description

      hadoop jar hadoop-mapreduce-examples.jar sort sort \
      -inFormat org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat \
      -outFormat org.apache.hadoop.mapreduce.lib.output.TextOutputFormat \
      -outKey org.apache.hadoop.io.Text \
      -outValue org.apache.hadoop.io.Text \
      -totalOrder 0.1 10000 10 \
      input output
      It prints cannot find the file _partition.lst in the container's cwd.
      Error: java.lang.IllegalArgumentException: Can't read partitions file
      at
      org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
      at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
      at
      org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
      at
      org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:642)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:710)
      at
      org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)

        Activity

        Hide
        Chao Zhang added a comment -

        Can't reproduce that again.

        Show
        Chao Zhang added a comment - Can't reproduce that again.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12706074/MAPREDUCE-5782.001.patch
        against trunk revision e1feb4e.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-examples.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5322//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5322//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12706074/MAPREDUCE-5782.001.patch against trunk revision e1feb4e. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-examples. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5322//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5322//console This message is automatically generated.
        Hide
        ComboZhc added a comment -
        Show
        ComboZhc added a comment - http://stackoverflow.com/questions/25051671/totalorderpartitioner-ignores-partition-file-location This is a working suggestion.
        Hide
        ComboZhc added a comment -
        Show
        ComboZhc added a comment - http://stackoverflow.com/questions/25051671/totalorderpartitioner-ignores-partition-file-location This is a working suggestion.
        Hide
        Chao Zhang added a comment -

        The same thing here:
        hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0-amzn-3.jar sort -r 27 -outKey org.apache.hadoop.io.IntWritable -outValue org.apache.hadoop.io.IntWritable -totalOrder 0.1 540 27 /integer/12g /user/hadoop/mapred/12g

        Exceptions:
        Error: java.lang.IllegalArgumentException: Can't read partitions file
        at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:687)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170)
        Caused by: java.io.FileNotFoundException: File file:/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1426795637584_0013/container_1426795637584_0013_01_000018/_partition.lst does not exist

        Show
        Chao Zhang added a comment - The same thing here: hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0-amzn-3.jar sort -r 27 -outKey org.apache.hadoop.io.IntWritable -outValue org.apache.hadoop.io.IntWritable -totalOrder 0.1 540 27 /integer/12g /user/hadoop/mapred/12g Exceptions: Error: java.lang.IllegalArgumentException: Can't read partitions file at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:687) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:175) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:170) Caused by: java.io.FileNotFoundException: File file:/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/appcache/application_1426795637584_0013/container_1426795637584_0013_01_000018/_partition.lst does not exist
        Hide
        Bing Jiang added a comment -

        Checking the code,

        ========org.apache.hadoop.examples.Sort========
        URI partitionUri = new URI(partitionFile.toString() +
        "#" + "_sortPartitioning");
        DistributedCache.addCacheFile(partitionUri, conf);

        which rename default name '_partition.lst' into '_sortPartitioning' when container finishes localization.
        So the problem can be tackled by add -Dmapreduce.totalorderpartitioner.path=_sortPartitioning
        or remove the setting of symlink here.

        Show
        Bing Jiang added a comment - Checking the code, ========org.apache.hadoop.examples.Sort======== URI partitionUri = new URI(partitionFile.toString() + "#" + "_sortPartitioning"); DistributedCache.addCacheFile(partitionUri, conf); which rename default name '_partition.lst' into '_sortPartitioning' when container finishes localization. So the problem can be tackled by add -Dmapreduce.totalorderpartitioner.path=_sortPartitioning or remove the setting of symlink here.

          People

          • Assignee:
            Unassigned
            Reporter:
            Bing Jiang
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development