Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4396

Make LocalJobRunner work with private distributed cache

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.3
    • Fix Version/s: 1.1.2
    • Component/s: client
    • Labels:
      None

      Description

      Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive.

      1. HADOOP-8734-LocalJobRunner.patch
        2 kB
        Matt Foley
      2. mapreduce-4396-branch-1.patch
        0.7 kB
        Yu Gao
      3. test-afterpatch.result
        54 kB
        Yu Gao
      4. test-beforepatch.result
        45 kB
        Yu Gao
      5. test-patch.result
        491 kB
        Yu Gao

        Activity

        Hide
        Yu Gao added a comment -

        Attached ant test and test-patch result.

        This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700), so no new test is added.

        Before this patch, TestMRWithDistributedCache#testLocalJobRunner could fail when the test directory - test.build.data - is in a path where any ancestor doesn't have +x permissions for others, with following exception:
        java.io.FileNotFoundException: File build/test/mapred/local/archive/5659425182264546915_1878220893_1556512037/file/home/biadmin/Hadoop_upstream/branch-1/build/test/data/distributed.first does not exist.
        at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402)
        at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
        at org.apache.hadoop.filecache.TestMRWithDistributedCache$DistributedCacheChecker.setup(TestMRWithDistributedCache.java:93)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)

        This results from private distributed cache not localized by LocalJobRunner. The patch adds the localization for private distributed cache, so that no matter whether the test directory permission is too restrictive or not, distributed cache (public/private) will be localized.

        Show
        Yu Gao added a comment - Attached ant test and test-patch result. This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700), so no new test is added. Before this patch, TestMRWithDistributedCache#testLocalJobRunner could fail when the test directory - test.build.data - is in a path where any ancestor doesn't have +x permissions for others, with following exception: java.io.FileNotFoundException: File build/test/mapred/local/archive/5659425182264546915_1878220893_1556512037/file/home/biadmin/Hadoop_upstream/branch-1/build/test/data/distributed.first does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.filecache.TestMRWithDistributedCache$DistributedCacheChecker.setup(TestMRWithDistributedCache.java:93) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) This results from private distributed cache not localized by LocalJobRunner. The patch adds the localization for private distributed cache, so that no matter whether the test directory permission is too restrictive or not, distributed cache (public/private) will be localized.
        Hide
        Eric Yang added a comment -

        Does trunk need a patch?

        Show
        Eric Yang added a comment - Does trunk need a patch?
        Hide
        Ivan Mitic added a comment -

        This was fixed with HADOOP-8734 in branch-1-win. Maybe just integrate the same patch to branch-1?

        Show
        Ivan Mitic added a comment - This was fixed with HADOOP-8734 in branch-1-win. Maybe just integrate the same patch to branch-1?
        Hide
        Yu Gao added a comment -

        @Eric Yang Trunk does not have this issue.

        Show
        Yu Gao added a comment - @ Eric Yang Trunk does not have this issue.
        Hide
        Yu Gao added a comment -

        Hmm, the patch here is the same as that in HADOOP-8734 for LocalJobRunner, so either way is ok.

        Show
        Yu Gao added a comment - Hmm, the patch here is the same as that in HADOOP-8734 for LocalJobRunner, so either way is ok.
        Hide
        Eric Yang added a comment -

        +1 looks good.

        Show
        Eric Yang added a comment - +1 looks good.
        Hide
        Eric Yang added a comment -

        I just committed this. Thanks Ivan and Yu.

        Show
        Eric Yang added a comment - I just committed this. Thanks Ivan and Yu.
        Hide
        Matt Foley added a comment -

        The patch for HADOOP-8734 is the same in LocalJobRunner.java. However, that patch includes a proper unit test change in TestMRWithDistributedCache.java. @Yu, your comment above that "This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700)" is correct, but the conclusion that therefore no new unit test is necessary is incorrect. What's necessary is a unit test that does precisely that. The patch from HADOOP-8734 provides such a patch.

        I've merged the unit test patch to hadoop-1 and hadoop-1.1.

        Show
        Matt Foley added a comment - The patch for HADOOP-8734 is the same in LocalJobRunner.java. However, that patch includes a proper unit test change in TestMRWithDistributedCache.java. @Yu, your comment above that "This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700)" is correct, but the conclusion that therefore no new unit test is necessary is incorrect. What's necessary is a unit test that does precisely that. The patch from HADOOP-8734 provides such a patch. I've merged the unit test patch to hadoop-1 and hadoop-1.1.
        Hide
        Matt Foley added a comment -

        Closed upon successful release of 1.1.2.

        Show
        Matt Foley added a comment - Closed upon successful release of 1.1.2.

          People

          • Assignee:
            Yu Gao
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development