Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6550

archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      The archive-logs tool added in MAPREDUCE-6415 leverages the Distributed Shell app. When using the DefaultContainerExecutor, this means that the job will actually run as the Yarn user, so the resulting har files are owned by the Yarn user instead of the original owner. The permissions are also now world-readable.

      In the below example, the archived logs are owned by 'yarn' instead of 'paul' and are now world-readable:

      [root@gs28-centos66-5 ~]# sudo -u hdfs hdfs dfs -ls -R /tmp/logs
      ...
      drwxrwx---   - paul  hadoop          0 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005
      drwxr-xr-x   - yarn  hadoop          0 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har
      -rw-r--r--   3 yarn  hadoop          0 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_SUCCESS
      -rw-r--r--   3 yarn  hadoop       1256 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_index
      -rw-r--r--   3 yarn  hadoop         24 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/_masterindex
      -rw-r--r--   3 yarn  hadoop    8451177 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0005/application_1443805425363_0005.har/part-0
      drwxrwx---   - paul  hadoop          0 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0006
      -rw-r-----   3 paul  hadoop       1155 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0006/gs-centos66-2.vpc.cloudera.com_8041
      -rw-r-----   3 paul  hadoop       4880 2015-10-02 13:24 /tmp/logs/paul/logs/application_1443805425363_0006/gs28-centos66-3.vpc.cloudera.com_8041
      ...
      

        Attachments

        1. MAPREDUCE-6550.001.patch
          10 kB
          Robert Kanter
        2. MAPREDUCE-6550.002.patch
          17 kB
          Robert Kanter
        3. MAPREDUCE-6550.003.patch
          18 kB
          Robert Kanter
        4. MAPREDUCE-6550.004.patch
          18 kB
          Robert Kanter
        5. MAPREDUCE-6550.005.patch
          18 kB
          Robert Kanter

        Issue Links

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

            • Assignee:
              rkanter Robert Kanter Assign to me
              Reporter:
              rkanter Robert Kanter

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment