Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4246

Failure in deleting user directories in Secure hadoop

    Details

    • Type: Task Task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.1
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      This happens when security is enabled on 22

      When TaskTracker starts p, it invokes MRAsyncDiskService moves the contents of scratch/tasktracker to toBeDeleted under a directory created with the current timestamp.
      The owner of this directory is hadoop (owner of tasktracker)

      The contents of this directory are various usernames and they are owned by individual users. But MRAsyncDiskService tries to delete the newly created directory as hadoop and it fails.

      2012-05-01 16:04:58,805 DEBUG org.apache.hadoop.mapred.LinuxTaskController: deleteAsUser: [/apache/hadoop-assemble-argon-0.228/hadoop-0.22-argon-0.105/bin/../bin/task-controller, hadoop, 3, /hadoop00/scratch/toBeDeleted/2012-04-17_16-22-03.965_0]
      2012-05-01 16:04:58,809 WARN org.apache.hadoop.mapreduce.util.MRAsyncDiskService: Failure in deletion of toBeDeleted/2012-04-17_16-22-03.965_0 on /hadoop00/scratch with original name /hadoop00/scratch/toBeDeleted/2012-04-17_16-22-03.965_0 with exception org.apache.hadoop.util.Shell$ExitCodeException:
      at org.apache.hadoop.util.Shell.runCommand(Shell.java:256)
      at org.apache.hadoop.util.Shell.run(Shell.java:183)
      at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:376)
      at org.apache.hadoop.mapred.LinuxTaskController.deleteAsUser(LinuxTaskController.java:273)
      at org.apache.hadoop.mapreduce.util.MRAsyncDiskService$DeleteTask.run(MRAsyncDiskService.java:237)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

        Issue Links

          Activity

          Benoy Antony created issue -
          Benoy Antony made changes -
          Field Original Value New Value
          Attachment cleanup-user-dir.patch [ 12526275 ]
          Benoy Antony made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Benoy Antony made changes -
          Parent HADOOP-8357 [ 12553870 ]
          Issue Type Sub-task [ 7 ] Task [ 3 ]
          Benoy Antony made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-8382 MAPREDUCE-4246
          Affects Version/s 0.22.0 [ 12314184 ]
          Affects Version/s 0.22.0 [ 12314296 ]
          Fix Version/s 0.22.1 [ 12319242 ]
          Fix Version/s 0.22.1 [ 12319240 ]
          Component/s security [ 12313041 ]
          Component/s security [ 12312526 ]
          Benoy Antony made changes -
          Link This issue is depended upon by HADOOP-8357 [ HADOOP-8357 ]
          Hide
          Konstantin Shvachko added a comment -

          +1
          I just committed this to branch 0.22.1. Thank you Benoy.

          Show
          Konstantin Shvachko added a comment - +1 I just committed this to branch 0.22.1. Thank you Benoy.
          Konstantin Shvachko made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 0.22.1 [ 12319242 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/)
          MAPREDUCE-4246. Failure in deleting user directories in secure Hadoop. Contributed by Benoy Antony. (Revision 1346270)

          Result = SUCCESS
          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346270
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/CleanupQueue.java
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/LinuxTaskController.java
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/TaskController.java
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java
          • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapreduce/util/MRAsyncDiskService.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/ ) MAPREDUCE-4246 . Failure in deleting user directories in secure Hadoop. Contributed by Benoy Antony. (Revision 1346270) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346270 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/CleanupQueue.java /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/LinuxTaskController.java /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/TaskController.java /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/TaskTracker.java /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapreduce/util/MRAsyncDiskService.java
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          5h 35m 1 Benoy Antony 10/May/12 08:17
          In Progress In Progress Resolved Resolved
          26d 23m 1 Konstantin Shvachko 05/Jun/12 08:41

            People

            • Assignee:
              Benoy Antony
              Reporter:
              Benoy Antony
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development