Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2651

Race condition in Linux Task Controller for job log directory creation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0
    • Component/s: task-controller
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      task controller,

      Description

      There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time.

      1. MAPREDUCE-2651-1.patch
        2 kB
        Bharath Mundlapudi
      2. mr-2651-0.22.patch
        2 kB
        Benoy Antony

        Issue Links

          Activity

          Hide
          Bharath Mundlapudi added a comment -

          There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like,

          "Directory %s owned by wrong user or group. "
          "Expected %d:%d and found %d:%d.\n",
          
          Show
          Bharath Mundlapudi added a comment - There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like, "Directory %s owned by wrong user or group. " "Expected %d:%d and found %d:%d.\n" ,
          Hide
          Bharath Mundlapudi added a comment -

          Attaching a patch.

          Show
          Bharath Mundlapudi added a comment - Attaching a patch.
          Hide
          Luke Lu added a comment -

          +1. lgtm.

          Show
          Luke Lu added a comment - +1. lgtm.
          Hide
          Luke Lu added a comment -

          Committed to branch-0.20-security (will be in 205). Thanks Bharath!

          Show
          Luke Lu added a comment - Committed to branch-0.20-security (will be in 205). Thanks Bharath!
          Hide
          Bharath Mundlapudi added a comment -

          Some more details:

          1. This is a port of the same change Chris Douglas made for MR-279.
          2. I have unit tested this code using test-task-controller and all tests pass.
          3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.

          Show
          Bharath Mundlapudi added a comment - Some more details: 1. This is a port of the same change Chris Douglas made for MR-279. 2. I have unit tested this code using test-task-controller and all tests pass. 3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
          Hide
          Benoy Antony added a comment -

          Patch for 22

          Show
          Benoy Antony added a comment - Patch for 22
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to branch 0.22.1. Thank you Benoy.

          Show
          Konstantin Shvachko added a comment - I just committed this to branch 0.22.1. Thank you Benoy.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/)
          MAPREDUCE-2651. Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242)

          Result = SUCCESS
          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/ ) MAPREDUCE-2651 . Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c

            People

            • Assignee:
              Bharath Mundlapudi
              Reporter:
              Bharath Mundlapudi
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development