Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2651

Race condition in Linux Task Controller for job log directory creation

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.204.0
    • Fix Version/s: 0.20.204.0
    • Component/s: task-controller
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Tags:
      task controller,

      Description

      There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time.

      1. MAPREDUCE-2651-1.patch
        2 kB
        Bharath Mundlapudi
      2. mr-2651-0.22.patch
        2 kB
        Benoy Antony

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          20d 6m 1 Luke Lu 27/Jul/11 00:16
          Resolved Resolved Closed Closed
          37d 22h 56m 1 Owen O'Malley 02/Sep/11 23:13
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/)
          MAPREDUCE-2651. Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242)

          Result = SUCCESS
          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/ ) MAPREDUCE-2651 . Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to branch 0.22.1. Thank you Benoy.

          Show
          Konstantin Shvachko added a comment - I just committed this to branch 0.22.1. Thank you Benoy.
          Benoy Antony made changes -
          Attachment mr-2651-0.22.patch [ 12526264 ]
          Hide
          Benoy Antony added a comment -

          Patch for 22

          Show
          Benoy Antony added a comment - Patch for 22
          Benoy Antony made changes -
          Link This issue is depended upon by HADOOP-8357 [ HADOOP-8357 ]
          Matt Foley made changes -
          Fix Version/s 0.20.205.0 [ 12316391 ]
          Owen O'Malley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Owen O'Malley added a comment -

          Hadoop 0.20.204.0 was just released.

          Show
          Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
          Owen O'Malley made changes -
          Fix Version/s 0.20.204.0 [ 12316318 ]
          Hide
          Bharath Mundlapudi added a comment -

          Some more details:

          1. This is a port of the same change Chris Douglas made for MR-279.
          2. I have unit tested this code using test-task-controller and all tests pass.
          3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.

          Show
          Bharath Mundlapudi added a comment - Some more details: 1. This is a port of the same change Chris Douglas made for MR-279. 2. I have unit tested this code using test-task-controller and all tests pass. 3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.
          Luke Lu made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Tags task controller,
          Hide
          Luke Lu added a comment -

          Committed to branch-0.20-security (will be in 205). Thanks Bharath!

          Show
          Luke Lu added a comment - Committed to branch-0.20-security (will be in 205). Thanks Bharath!
          Hide
          Luke Lu added a comment -

          +1. lgtm.

          Show
          Luke Lu added a comment - +1. lgtm.
          Bharath Mundlapudi made changes -
          Attachment MAPREDUCE-2651-1.patch [ 12487911 ]
          Hide
          Bharath Mundlapudi added a comment -

          Attaching a patch.

          Show
          Bharath Mundlapudi added a comment - Attaching a patch.
          Bharath Mundlapudi made changes -
          Field Original Value New Value
          Link This issue is related to MAPREDUCE-2657 [ MAPREDUCE-2657 ]
          Hide
          Bharath Mundlapudi added a comment -

          There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like,

          "Directory %s owned by wrong user or group. "
          "Expected %d:%d and found %d:%d.\n",
          
          Show
          Bharath Mundlapudi added a comment - There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like, "Directory %s owned by wrong user or group. " "Expected %d:%d and found %d:%d.\n" ,
          Bharath Mundlapudi created issue -

            People

            • Assignee:
              Bharath Mundlapudi
              Reporter:
              Bharath Mundlapudi
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development