Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2651

Race condition in Linux Task Controller for job log directory creation

        Details

        • Type: Bug Bug
        • Status: Closed
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: 0.20.204.0
        • Fix Version/s: 0.20.204.0
        • Component/s: task-controller
        • Labels:
          None
        • Hadoop Flags:
          Reviewed
        • Tags:
          task controller,

          Description

          There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time.

          1. Text File
            MAPREDUCE-2651-1.patch
            2 kB
            Bharath Mundlapudi
          2. Text File
            mr-2651-0.22.patch
            2 kB
            Benoy Antony

            Issue Links

            is depended upon by

            Task - A task that needs to be done. HADOOP-8357 Restore security in Hadoop 0.22 branch

            • Major - Major loss of function.
            • Resolved
            is related to

            New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-2657 TaskTracker should handle disk failures

            • Major - Major loss of function.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              Bharath Mundlapudi added a comment -

              There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like,

              "Directory %s owned by wrong user or group. "
"Expected %d:%d and found %d:%d.\n",
              Show
              Bharath Mundlapudi added a comment - There is a rare race condition in linux task controller when concurrent task processes tries to create job log directory at the same time. Since we change the permissions for a brief moment, at this time, if another task tries to create a directory then the later task will fail with the error like, "Directory %s owned by wrong user or group. " "Expected %d:%d and found %d:%d.\n" ,
              Hide
              Permalink
              Bharath Mundlapudi added a comment -

              Attaching a patch.

              Show
              Bharath Mundlapudi added a comment - Attaching a patch.
              Hide
              Permalink
              Luke Lu added a comment -

              +1. lgtm.

              Show
              Luke Lu added a comment - +1. lgtm.
              Hide
              Permalink
              Luke Lu added a comment -

              Committed to branch-0.20-security (will be in 205). Thanks Bharath!

              Show
              Luke Lu added a comment - Committed to branch-0.20-security (will be in 205). Thanks Bharath!
              Hide
              Permalink
              Bharath Mundlapudi added a comment -

              Some more details:

              1. This is a port of the same change Chris Douglas made for MR-279.
              2. I have unit tested this code using test-task-controller and all tests pass.
              3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.

              Show
              Bharath Mundlapudi added a comment - Some more details: 1. This is a port of the same change Chris Douglas made for MR-279. 2. I have unit tested this code using test-task-controller and all tests pass. 3. Also, I have tested on a multiple user cluster with security and linux task controller configured. And verified all the permissions on the directories.
              Hide
              Permalink
              Owen O'Malley added a comment -

              Hadoop 0.20.204.0 was just released.

              Show
              Owen O'Malley added a comment - Hadoop 0.20.204.0 was just released.
              Hide
              Permalink
              Benoy Antony added a comment -

              Patch for 22

              Show
              Benoy Antony added a comment - Patch for 22
              Hide
              Permalink
              Konstantin Shvachko added a comment -

              I just committed this to branch 0.22.1. Thank you Benoy.

              Show
              Konstantin Shvachko added a comment - I just committed this to branch 0.22.1. Thank you Benoy.
              Hide
              Permalink
              Hudson added a comment -

              Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/)
              MAPREDUCE-2651. Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242)

              Result = SUCCESS
              shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242
              Files :

              • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
              • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c
              Show
              Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/ ) MAPREDUCE-2651 . Fix race condition in Linux task controller for job log directory creation. Contributed by Bharath Mundlapudi and Benoy Antony. (Revision 1346242) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346242 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/impl/task-controller.c

                People

                • Assignee:
                  Bharath Mundlapudi
                  Reporter:
                  Bharath Mundlapudi
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  7 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development