Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2376

test-task-controller fails if run as a userid < 1000

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.20.205.0, 0.22.0
    • Fix Version/s: 1.1.0, 0.22.1
    • Component/s: task-controller, test
    • Labels:
      None

      Description

      test-task-controller tries to verify that the task-controller won't run on behalf of users with uid < 1000. This makes the test fail when running in some test environments - eg our hudson jobs internally run as a system user with uid 101.

      1. mr-2376-0.22.patch
        0.6 kB
        Konstantin Shvachko
      2. mapreduce-2376-20.txt
        0.7 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          This is for the task-controller after MAPREDUCE-2178.

          The test program allows the tester to pass in a username to use instead of the current user on the command line. However, there's no way to pass this from the ant task which calls "make check".

          Instead I think we should use an env var which is easier to pass via make.

          Show
          Todd Lipcon added a comment - This is for the task-controller after MAPREDUCE-2178 . The test program allows the tester to pass in a username to use instead of the current user on the command line. However, there's no way to pass this from the ant task which calls "make check". Instead I think we should use an env var which is easier to pass via make.
          Hide
          Allen Wittenauer added a comment -

          IIRC, the history behind this was that on most UNIX and UNIX-like systems (OS X being a big exception) assigns valid user accounts uid > 1000 when using their utilities. So this check was a safety to prevent accounts with special privileges from executing it.

          Show
          Allen Wittenauer added a comment - IIRC, the history behind this was that on most UNIX and UNIX-like systems (OS X being a big exception) assigns valid user accounts uid > 1000 when using their utilities. So this check was a safety to prevent accounts with special privileges from executing it.
          Hide
          Todd Lipcon added a comment -

          Here's a patch vs 0.20. Waiting on MAPREDUCE-2178 to do this for trunk.

          Show
          Todd Lipcon added a comment - Here's a patch vs 0.20. Waiting on MAPREDUCE-2178 to do this for trunk.
          Hide
          Todd Lipcon added a comment -

          Allen - yep, makes sense. For a deployment it's configurable in taskcontroller.cfg. The issue is just that when running tests there should be a way to bypass this, since you don't always run tests as a user >1000, and the test writes its own configuration inline.

          Show
          Todd Lipcon added a comment - Allen - yep, makes sense. For a deployment it's configurable in taskcontroller.cfg. The issue is just that when running tests there should be a way to bypass this, since you don't always run tests as a user >1000, and the test writes its own configuration inline.
          Hide
          Wing Yew Poon added a comment -

          On CentOS 5.5 (and possibly other systems), non-system users have id >= 500, not 1000.
          The test should check that the user id is between UID_MIN and UID_MAX in /etc/login.defs of the system.
          We are being bitten by this issue on our kerberos-secured cluster.

          Show
          Wing Yew Poon added a comment - On CentOS 5.5 (and possibly other systems), non-system users have id >= 500, not 1000. The test should check that the user id is between UID_MIN and UID_MAX in /etc/login.defs of the system. We are being bitten by this issue on our kerberos-secured cluster.
          Hide
          Tim Broberg added a comment -

          I am also seeing this trying to build 0.23 under Fedora. How are others working around this?

          Show
          Tim Broberg added a comment - I am also seeing this trying to build 0.23 under Fedora. How are others working around this?
          Hide
          Tim Broberg added a comment -

          I tried the proposed patch, but it's not changing users for me unless I'm running as root (please inform me if there is a better way to handle that), so that doesn't really help my situation either.

          What would be useful would be to be able to configure the conf file used for the test to modify the minimum uid, or better yet figure it out from context.

          As a workaround, I hacked DEFAULT_MIN_USERID in the various C source files to set it to 500 and it works for the moment.

          Show
          Tim Broberg added a comment - I tried the proposed patch, but it's not changing users for me unless I'm running as root (please inform me if there is a better way to handle that), so that doesn't really help my situation either. What would be useful would be to be able to configure the conf file used for the test to modify the minimum uid, or better yet figure it out from context. As a workaround, I hacked DEFAULT_MIN_USERID in the various C source files to set it to 500 and it works for the moment.
          Hide
          Eli Collins added a comment -

          +1 lgtm

          Show
          Eli Collins added a comment - +1 lgtm
          Hide
          Eli Collins added a comment -

          I've committed this for 206. Leaving open for trunk, 22 and 23.

          Show
          Eli Collins added a comment - I've committed this for 206. Leaving open for trunk, 22 and 23.
          Hide
          Benoy Antony added a comment -

          Patch for 0.22

          Show
          Benoy Antony added a comment - Patch for 0.22
          Hide
          Konstantin Shvachko added a comment -

          Attachments to MAPREDUCE-2377 and MAPREDUCE-2376 were intermixed. Attaching the right one.

          Show
          Konstantin Shvachko added a comment - Attachments to MAPREDUCE-2377 and MAPREDUCE-2376 were intermixed. Attaching the right one.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this to branch 0.22.1. Thank you Todd and Benoy.
          Will close it if not needed in other versions.

          Show
          Konstantin Shvachko added a comment - I just committed this to branch 0.22.1. Thank you Todd and Benoy. Will close it if not needed in other versions.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/)
          MAPREDUCE-2376. test-task-controller fails if run as a userid < 1000. Contributed by Todd Lipcon and Benoy Antony. (Revision 1346256)

          Result = SUCCESS
          shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346256
          Files :

          • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
          • /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/test/test-task-controller.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #104 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/104/ ) MAPREDUCE-2376 . test-task-controller fails if run as a userid < 1000. Contributed by Todd Lipcon and Benoy Antony. (Revision 1346256) Result = SUCCESS shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1346256 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/c++/task-controller/test/test-task-controller.c
          Hide
          Matt Foley added a comment -

          Made Target Versions and Fix Versions consistent.
          Leaving bug open because the fix is still missing from Hadoop-2 line(s) and trunk.

          Show
          Matt Foley added a comment - Made Target Versions and Fix Versions consistent. Leaving bug open because the fix is still missing from Hadoop-2 line(s) and trunk.
          Hide
          Matt Foley added a comment -

          Hoping this will be picked up in 2.0.3-alpha.
          Removed 0.23 as target, since believed moribund.

          Show
          Matt Foley added a comment - Hoping this will be picked up in 2.0.3-alpha. Removed 0.23 as target, since believed moribund.

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development