Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      On Linux platforms where user IDs start at 500 rather than 1000, the build currently is broken. This includes CentOS, RHEL, Fedora, SuSE, and probably most other Linux platforms. It does happen to work on Debian and Ubuntu, which explains why Jenkins hasn't caught it yet.

      Other users will see something like this:

      [INFO] Requested user cmccabe has id 500, which is below the minimum allowed 1000
      [INFO] FAIL: test-container-executor
      [INFO] ================================================
      [INFO] 1 of 1 test failed
      [INFO] Please report to mapreduce-dev@hadoop.apache.org
      [INFO] ================================================
      [INFO] make[1]: *** [check-TESTS] Error 1
      [INFO] make[1]: Leaving directory `/home/cmccabe/hadoop4/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn
      -server/hadoop-yarn-server-nodemanager/target/native/container-executor'
      

      And then the build fails. Since native unit tests are currently unskippable (HADOOP-8480) this makes the project unbuildable.

      The easy solution to this is to relax the constraint for the unit test. Since the unit test already writes its own configuration file, we just need to change it there.

      In general, I believe that it would make sense to change this to 500 across the board. I'm not aware of any Linuxes that create system users with IDs higher than or equal to 500. System user IDs tend to be below 200.

      However, if we do nothing else, we should at least fix the build by relaxing the constraint for unit tests.

      1. HADOOP-8499.002.patch
        0.9 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Colin Patrick McCabe created issue -
          Colin Patrick McCabe made changes -
          Field Original Value New Value
          Link This issue is related to MAPREDUCE-2376 [ MAPREDUCE-2376 ]
          Colin Patrick McCabe made changes -
          Attachment HADOOP-8499.002.patch [ 12531464 ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Link This issue is related to HADOOP-8480 [ HADOOP-8480 ]
          Hide
          Aaron T. Myers added a comment -

          I'm not aware of any Linuxes that create system users with IDs higher than or equal to 500.

          Debian and Ubuntu do by default, I believe.

          Show
          Aaron T. Myers added a comment - I'm not aware of any Linuxes that create system users with IDs higher than or equal to 500. Debian and Ubuntu do by default, I believe.
          Hide
          Colin Patrick McCabe added a comment -
          cmccabe@vm0:~$ cat /etc/issue
          Ubuntu 12.04 LTS \n \l
          
          cmccabe@vm0:~$ cat /etc/passwd
          root:x:0:0:root:/root:/bin/bash
          daemon:x:1:1:daemon:/usr/sbin:/bin/sh
          bin:x:2:2:bin:/bin:/bin/sh
          sys:x:3:3:sys:/dev:/bin/sh
          sync:x:4:65534:sync:/bin:/bin/sync
          games:x:5:60:games:/usr/games:/bin/sh
          man:x:6:12:man:/var/cache/man:/bin/sh
          lp:x:7:7:lp:/var/spool/lpd:/bin/sh
          mail:x:8:8:mail:/var/mail:/bin/sh
          news:x:9:9:news:/var/spool/news:/bin/sh
          uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
          proxy:x:13:13:proxy:/bin:/bin/sh
          www-data:x:33:33:www-data:/var/www:/bin/sh
          backup:x:34:34:backup:/var/backups:/bin/sh
          list:x:38:38:Mailing List Manager:/var/list:/bin/sh
          irc:x:39:39:ircd:/var/run/ircd:/bin/sh
          gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
          nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
          libuuid:x:100:101::/var/lib/libuuid:/bin/sh
          syslog:x:101:103::/home/syslog:/bin/false
          messagebus:x:102:105::/var/run/dbus:/bin/false
          whoopsie:x:103:106::/nonexistent:/bin/false
          landscape:x:104:109::/var/lib/landscape:/bin/false
          sshd:x:105:65534::/var/run/sshd:/usr/sbin/nologin
          cmccabe:x:1000:1000:cmccabe,,,:/home/cmccabe:/bin/bash
          

          Notice that there are no user IDs between 500 and 1000 whatsoever.

          Also notice that nobody is user id 65534, so we'll happily allow people to run mapreduce jobs as it (apparently).

          This whole "check the numberic value of the user id" thing stinks. I think we should either be validating against a whitelist, or checking if the given user has a valid login shell.

          Show
          Colin Patrick McCabe added a comment - cmccabe@vm0:~$ cat /etc/issue Ubuntu 12.04 LTS \n \l cmccabe@vm0:~$ cat /etc/passwd root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/bin/sh bin:x:2:2:bin:/bin:/bin/sh sys:x:3:3:sys:/dev:/bin/sh sync:x:4:65534:sync:/bin:/bin/sync games:x:5:60:games:/usr/games:/bin/sh man:x:6:12:man:/ var /cache/man:/bin/sh lp:x:7:7:lp:/ var /spool/lpd:/bin/sh mail:x:8:8:mail:/ var /mail:/bin/sh news:x:9:9:news:/ var /spool/news:/bin/sh uucp:x:10:10:uucp:/ var /spool/uucp:/bin/sh proxy:x:13:13:proxy:/bin:/bin/sh www-data:x:33:33:www-data:/ var /www:/bin/sh backup:x:34:34:backup:/ var /backups:/bin/sh list:x:38:38:Mailing List Manager:/ var /list:/bin/sh irc:x:39:39:ircd:/ var /run/ircd:/bin/sh gnats:x:41:41:Gnats Bug-Reporting System (admin):/ var /lib/gnats:/bin/sh nobody:x:65534:65534:nobody:/nonexistent:/bin/sh libuuid:x:100:101::/ var /lib/libuuid:/bin/sh syslog:x:101:103::/home/syslog:/bin/ false messagebus:x:102:105::/ var /run/dbus:/bin/ false whoopsie:x:103:106::/nonexistent:/bin/ false landscape:x:104:109::/ var /lib/landscape:/bin/ false sshd:x:105:65534::/ var /run/sshd:/usr/sbin/nologin cmccabe:x:1000:1000:cmccabe,,,:/home/cmccabe:/bin/bash Notice that there are no user IDs between 500 and 1000 whatsoever. Also notice that nobody is user id 65534, so we'll happily allow people to run mapreduce jobs as it (apparently). This whole "check the numberic value of the user id" thing stinks. I think we should either be validating against a whitelist, or checking if the given user has a valid login shell.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12531464/HADOOP-8499.002.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1103//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1103//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12531464/HADOOP-8499.002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1103//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1103//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Notice that there are no user IDs between 500 and 1000 whatsoever.

          That's just because you haven't created 400+ system users.

          [06:40:34] atm@simon:~$ cat /etc/issue
          Ubuntu 12.04 LTS \n \l
          
          [06:41:00] atm@simon:~$ grep UID /etc/login.defs 
          UID_MIN			 1000
          UID_MAX			60000
          #SYS_UID_MIN		  100
          #SYS_UID_MAX		  999
          

          Even though the SYS_UID_ lines are commented out, I believe they're still the default values on Ubuntu.

          Show
          Aaron T. Myers added a comment - Notice that there are no user IDs between 500 and 1000 whatsoever. That's just because you haven't created 400+ system users. [06:40:34] atm@simon:~$ cat /etc/issue Ubuntu 12.04 LTS \n \l [06:41:00] atm@simon:~$ grep UID /etc/login.defs UID_MIN 1000 UID_MAX 60000 #SYS_UID_MIN 100 #SYS_UID_MAX 999 Even though the SYS_UID_ lines are commented out, I believe they're still the default values on Ubuntu.
          Hide
          Robert Joseph Evans added a comment -

          I think changing it to 500 for the tests is fine. The patch is doing it only for testing, and we are not actually running with the setuid bit on for this part of the tests so there is no risk of a security issue. It would probably be more maintainable to change it so that it sets the min id to be whatever the current user id is. That way we don't have to worry about it in the future is something else changes, or someone does install 501 system users .

          Show
          Robert Joseph Evans added a comment - I think changing it to 500 for the tests is fine. The patch is doing it only for testing, and we are not actually running with the setuid bit on for this part of the tests so there is no risk of a security issue. It would probably be more maintainable to change it so that it sets the min id to be whatever the current user id is. That way we don't have to worry about it in the future is something else changes, or someone does install 501 system users .
          Hide
          Eli Collins added a comment -

          ATM, reasonable to lower the min id to 500 for the tests?

          Show
          Eli Collins added a comment - ATM, reasonable to lower the min id to 500 for the tests?
          Eli Collins made changes -
          Summary fix mvn compile -Pnative on CentOS / RHEL / Fedora / SuSE / etc Lower min.user.id to 500 for the tests
          Affects Version/s 2.0.0-alpha [ 12320352 ]
          Target Version/s 2.0.1-alpha [ 12321441 ]
          Priority Major [ 3 ] Minor [ 4 ]
          Hide
          Aaron T. Myers added a comment -

          ATM, reasonable to lower the min id to 500 for the tests?

          Sure, fine by me.

          Show
          Aaron T. Myers added a comment - ATM, reasonable to lower the min id to 500 for the tests? Sure, fine by me.
          Hide
          Eli Collins added a comment -

          +1 patch lgtm

          Show
          Eli Collins added a comment - +1 patch lgtm
          Hide
          Eli Collins added a comment -

          I've committed this and merged to branch-2. Thanks Colin.

          Show
          Eli Collins added a comment - I've committed this and merged to branch-2. Thanks Colin.
          Eli Collins made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 2.0.1-alpha [ 12321441 ]
          Fix Version/s 2.0.1-alpha [ 12321441 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2532 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2532/)
          HADOOP-8499. Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2532 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2532/ ) HADOOP-8499 . Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2467 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2467/)
          HADOOP-8499. Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2467 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2467/ ) HADOOP-8499 . Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2487 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2487/)
          HADOOP-8499. Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2487 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2487/ ) HADOOP-8499 . Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/)
          HADOOP-8499. Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438)

          Result = FAILURE
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1103 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1103/ ) HADOOP-8499 . Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/)
          HADOOP-8499. Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438)

          Result = SUCCESS
          eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438
          Files :

          • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1136 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1136/ ) HADOOP-8499 . Lower min.user.id to 500 for the tests. Contributed Colin Patrick McCabe (Revision 1361438) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1361438 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322473 ]
          Fix Version/s 2.1.0-alpha [ 12321441 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2m 9s 1 Colin Patrick McCabe 08/Jun/12 23:39
          Patch Available Patch Available Resolved Resolved
          35d 25m 1 Eli Collins 14/Jul/12 00:05
          Resolved Resolved Closed Closed
          89d 18h 39m 1 Arun C Murthy 11/Oct/12 18:45

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development