|
[
Permlink
| « Hide
]
dhruba borthakur added a comment - 27/Jul/09 08:48 PM
Is it possible for you to find out the name of the equivalent command in Solaris? Better still, if you can submit a patch that incorates your findings, that will be great. Thanks.
I think "id -nu" is the correct command in Solaris.
I'm working on a patch to solve this issue.
You're right, on Solaris is "id -un" but this command is at "/usr/xpg4/bin/id" this's a BSD style command. The "/usr/bin/id" command is a SRV style so it has only the "-a" and "-p" options
But the question is at UnixUserGroupInformation.java the method static String getUnixUserName() throws IOException calls to executeShellCommand( new String[]{Shell.USER_NAME_COMMAND}) but i think this can be rewrite to
Or change the USER_NAME_COMMAND and getGROUPS_COMMAND What do you think is better solution? > Solaris enviroment has no whoami command ...
In this post  , they said /usr/ucb/whoam is available in Solaris.
I suggest not to change the existing codes unless whoami is absolutely unavailable. BTW, Hudson is using Solaris (see http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/
). So I guess the whoami command exists in Solaris. There're some issues and i'll try to explain what's happend onto Solaris.
1.- By default the Shell class calls to "whoami" without full path, this is a security risk Here's the example 1.- Create a new directory onto Hadoop HDFS on Solaris 10 and set the user and group to *hadoop, then set the perm to 700 so, only hadoop user can read/write onto* $ uname -a 2.- Now from another Sun Box try to run the sample application that i've attached ( test-hadoop-security.tar.gz ) this one has no whoami at path $ uname -a 3.- Run the Sample Application $ cd TestHadoop/ init: compile: dist: run: BUILD SUCCESSFUL as you can see, with the hadoop-0.20-core.jar the user is ignored, but with me hadoop-0.20.1-dev-core.jar the user and group is correct identified as postgres and group postgres and a AccessControlException is launched when try to access testdir 4.- Verify the HDFS structure now $ hadoop fs -lsr 5.- The hdfs-site.xml i've defined the value dfs.permission.supergroup to hadoop as
But the user who launched the TestApp is postgres with groups postgres and dba no hadoop The Second issue is if i make a "fake" of whois i can bypass the security layout as the above example 1.- At Hadoop Cluster clean and create the testdir 2.- At the second SunBox try to make a whoami fake $ uname -a 3.- Launch TestApp $ cd TestHadoop/ init: compile: dist: run: BUILD SUCCESSFUL As you can see, now hadoop say that user is hadoop 4.- View HDFS files $ hadoop fs -lsr I think now it's more clear no? I've attached some files to evaluate the new code. This's the sample application to test missing the whoami command at Solaris (and other)
Urko Benito made changes - 29/Jul/09 12:40 PM
Urko Benito made changes - 29/Jul/09 01:39 PM
Urko, first, thanks for testing in such great details.
You are totally correct about the security risks because the real security mechanism implementation is not completed yet (see HADOOP-4487). In order to provide permission checking in Namenode, etc., UnixUserGroupInformation simply greps the username and groups from the shell and then forwards them. We have an assumption that the whoami command is available in the OS. Note that changing whoami to other commands does not really increase security since a malicious user can modify the codes to impersonate another user. Thanks Tsz Wo,
So what do you think about the patch? Do you think it's better to wait for HADOOP-4487 be solved or to apply the patch to the current version? > So what do you think about the patch? Do you think it's better to wait for HADOOP-4487 be solved or to apply the patch to the current version?
The patch mainly changes the shell commands from whoami + groups to id and makes SunOS a special case. As mentioned before, it does not increase security. So we have to wait for HADOOP-4487 anyway. The question is whether it is good to (1) replace whoami + groups with id and (2) make SunOS a special case. Both (1) and/or (2) are incompatible changes, i.e. it potentially breaks some existing systems. I think we should minimize incompatible changes in general. Could you simply install whoami in your SunOS systems? (BTW, you may use "svn diff > a.patch" or the corresponding comnand in git for generating patches.) One can install a whoami script on the solaris machine. It could be a shell script that internally uses the id command. In that case, no changes are needed to Hadoop. is my assumption correct?
Hi
I think the solution is not to change the source code but, keep in mind that this is not the correct way (but until HADOOP-4487 is solved) i proposed a solution
I'm writing a Installing Hadoop on Solaris Guide at my blog, but at the moment is in Spanish, i'll translated it to English and add to the issue. What do you think about? +1 on adding more documentation and wiki.
Tsz Wo (Nicholas), SZE made changes - 30/Jul/09 06:35 PM
This is basically a duplicate of HADOOP-4603 which provides a very simple fix.
Tsz Wo (Nicholas), SZE made changes - 28/Aug/09 11:08 PM
I would say this is related to but not duplicates HADOOP-4603 since HADOOP-4603 assumes /usr/ucb/whoami on Solaris.
From where I sit, this is what I see happening:
a) We are afraid of writing JNI code, even tho the JNI code that needs to be written should be highly portable since it is all POSIX stuff. HADOOP-4998 b) We are not willing to write special case OS code for determining what the user is and instead dependent upon user's hacking up their path in likely incompatible ways. [what this and other bugs are about] c) We are willing to write special case OS code for dealing with memory management for the MapReduce framework. [See LinuxTaskController] So we now have (essentially) two patches that may make the situation better. Would someone explain to me why either HADOOP-4603 or this one isn't getting committed? > So we now have (essentially) two patches that may make the situation better. Would someone explain to me why either HADOOP-4603 or this one isn't getting committed?
The patch in HADOOP-4603 only add an absolute path to whoami. How does it make the situation is better? On the other hand, if someone put their whoami in a different path, say /usr/bin/whoami. The patch will breaks their systems. The part you don't understand is that Solaris ships with /usr/ucb/whoami as part of the base OS install.
Or, as an alternative, on Solaris, try path-less whoami, if it fails, try /usr/ucb/whoami specifically.
In either case, it is reprehensible that this still isn't fixed. > The part you don't understand is that Solaris ships with /usr/ucb/whoami as part of the base OS install.
Are you sure that /usr/ucb/whoami exists in all versions of Solaris? How come Urko said that "Solaris enviroment has no whoami command, ..." in the description? > Or, as an alternative, on Solaris, try path-less whoami, if it fails, try /usr/ucb/whoami specifically. I won't object doing this since this is no longer incompatible. The only versions that don't have /usr/ucb/whoami are Solaris 1.x aka SunOS 4.x. Perhaps a bit of (rough) history will help explain:
When the transition from BSD (Solaris 1.x) to SysV (Solaris 2.x) was made, there was a need to provide some backward compatibility. Given that whoami is BSD in origin, it was moved to /usr/ucb with the rest of the BSD-specific commands. [This is why /usr/bin/ps -ef (SysV) and /usr/ucb/ps auwwx (BSD) work the way they do). In addition, commands that had different semantics between SysV and POSIX were also shifted around. So if you want POSIX-style semantics for something like id, you need to use /usr/xpg4/bin/id or /usr/xpg6/bin/id . The reality is, is that if we're going to keep this inane fork()'ing for the user and group information, the patch that uses id (and /usr/xpg4/bin/id on Solaris) is probably a better fix simply because it is easy to point to a spec that says "these command line options will provide this information". whoami, being BSD, has little-to-no such spec and is standard by defacto... but only BSD and systems that want to be compatible with it. [Linux being a kitchen sink sort of OS falls into the latter.] That said, the only machines where /usr/ucb/whoami won't exist are those machines that don't install the BSD backward compat packages. Thus my comment on 'hardened' machines up above. Allen, what is the problem on Solaris if we require the path-less whoami?
/usr/ucb commands have BSD-style semantics. Putting in the path will likely break user environments and potentially break Hadoop in other ways. For example, /usr/ucb/du, df, chown, etc have different semantics than their SysV brethren in /usr/bin. /usr/ucb/cc enables /usr/ucbinclude prior to /usr/include, etc, etc, etc. Putting /usr/ucb in the path is not recommended by Sun.
Could we use alias or create a symbolic link?
Why are you so against committing this patch when there is already OS specific code in the rest of the system? Look, fork()ing for identity is already a giant turd. This just makes the turd work on another OS without the user having to do anything. So, no, I don't believe asking for a symlink is reasonable when we could make it Just Work(tm) out of the box with little-to-no effort on our part. By the time we finish this conversation, either one of these patches could have been committed and tested.
> Why are you so against committing this patch when there is already OS specific code in the rest of the system? ...
Because we have to maintain such OS specific codes forever. Once the codes have got in, it is hardly to remove them because removing them becomes an incompatible change. I do not know much details on LinuxTaskController and the use of fork(), the problems you mentioned previously. But if they are done by mistakes, they are not excuses to make more mistakes.
I agree with this sentiment about maintenance, but I don't see how this would be an incompatible change. The existence of a whoami executable is not something that's part of the API - the user assumes that most software will "just work". Calling out to "whoami" is a hack in the first place, and hopefully will be removed soon when true authentication support makes it into Hadoop. +1 for fixing this, though I don't think the patch is so great. Can you format this as a proper single-file patch against the hadoop-common repository? I'll move this to the HADOOP JIRA since this is not HDFS-specific. See http://wiki.apache.org/hadoop/HowToContribute
Todd Lipcon made changes - 18/Sep/09 09:33 PM
I think there are some questions.
System.getProperty("user.name"); As a simple note, yes, you can hack the System.getProperty with a System.setProperty
I will create a HADOOP-6272 patch file with the changes, on tomorroy with JUnit test and so Even tho I agree that maintaining OS specific code is ugly, I think it is unavoidable as long as we lack portable hooks into the OS. (No, reading the output of shell commands is not portable, this bug providing a relevant example.)
Using System.getProperty("user.name") is an interesting idea. As you say, that leaves us with group. So.... I'd like to suggest that we create a file in hadoop/bin with the sole purpose of providing portable ways to determine group information. Then all the OS-specific ugliness could be pushed into that and outside the Java code. Make it something like "hadoop-portable-groups" to lower the chances of hitting something real. You also eliminate any double fork()'s at the Java-level that would likely be the result of any OS-specific issues (consider the 'whoami' vs. the '/usr/ucb/whoami' Solaris case). > I'd like to suggest that we create a file in hadoop/bin with the sole
> purpose of providing portable ways to determine group information. +1 – amr
Tsz Wo (Nicholas), SZE made changes - 21/Sep/09 10:55 PM
{bq}
I'd like to suggest that we create a file in hadoop/bin with the sole purpose of providing portable ways to determine group information. Then all the OS-specific ugliness could be pushed into that and outside the Java code. Make it something like "hadoop-portable-groups" to lower the chances of hitting something real. You also eliminate any double fork()'s at the Java-level that would likely be the result of any OS-specific issues (consider the 'whoami' vs. the '/usr/ucb/whoami' Solaris case).{bq} There is a jira, HADOOP-4998, on this direction. No, HADOOP-4998 is about using JNI so that we can stop forking. Getting that implemented would fix this entire issue.
If someone has free cycles, I agree that fixing HADOOP-4998 would be great.
In the mean time, I don't want to hard code paths for each particular operating system. If we take a patch for Solaris, it doesn't make sense to reject patches that add the path for the whoami command under BSD, etc. In the short term, I'd suggest documenting the soft link fix in the hadoop/bin directory. In the medium (0.22) term, we'll have Kerberos and Unix login via Jaas and groups done server side via HADOOP-4656. |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
|
Improvement
Open
Major
Incorrect UserName at Solaris because it has no "whoami" command by default
