Hadoop Common
  1. Hadoop Common
  2. HADOOP-6978

Add JNI support for secure IO operations

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: io, native, security
    • Labels:
      None

      Description

      In support of MAPREDUCE-2096, we need to add some JNI functionality. In particular, we need the ability to use fstat() on an open file stream, and to use open() with O_EXCL, O_NOFOLLOW, and without O_CREAT.

      1. hadoop-6978.txt
        1.80 MB
        Todd Lipcon
      2. fstat.patch
        0.9 kB
        Devaraj Das
      3. hadoop-6978.txt
        1.77 MB
        Todd Lipcon
      4. hadoop-6978.txt
        1.77 MB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          Here's a patch against trunk. It's large due to regenerating autotools - I agree with a previous comment (I think by Arun) that we should stop checking this stuff in some day, but that's out of scope for this patch.

          Show
          Todd Lipcon added a comment - Here's a patch against trunk. It's large due to regenerating autotools - I agree with a previous comment (I think by Arun) that we should stop checking this stuff in some day, but that's out of scope for this patch.
          Hide
          Todd Lipcon added a comment -

          Patch up on review board at:
          https://reviews.apache.org/r/52/

          Show
          Todd Lipcon added a comment - Patch up on review board at: https://reviews.apache.org/r/52/
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459125/hadoop-6978.txt
          against trunk revision 1032730.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459125/hadoop-6978.txt against trunk revision 1032730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. -1 javac. The applied patch generated 1050 javac compiler warnings (more than the trunk's current 1048 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/81//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          New patch revision fixes the javac warnings (missing serialVersionUID on the new Exceptions) and the javadoc warning (had a @see reference to an MR class)

          Show
          Todd Lipcon added a comment - New patch revision fixes the javac warnings (missing serialVersionUID on the new Exceptions) and the javadoc warning (had a @see reference to an MR class)
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12459130/hadoop-6978.txt
          against trunk revision 1032730.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459130/hadoop-6978.txt against trunk revision 1032730. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/82//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          We have noticed that sometimes the C calls like getpwuid_r ends up making direct calls to the ldap server. It probably is configuration/environment specific, but in Yahoo! the password entries are maintained by the ldap server. In order to prevent ldap servers from getting overloaded with password look-ups, we have a daemon called nscd run on all the compute nodes, that caches the results of such look-ups. The calls such as getpwuid_r should terminate at the local nscd daemon, but if, for whatever reason, the nscd daemon is down on the node, the calls end up talking to the ldap server directly. Apparently, nscd is not that stable...

          We have seen the above happening at Yahoo! and in a couple of occasions brought down the ldap servers. So I was wondering whether we should reduce the number of calls to the getpwuid_r and such by caching the resolutions

          {uid,gid}

          ->

          {username,groupname}

          in Hadoop.. Thoughts?

          Show
          Devaraj Das added a comment - We have noticed that sometimes the C calls like getpwuid_r ends up making direct calls to the ldap server. It probably is configuration/environment specific, but in Yahoo! the password entries are maintained by the ldap server. In order to prevent ldap servers from getting overloaded with password look-ups, we have a daemon called nscd run on all the compute nodes, that caches the results of such look-ups. The calls such as getpwuid_r should terminate at the local nscd daemon, but if, for whatever reason, the nscd daemon is down on the node, the calls end up talking to the ldap server directly. Apparently, nscd is not that stable... We have seen the above happening at Yahoo! and in a couple of occasions brought down the ldap servers. So I was wondering whether we should reduce the number of calls to the getpwuid_r and such by caching the resolutions {uid,gid} -> {username,groupname} in Hadoop.. Thoughts?
          Hide
          Todd Lipcon added a comment -

          Hey Devaraj, That seems fairly reasonable (adding a cache here). Since this is a blocker security bug for 0.22, though, maybe we should get this in and then add the cache as a follow-on? If you think this is un-deployable without the cache we may as well do it now.

          Show
          Todd Lipcon added a comment - Hey Devaraj, That seems fairly reasonable (adding a cache here). Since this is a blocker security bug for 0.22, though, maybe we should get this in and then add the cache as a follow-on? If you think this is un-deployable without the cache we may as well do it now.
          Hide
          Devaraj Das added a comment -

          Yeah, our belief is that the shuffle process ends up making a lot of the getpw* calls and we have already seen a couple of ldap servers outages. We can do a follow up patch though. If the cluster has a configuration similar to what i mentioned earlier, then yeah, it'd be really good to have this cache before deployment...

          Show
          Devaraj Das added a comment - Yeah, our belief is that the shuffle process ends up making a lot of the getpw* calls and we have already seen a couple of ldap servers outages. We can do a follow up patch though. If the cluster has a configuration similar to what i mentioned earlier, then yeah, it'd be really good to have this cache before deployment...
          Hide
          Rajiv Chittajallu added a comment -

          That seems fairly reasonable (adding a cache here).

          Systems generally have nscd or an equivalent to handle nss cache. I think we should leave that uid/gid caching to underlying system.

          Most of the getpw calls are for uid to username lookups since hadoop deals only with username. Wouldn't it be simple to pass the uid along with the username when the JT hands off the task to the tt?

          Show
          Rajiv Chittajallu added a comment - That seems fairly reasonable (adding a cache here). Systems generally have nscd or an equivalent to handle nss cache. I think we should leave that uid/gid caching to underlying system. Most of the getpw calls are for uid to username lookups since hadoop deals only with username. Wouldn't it be simple to pass the uid along with the username when the JT hands off the task to the tt?
          Hide
          Devaraj Das added a comment -

          We could move the discussion/fix on the caching to a separate jira.
          In testing the patch, it was found that NativeIO fails when the map outputs are large. Owen fixed this issue (patch attached). We should include the fix in this patch.

          +1 on the patch otherwise.

          Show
          Devaraj Das added a comment - We could move the discussion/fix on the caching to a separate jira. In testing the patch, it was found that NativeIO fails when the map outputs are large. Owen fixed this issue (patch attached). We should include the fix in this patch. +1 on the patch otherwise.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12460400/fstat.patch
          against trunk revision 1038493.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/121//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12460400/fstat.patch against trunk revision 1038493. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/121//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Here's updated patch including Owen's fix to add AC_SYS_LARGEFILE (fstat.patch)

          Also fixed a javadoc warning and added serialVersionUID fields to the exceptions

          Show
          Todd Lipcon added a comment - Here's updated patch including Owen's fix to add AC_SYS_LARGEFILE (fstat.patch) Also fixed a javadoc warning and added serialVersionUID fields to the exceptions
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12460406/hadoop-6978.txt
          against trunk revision 1038493.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 5 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//testReport/
          Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12460406/hadoop-6978.txt against trunk revision 1038493. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/122//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          I just committed this to 0.22 and trunk. Thanks Todd and Owen!

          Show
          Devaraj Das added a comment - I just committed this to 0.22 and trunk. Thanks Todd and Owen!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #450 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/450/)
          HADOOP-6978. Adds support for NativeIO using JNI. Contributed by Todd Lipcon, Devaraj Das & Owen O'Malley.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #450 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/450/ ) HADOOP-6978 . Adds support for NativeIO using JNI. Contributed by Todd Lipcon, Devaraj Das & Owen O'Malley.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #534 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/534/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #534 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/534/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-22-branch #24 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/24/)
          HADOOP-6978. svn merge -c 1070021 from trunk

          Show
          Hudson added a comment - Integrated in Hadoop-Common-22-branch #24 (See https://hudson.apache.org/hudson/job/Hadoop-Common-22-branch/24/ ) HADOOP-6978 . svn merge -c 1070021 from trunk

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development