Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6435

Add support for specifying a static uid/gid mapping for the NFS gateway

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: nfs
    • Labels:
      None
    • Target Version/s:

      Description

      It's quite reasonable that folks will want to access the HDFS NFS Gateway from client machines where the UIDs/GIDs do not line up with those on the NFS Gateway itself. We should provide a way to map these UIDs/GIDs between the systems.

      1. HDFS-6435.patch
        15 kB
        Aaron T. Myers
      2. HDFS-6435.patch
        15 kB
        Aaron T. Myers

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5608 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5608/)
          HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5608 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5608/ ) HDFS-6435 . Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966 ) /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1755 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1755/)
          HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1755 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1755/ ) HDFS-6435 . Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966 ) /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1781 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1781/)
          HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1781 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1781/ ) HDFS-6435 . Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966 ) /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #563 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/563/)
          HDFS-6435. Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #563 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/563/ ) HDFS-6435 . Add support for specifying a static uid/gid mapping for the NFS gateway. (atm via wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1596966 ) /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/IdUserGroup.java /hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/nfs/nfs3/TestIdUserGroup.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
          Hide
          Andrew Wang added a comment -

          Committed to trunk and branch-2, thanks for your contribution Aaron.

          Show
          Andrew Wang added a comment - Committed to trunk and branch-2, thanks for your contribution Aaron.
          Hide
          Andrew Wang added a comment -

          I ran these tests locally, i think they're just flakes. I'm still +1, will commit shortly.

          Show
          Andrew Wang added a comment - I ran these tests locally, i think they're just flakes. I'm still +1, will commit shortly.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12646122/HDFS-6435.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.TestDistributedFileSystem
          org.apache.hadoop.fs.TestHdfsNativeCodeLoader

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6950//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6950//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12646122/HDFS-6435.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.TestDistributedFileSystem org.apache.hadoop.fs.TestHdfsNativeCodeLoader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6950//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6950//console This message is automatically generated.
          Hide
          Andrew Wang added a comment -

          +1 pending, LGTM!

          Show
          Andrew Wang added a comment - +1 pending, LGTM!
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the review, Andrew. Those are all good suggestions. Done.

          I also took this opportunity to add a little logging about which static mapping file is being used (or if the file doesn't exist). I also fixed a parsing bug I noticed while addressing your review feedback which caused lines which end in a comment, without white space before the comment, to cause a parse error.

          Please have a look at this updated patch and let me know if it looks good to you.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the review, Andrew. Those are all good suggestions. Done. I also took this opportunity to add a little logging about which static mapping file is being used (or if the file doesn't exist). I also fixed a parsing bug I noticed while addressing your review feedback which caused lines which end in a comment, without white space before the comment, to cause a parse error. Please have a look at this updated patch and let me know if it looks good to you.
          Hide
          Andrew Wang added a comment -

          Hi ATM, thanks for working on this. I only have some nitty review comments:

          IdUserGroup:

          • Extra newline by the new config key
          • Minor, but do you want to define the Patterns for the regexs in parseStaticMap once rather than compiling them each time?
          • On comment parsing, the startsWith and matches at the top look like the do the same thing, did you mean for the matches to check for zero or more spaces followed by a #? Could maybe get rid of the startsWith entirely then. There's also another comment check in lineComponents[0] that might be unnecessary since the entire line is already verified with the big regex.

          +1 once addressed, mostly optional.

          Show
          Andrew Wang added a comment - Hi ATM, thanks for working on this. I only have some nitty review comments: IdUserGroup: Extra newline by the new config key Minor, but do you want to define the Patterns for the regexs in parseStaticMap once rather than compiling them each time? On comment parsing, the startsWith and matches at the top look like the do the same thing, did you mean for the matches to check for zero or more spaces followed by a # ? Could maybe get rid of the startsWith entirely then. There's also another comment check in lineComponents [0] that might be unnecessary since the entire line is already verified with the big regex. +1 once addressed, mostly optional.
          Hide
          Aaron T. Myers added a comment -

          I should also mention that I just did some manual testing between a client machine with a completely different set of UIDs/GIDs from the machine running the NFS Gateway, and this feature works as expected once the static mapping file is configured correctly. In my case, I used the following mapping file, and while browsing HDFS over NFS was able to see the correct user/group names displayed:

          $ cat /etc/nfs.map
          uid 1037  1000 # atm
          gid 516   1000 # atm
          uid 17    115  # hdfs
          gid 18    125  # hdfs
          gid 23    999  # supergroup
          
          Show
          Aaron T. Myers added a comment - I should also mention that I just did some manual testing between a client machine with a completely different set of UIDs/GIDs from the machine running the NFS Gateway, and this feature works as expected once the static mapping file is configured correctly. In my case, I used the following mapping file, and while browsing HDFS over NFS was able to see the correct user/group names displayed: $ cat /etc/nfs.map uid 1037 1000 # atm gid 516 1000 # atm uid 17 115 # hdfs gid 18 125 # hdfs gid 23 999 # supergroup
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12645761/HDFS-6435.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.fs.TestHdfsNativeCodeLoader

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6940//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6940//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12645761/HDFS-6435.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-nfs hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.TestHdfsNativeCodeLoader +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6940//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6940//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Here's a patch which implements the feature by adding support for specifying an arbitrary static UID/GID mapping file. The format of the file is modeled after what's specified in the exports(5) manual page.

          Please review.

          Show
          Aaron T. Myers added a comment - Here's a patch which implements the feature by adding support for specifying an arbitrary static UID/GID mapping file. The format of the file is modeled after what's specified in the exports(5) manual page. Please review.

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Aaron T. Myers
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development