Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14738

Reading and writing HDFS data through HDFS nfs3 server often getting stuck

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.7.2
    • None
    • nfs
    • None

    Description

       We know the NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of the client’s local file system.
      I start the portmap and nfs3server in the hadoop node,and use the command (mount -t nfs -o vers=3,proto=tcp,nolock nfs3serverIP:/ /hdfs) to mount.
      It works well。Then i ues this command to mount on many client.
      And i used linux cp to copy files to the mounted dir(/hdfs), found the speed was very slow . There is a lot of information in the log as the follow:

      2019-08-14 15:36:28,347 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:37:03,093 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:37:03,850 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:37:14,780 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:37:14,928 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:37:28,410 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:38:09,310 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:38:10,069 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:38:14,856 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:38:14,957 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:38:28,475 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:39:14,923 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:39:14,987 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:39:15,541 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:39:16,287 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:39:28,530 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:15,015 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:15,024 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:15,028 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:21,757 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:22,508 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:40:28,583 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:41:15,063 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:41:15,086 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:41:27,983 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:41:28,647 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:41:28,729 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:42:15,184 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:42:15,191 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
      2019-08-14 15:42:28,707 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

       

      Then i print the jstack log of nfs3 server jvm, and i found that many threads waiting for monitor entry : BLOCKED(on object monitor). After checking the codes of ShellBasedIdMapping, found that It will invoke the updateMapInternal every time if the user or group doesn't exits. And many methods in this class are made with synchronized. So i think we can avoid to invoke updateMapInternal  every time.

       

       

       

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            alexking_lee liying Assign to me
            alexking_lee liying

            Dates

              Created:
              Updated:

              Slack

                Issue deployment