[HDFS-14738] Reading and writing HDFS data through HDFS nfs3 server often getting stuck

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.2
• Component/s: nfs
• Assignee: liying
• Reporter: liying
• Created: 15/Aug/19 07:18
• Updated: 03/Sep/19 01:19

Description

 We know the NFS Gateway supports NFSv3 and allows HDFS to be mounted as part of the client’s local file system.
I start the portmap and nfs3server in the hadoop node，and use the command （mount -t nfs -o vers=3,proto=tcp,nolock nfs3serverIP:/ /hdfs） to mount.
It works well。Then i ues this command to mount on many client.
And i used linux cp to copy files to the mounted dir(/hdfs), found the speed was very slow . There is a lot of information in the log as the follow:

2019-08-14 15:36:28,347 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,093 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:03,850 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,780 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:14,928 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:37:28,410 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:09,310 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:10,069 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,856 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:14,957 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:38:28,475 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,923 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:14,987 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:15,541 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:16,287 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:39:28,530 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,015 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,024 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:15,028 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:21,757 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:22,508 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:40:28,583 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,063 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:15,086 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:27,983 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,647 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:41:28,729 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,184 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:15,191 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332
2019-08-14 15:42:28,707 INFO org.apache.hadoop.security.ShellBasedIdMapping: Can't map group supergroup. Use its string hashcode:-1710818332

 

Then i print the jstack log of nfs3 server jvm, and i found that many threads waiting for monitor entry : BLOCKED(on object monitor). After checking the codes of ShellBasedIdMapping, found that It will invoke the updateMapInternal every time if the user or group doesn't exits. And many methods in this class are made with synchronized. So i think we can avoid to invoke updateMapInternal  every time.

 

 

 

Issue Links

• links to: GitHub Pull Request #1300
• links to: PR ：update updateMapIncr to avoid loading information form os every time