[HDFS-12748] NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.2
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
• Component/s: hdfs
• Target Version/s: 3.0.0
• Hadoop Flags: Reviewed
• Assignee: Weiwei Yang
• Reporter: Jiandan Yang
• Created: 31/Oct/17 03:23
• Updated: 25/Jul/19 20:59
• Resolved: 11/Jul/19 14:38

Description

In our production environment, the standby NN often do fullgc, through mat we found the largest object is FileSystem$Cache, which contains 7,844,890 DistributedFileSystem.
By view hierarchy of method FileSystem.get() , I found only NamenodeWebHdfsMethods#get call FileSystem.get(). I don't know why creating different DistributedFileSystem every time instead of get a FileSystem from cache.

    case GETHOMEDIRECTORY: {
      final String js = JsonUtil.toJsonString("Path",
          FileSystem.get(conf != null ? conf : new Configuration())
              .getHomeDirectory().toUri().getPath());
      return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
    }

When we close FileSystem when GETHOMEDIRECTORY, NN don't do fullgc.

    case GETHOMEDIRECTORY: {
      FileSystem fs = null;
      try {
        fs = FileSystem.get(conf != null ? conf : new Configuration());
        final String js = JsonUtil.toJsonString("Path",
            fs.getHomeDirectory().toUri().getPath());
        return Response.ok(js).type(MediaType.APPLICATION_JSON).build();
      } finally {
        if (fs != null) {
          fs.close();
        }
      }
    }

Issue Links

• is broken by: HDFS-8542 WebHDFS getHomeDirectory behavior does not match specification