Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15052

WebHDFS getTrashRoot leads to OOM due to FileSystem object creation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.0, 3.0.0-alpha2
    • Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
    • Component/s: webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Quoting Daryn Sharp inĀ HDFS-10756 :

      Surprised nobody has discovered this will lead to an inevitable OOM in the NN. The NN should not be creating filesystems to itself, and must never create filesystems in a remote user's context or the cache will explode.

      I guess the problem lies in side NamenodeWebHdfsMethods#getTrashRoot

      	  private static String getTrashRoot(String fullPath,
      	      Configuration conf) throws IOException {
      	    FileSystem fs = FileSystem.get(conf != null ? conf : new Configuration());
      	    return fs.getTrashRoot(
      	        new org.apache.hadoop.fs.Path(fullPath)).toUri().getPath();
      	  }
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                iwasakims Masatake Iwasaki
                Reporter:
                weichiu Wei-Chiu Chuang
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: