Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-946

NameNode should not return full path name when lisitng a diretory or getting the status of a file

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      FSDirectory#getListring(String src) has the following code:
      int i = 0;
      for (INode cur : contents)

      { listing[i] = createFileStatus(srcs+cur.getLocalName(), cur); i++; }

      So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.

      FSDirectory#getFileInfo(String) does not need to return the file name either.

      Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.

      1. HDFSFileStatus.patch
        57 kB
        Hairong Kuang
      2. HDFSFileStatus1.patch
        58 kB
        Hairong Kuang
      3. HdfsFileStatus3.patch
        59 kB
        Hairong Kuang
      4. HdfsFileStatus4.patch
        59 kB
        Hairong Kuang
      5. HdfsFileStatusProxy-Yahoo20.patch
        1 kB
        Hairong Kuang
      6. HdfsFileStatus-yahoo20.patch
        51 kB
        Hairong Kuang

        Issue Links

          Activity

          Hairong Kuang created issue -
          Hairong Kuang made changes -
          Field Original Value New Value
          Attachment HDFSFileStatus.patch [ 12435520 ]
          Hairong Kuang made changes -
          Attachment HDFSFileStatus.patch [ 12435521 ]
          Hairong Kuang made changes -
          Assignee Hairong Kuang [ hairong ]
          Hairong Kuang made changes -
          Attachment HDFSFileStatus1.patch [ 12435631 ]
          Hairong Kuang made changes -
          Attachment HDFSFileStatus.patch [ 12435521 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus3.patch [ 12436085 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Incompatible change]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags [Incompatible change] [Incompatible change, Reviewed]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus4.patch [ 12436405 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus4.patch [ 12436622 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus4.patch [ 12436405 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus-yahoo20.patch [ 12436743 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus-yahoo20.patch [ 12436743 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus-yahoo20.patch [ 12436746 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus-yahoo20.patch [ 12436746 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatus-yahoo20.patch [ 12436753 ]
          Hairong Kuang made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hairong Kuang made changes -
          Attachment HdfsFileStatusProxy-Yahoo20.patch [ 12436769 ]
          Tom White made changes -
          Fix Version/s 0.21.0 [ 12314046 ]
          Fix Version/s 0.22.0 [ 12314241 ]
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue is related to HDFS-1743 [ HDFS-1743 ]

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development