Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-946

NameNode should not return full path name when lisitng a diretory or getting the status of a file

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed

      Description

      FSDirectory#getListring(String src) has the following code:
      int i = 0;
      for (INode cur : contents)

      { listing[i] = createFileStatus(srcs+cur.getLocalName(), cur); i++; }

      So listing a directory will return an array of FileStatus. Each FileStatus element has the full path name. This increases the return message size and adds non-negligible CPU time to the operation.

      FSDirectory#getFileInfo(String) does not need to return the file name either.

      Another optimization is that in the version of FileStatus that's used in the wire protocol, the field path does not need to be Path; It could be a String or a byte array ideally. This could avoid unnecessary creation of the Path objects at NameNode, thus help reduce the GC problem observed when a large number of getFileInfo or getListing operations hit NameNode.

        Attachments

        1. HDFSFileStatus.patch
          57 kB
          Hairong Kuang
        2. HDFSFileStatus1.patch
          58 kB
          Hairong Kuang
        3. HdfsFileStatus3.patch
          59 kB
          Hairong Kuang
        4. HdfsFileStatus4.patch
          59 kB
          Hairong Kuang
        5. HdfsFileStatusProxy-Yahoo20.patch
          1 kB
          Hairong Kuang
        6. HdfsFileStatus-yahoo20.patch
          51 kB
          Hairong Kuang

          Issue Links

            Activity

              People

              • Assignee:
                hairong Hairong Kuang
                Reporter:
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: